Given a static scene, a human can trivially enumerate the myriad of things that can happen next and characterize the relative likelihood of each. In the process, we make use of enormous amounts of commonsense knowledge about how the world works. In this paper, we investigate learning this commonsense knowledge from data. To overcome a lack of densely annotated spatiotemporal data, we learn from sequences of abstract images gathered using crowdsourcing. The abstract scenes provide both object location and attribute information. We demonstrate qualitatively and quantitatively that our models produce plausible scene predictions on both the abstract images, as well as natural images taken from the Internet.
This is a joint work with Larry Zitnick, Microsoft Research.
David Fouhey is a Ph.D. student at the Robotics Institute, Carnegie Mellon University, where he is supervised by Abhinav Gupta and Martial Hebert. He holds an A.B. in Computer Science from Middlebury College. His research addresses a variety of scene understanding tasks in computer vision, with a particular interest in inferring 3D properties of a scene from a single image.