3D Object Manipulation in a Single Photograph using Stock 3D Models

Natasha Kholgade 1Tomas Simon 1Alexei Efros 2Yaser Sheikh 1
1 Carnegie Mellon University2 University of California, Berkeley


Latest
Support our campaign at Experiment.com for the next stage of our research on aligning 3D models to photographs at interactive rates. With your help, users will be able to instantly manipulate photographs in three dimensions with minimal effort.

We have been featured in Popular Science magazine's Best of What's New list of the top 100 innovations for 2014!

Press
Check out the press on our work on editing photographs in three dimensions:

TechCrunch


We were interviewed about our work by WESA (the NPR radio station in Pittsburgh), and we were featured live on the local radio station KQV 1410 AM.

Full Length Video

Abstract
Photo-editing software restricts the control of objects in a photograph to the 2D image plane. We present a method that enables users to perform the full range of 3D manipulations, including scaling, rotation, translation, and nonrigid deformations, to an object in a photograph. As 3D manipulations often reveal parts of the object that are hidden in the original photograph, our approach uses publicly available 3D models to guide the completion of the geometry and appearance of the revealed areas of the object. The completion process leverages the structure and symmetry in the stock 3D model to factor out the effects of illumination, and to complete the appearance of the object. We demonstrate our system by producing object manipulations that would be impossible in traditional 2D photo-editing programs, such as turning a car over, making a paper-crane flap its wings, or manipulating airplanes in a historical photograph to change its story.


Paper (14.4M) | Supplementary (35.7M) | BibTeX
SIGGRAPH 2014 Keynote | SIGGRAPH 2014 Powerpoint (Converted from Keynote)
Short Video (1280x720, 23.8M) | Full Length Video (1280x720, 150.9M) | Alignment Example
OS X (Mavericks) Executable Code and Examples | OS X Source Code (under the GPLv2 license)