Sundar's Research Page

This page summarizes my work at CMU upto Dec. 2001. My work since then is all corporate and not available publicly.

I'm a part of the Virtualized Reality project in the Vision and Autonomous Systems (VASC) group at Carnegie Mellon University. Imagine being able to reconstruct virtual models of dynamic sequences, and then flying around such a reconstruction in a viewer controlled fashion! We do this by integrating views of this event from a number of video cameras around it, and creating CAD models and texture mapping them. Click here for the Project home page, and many more details and results. My research advisor is Prof. Takeo Kanade.

Thesis Work

My thesis work aims at creating a 4-D (space+time) model of a dynamic event, similar to the model shown on the left. While this image shows the model at discrete time intervals from a single camera position, I am working on algorithms to visualize physical events by creating virtual images from synthetic cameras located at arbitrary viewpoints, at arbitrary time instants. My Ph.D thesis proposal is titled Multi-view spatial and temporal interpolation for dynamic event visualization. It is available here (pdf).
mpeg movie

The problem of Shape and Motion recovery

Example input image Recovered Shape+Motion Mpeg movies
Shape interpolated along computed scene flow
Creating a virtual view from an arbitrary position at an arbitrary time requires algorithms for scene shape reconstruction, estimation of 3-D scene motion, and image based interpolation between sampled camera images.

3-D scene flow : We have developed algorithms for computing instantaneous motion of a general, non-rigid scene. For a scene whose shape is already known, we propose algorithms to compute 3-D scene flow, which is a non-rigid motion field of all points in an arbitrary scene. Knowledge of such a field allows us to determine the motion of any particular scene point over time, from images sampled at discrete locations and at discrete times. (for full details, download my ICCV '99 paper, Three Dimensional Scene Flow). This is joint work with Simon Baker, Pete Rander, Bob Collins, and Takeo Kanade.

Shape and Motion Carving in 6D: I have developed an algorithm for simultaneously recovering scene structure and scene flow. The idea is an extension of Voxel Coloring (Seitz, CVPR '96) which carves away voxels in a volume that do not project to photo-consistent pixels in multiple cameras. Instead, we compute this photo-consistency across space and time. Starting from the 6D space of all possible scene structures at two time instants and all possible flows connecting them, we carve away hexels (6D voxels) that are inconsistent with the captured images, leaving us with two shapes and the scene flow between them. This work was presented at CVPR 2000, click here to download the paper. This is joint work with Simon Baker, Steve Seitz, and Takeo Kanade.

Applications : Applications include visualization of complex surgical procedures, entertainment, or training, where the physical locations and/or image capture rates of cameras are constrained, but there is a need to accurately reconstruct the appearance from any position, at any time instant.

Other related work

I've done some work on an idea called "Model Enhanced Stereo". Typical stereo matching using widely separated cameras produces range estimates with a large number of outliers. Instead, the Model Enhanced Stereo algorithm uses a reconstructed 3-D model to guide the stereo matching process. This model is created automatically by volumetric integration of initial range estimates produced by conventional multi-baseline stereo. Check out the paper in The International Conference on Virtual Systems and Multimedia (VSMM) '98 (Gifu, Japan)

The CMU 3DRoom : During Fall 1998, I (along with Hideo Saito) redesigned our entire camera system, to capture images digitally, instead of on videotape. A cluster of PCs, with one digitizer for each video stream is used to store the image streams directly onto disk. The digitizers are all synchronized with each other in real-time to start and end video capture at the same time. More information on the CMU 3DRoom.

On a project of this scale, I spend a lot of time putting together hardware and (mostly) software systems. Its great to have tons of tools around - see my page of software tools.

Back to my Home Page

Sundar Vedula
srv@cs.cmu.edu
July 2000