15-869 Image Based Modeling & Rendering

Assignment 2: View Transformation

Results page for:  Chris Rodriguez (cjr2@andrew.cmu.edu)

For this assignment, we used a Minolta Vivid 700 scanner to get a texture and depth range to work with.  Starting with Prof. Heckbert's Inventor format converter code and Prof Seitz's fltk windowing code, I created a program that can read in a Inventor file and display it on the screen.

Here are the images of the object I scanned, a monkey with a banana:

Scene Scanned	Depth information retrieved from Monkey (w/ Texture)

From then, I started working on the algorithm from the McMillan Paper.  It was relatively easy to understand and code.  The one problem that I did have was in interpreting the delta values (that they are actually 1/z, not z itself).  This caused a major problem with my interface for a long time - when you rotated the object, it would just fly off the screen!  

Here is a picture of the interface I was using:

I had sliders for the rotation and translation.  I had thought of doing a virtual trackball type thing and doing incremental changes to u, v, o and c and saving them, but it did not seem necessary.  I also have a button for adding in the Z unknown pixels back in, but for the most part it didn't look as good to me.  Also, you can turn hole filling on an off as a speed up.

The way I did the program is with just using OpenGL's raster commands.  I had no real interest in creating a polygon scan converter, and we couldn't use OpenGL.  I used the Grid Data structure from the ivgrid.c file, modified the grid.h to carry the extra data I needed to allow my program to read in Inventor files without outside help.  From there I did a little modification of the mosaic starter code to create my interface, and added a file to do the parts of McMillan's algorithm such as computing the warp as well as the new point of view and visibility determination.  

Here is a picture from before visibility determination and hole filling, but you can clearly see the 3d nature of the point cloud, as well as see some of the missing data::

Finished Product:  Here are some images from the 9 different camera projection positions, as described in the paper.

Here is a 90 image that looks pretty good, but definitely has some issues with hole filling colors, as you can see to the right of his arm where it is a bit lighter color with dark speckles.

My implementation seems to work for the most part, but there are some definite problems. 

• One is that because I am using glPixelZoom, I only check for holes in every other pixel, and this sometimes does not seem to be enough.  You can see one of these errors in the left center picture -- there is a line across the bottom of the banana that doesn't seem to go away with hole filling.  This is especially clear in this picture with the background on: you can see the cracks in the background picture.  I assume that my hole filling is skipping over this because it works at a 200x200 resolution, and OpenGL is displaying at a 400x400 resolution.
• Hole filling is very primitive.  All I do is go along the x scanline, look for blue pixels (my model had black in it, so I had to change the background color) that are between non-blue pixels and linearly interpolate the color.  Sometimes between the two colors wasn't a real hole in the picture data.  Also, my hole filling would screw up if the background picture was turned on. Example of bad span
• Sometimes it seemed as if the visibility determination didn't work for me.  Perhaps this is because the data on the sides of the monkey was sparse, and you could see through it, then the hole filling would happen.  Here is an example of what I am talking about:

you can clearly see the monkey's left arm through his head.  When I turn the hole filling off, you can see through the monkey's face and I believe that's what causes it.

Problems:

Aside from a few small dumb mistakes, my biggest problem had to be the fact that I was misinterpreting the delta values as Z instead of 1/z which was not allowing me to test the code because I could not rotate.  Once I figured that small fact out, the rest of the project was pretty smooth