Interim Research Results:
Generating Images with Arbitrary Focus Regions


11/15/99

Thomas Drake Jr

drakejr@andrew.cmu.edu

Current panoramic images allow the user move and zoom freely in the given texture environment (QuickTimeVR, Ipix and iMove). However, the user is limited to the depth of field of the images when they were acquired. This can greatly reduce the information presented in a panorama when objects are very close the camera. The depth of field, in this case, is slim for real cameras [1]; if real cameras could approximate the depth of field of a computer simulated pin-hole camera, this topic would be unappropriate. It would useful if a panoramas could be constructed such that everything in the scene is in focus.

I present a novel approach for generating images with arbitrary focus regions (regions of the image that are in focus) from images take from multiple depth of field. But, first thing is first!

Image Acquisition
For now I have acquired image sequences at different depths, that I plan on stitching together. I am not satisfied with the current image quality, but it was the only scene that I could construct where changes in focus was highly visible. I sampled images from a dense woodland scene and an art gallery, but my cameras depth of field became too deep when the closest objects came into focus.

Automatic Image Registration
I am using Creating Full View Panoramic Image Mosaics and Envionment Maps> (Szeliski, Shum) to automatically stitch several sequences of images from a several different depth of fields. Their technique involves generating an approximate focal length from the image (this should be static across all images) from the standard 8-parameter model. Using a 3-parameter model (rotations about each coordinate axis in 3D), they then find the corresponding rotation difference between sequential images by minimizing the SSE of the differences of corresponding pixel illuminations. A novel approach, indeed.

Manual Mosaic Registration
My novel focus region generation technique requires that the mosaics be registered to each other, or to some global coordinate system. I plan on projecting the images onto cylindrical coordinates and registering them by hand using transparency. I don't think that normal or abnormal registration could be applied here since each mosaic contains many differing blurred and focused regions.

NOVEL METHODS!
When first attempting to add two focus regions from separate images, I noticed this: I had been visited by the Focus Ghost. You see, a simple averaging of two images from the same viewpoint with different depth of field results in the halo surrounding detailed parts from both images. This result was not acceptable.

But then I realized, that there was a novel (or not-so-novel) solution. For each mosaic or, in this case, image generate a focus mask that represents the detailed regions of it's image. I searched for an existing algorithm for my problem, but found none. It's all Depth from Defocus [2] nowadays.

The problem is a little more slippery than just edge detection, my first solution. After some experimentation, I was able generate a decent mask as follows:



The original images: a completely defocused image, image focus on the closest object and furthest object.

Perform a single pass point derivative on the images with focus information.

Threshold each image at the top five percent of its histogram. I need to play with that percentage a little.

Blow up each resulting pixel in the image (I need to play with this value as well). I eyed the pixel scaling value here, to more than adequately mask over the circles of confusion, the blurring resulted from defocused points from an image [3].

Here we contrast the image created from simple pixel averaging, to the new image where points that fall within the masked region are weighted much heavier. You can see we have not yet implemented feathering of masks edge (Shum is falling behind). But I will. You can probably also see that the mask for this image picked up extra information. I'm hoping that feathering will fix this.


Conclusion

My novel method for determining focused regions in an image approximates the blurred region that results from defocused point in the scene projected onto the image plane. By replacing those blurred regions with detailed focus information, a deeper depth of field in a single image can be created; and thereby, better approximate an entire scene (or allow images to be rendered at arbitrary depth of field). I need to play with the abovementioned values. I also thinking of a weighting scheme based on the clusters from the mask.

And so, it is. As you can tell, I have not yet finished the image registration. My research proposal detailed a certain 3D interface that would allow for multiple regions of focus. I'm not so sure that will be completed, given the amount of image processing I think this project will need. But I'm sure there is something to be said for wishful thinking. I also plan on taking many more images to test the robustness of my not-so-novel technique.



Bibliography:

[1] R. Szeliski, H. Shum. Creating Full View Panoramic Image Mosaics and Environment Maps. ICCG, 1997.
[2] M. Watanabe and S. K. Nayar. Rational Filters for Passive Depth from Defocus.IJCV,1997.
[3] M. Potmesil, I. Chakravarty. A Lens and Aperture Camera Model for Synthetic Image Generation. SIGGRAPH, V15, No. 3. August, 1981.