HIGH DYNAMIC RANGE IMAGING
|COMPUTER VISION LABORATORY|
|Introduction | Exposure Patterns | Handling Color | Image Reconstruction | Other Applications | Another Result|
This page describes a new technology developed at
Columbia's Computer Vision Laboratory that can be used to
enhance the dynamic range (range of measurable brightness
values) of virtually any imaging system.
This technology includes a hardware and a software component. On the hardware side, a simple optical or electronic modification is made to an existing imaging system. The software is a set of algorithms that reconstruct high quality images from lower quality data captured using the modified imaging system.
Virtually any scene in the real world produces a very wide
range of brightness values. Imagine looking at a person's
face while the sun shines brightly from behind them, or a
large living room filled with antiques and lit by a single
lamp in one of its corners. Such conditions commonly produce
photographs (film or digital) that are too dark in some
regions and too bright (saturated) in others.
A typical digital camera can measure only about 8 bits (256 levels) of brightness. That is, all the brightness levels produced by the scene map to one of 256 brightness values. Of course, a photographer always tries to select the best exposure for each photograph. However, this does not change the number of brightness levels that the camera produces.
One might think that image processing methods could be used to enhance captured images and bring out scene details. This is true to a very limited extent. Areas of the captured image that are too saturated or really dark inherently lack scene information. No amount of post processing can recreate scene details that were never captured to begin with.
Consider the four 8-bit images of an outdoor scene shown in Figure 1. These images were taken using four different exposures (T, T/4, T/16, T/64). As is evident, the larger exposures produce images that are saturated in the well-lit regions of the scene (the sky and the surface of the clay oven). On the other hand, the two lower exposures produce images that are too dark in dimly lit areas (the inside of the clay oven and the shadowed regions). Brightening (enhancing) the dark regions only results in a grainy appearance as image noise dominates over scene brightness in these regions.
Figure 1: Four images of a scene taken using different exposures.
(click images to view high quality versions)
back to top
back to top
The assignment of the different exposures to pixels gets more
interesting in the case of a color camera. This problem was
recently explored by Shree Nayar and his graduate student
Typically, color imaging chips use a "mosaic" of color filters (red, green and blue). It turns out that, given this mosaic, a number of different exposure patterns can be used to ensure that, within every small neighborhood of pixels on the detector, each of the three colors is measured under different exposures.
Figure 3 shows one such assignment of exposures and color filters. The color filters are indicated by letters (R, G, B), and the exposures are indicated by different shades of gray. As one can imagine, such an imaging device will produce a rather complex (checkered in brightness and color) looking image.
A prototype camera that incorporates the SVEC imaging technique is currently being pursued by the Computer Vision Laboratory. Meanwhile, initial experiments have been done using SVEC images that are synthesized by appropriately sampling (in space and color) 8-bit images of a scene taken under different exposures.
Figure 4 is the single image that an SVEC camera would produce for the scene shown in Figure 1. This image is complex in terms of its local structure, as it represents a simultaneous sampling of scene radiance along three dimensions: space, color and exposure.
|RECONSTRUCTING HIGH QUALITY IMAGES|
Recently, Shree Nayar and Srinivasa Narasimhan have developed
a powerful set of algorithms that reconstruct high quality
images from a captured SVEC image.
The brightness/color of a scene point is highly correlated with the brightnesses/colors of its neighbors. This fact is exploited by the new algorithms to produce a high dynamic range image. This image is comparable to one produced by a 12 bit (4096 levels) camera, despite the fact that the imaging system used to capture the SVEC image uses a detector that yields only 8-bits (256 levels) of brightness information at each pixel. Even though the correlation between neighboring pixels is exploited by the algorithm, the reduction in spatial resolution is minimal (about 20%, the exact percentage depending on the nature of scene).
Figure 5 shows the high dynamic range image computed by the new algorithms from the SVEC image shown in Figure 4. To show all the details in this computed 12 bit image (which is being displayed on an 8-bit display - your computer monitor), we have used local contrast adjustments.
The richness of information in the image shown in Figure 5 is evident. One can see the subtle shadings of the bright clouds in the sky, the details of the texture on the outside surface of the clay oven, the very dimly lit logs of wood and the picture inside the oven. To a human observer the computed image appears comparable in dynamic range to that produced by an high-end (professional grade) digital camera.
back to top
We have described the new technology
in the context of digital still cameras.
The reconstruction of high quality images from the SVEC image
is efficient and hence can also be applied to digital video
cameras. A simple signal processing module embedded
within the electronics of the video camera can produce high
dynamic range images in real time (30 frames per second).
Such an implementation is also planned.
It is worth noting that the use of spatially varying pixel exposures has far reaching implications. The basic principles are applicable to any form of electromagnetic radiation and not just the visible light spectrum. Hence, the dynamic ranges of a wide variety of imaging systems can be enhanced using the present invention. These include X-ray, IR (infra-red), SAR (synthetic aperture radar), and MR (magnetic resonance) imaging systems.
back to top
|ANOTHER RESULT ON DIGITAL IMAGES|
Here, we show a another example of our results. We
first show four 8-bit images of a scene taken
using different exposures. These 8-bit images convey the fact
that, for these scenes, no single exposure can capture the
entire range of brightness variations; each image includes
one or more regions that are either of near-zero brightness
or saturated. Next, we show the 8-bit SVEC image for the
scene. Finally, we show the 12-bit image computed from the
8-bit SVEC image.
back to top