This page describes a new technology developed at Columbia's Computer Vision Laboratory that can be used to enhance the dynamic range (range of measurable brightness values) of virtually any imaging system.

This technology includes a hardware and a software component. On the hardware side, a simple optical or electronic modification is made to an existing imaging system. The software is a set of algorithms that reconstruct high quality images from lower quality data captured using the modified imaging system.

Virtually any scene in the real world produces a very wide range of brightness values. Imagine looking at a person's face while the sun shines brightly from behind them, or a large living room filled with antiques and lit by a single lamp in one of its corners. Such conditions commonly produce photographs (film or digital) that are too dark in some regions and too bright (saturated) in others.

A typical digital camera can measure only about 8 bits (256 levels) of brightness. That is, all the brightness levels produced by the scene map to one of 256 brightness values. Of course, a photographer always tries to select the best exposure for each photograph. However, this does not change the number of brightness levels that the camera produces.

One might think that image processing methods could be used to enhance captured images and bring out scene details. This is true to a very limited extent. Areas of the captured image that are too saturated or really dark inherently lack scene information. No amount of post processing can recreate scene details that were never captured to begin with.

Consider the four 8-bit images of an outdoor scene shown in Figure 1. These images were taken using four different exposures (T, T/4, T/16, T/64). As is evident, the larger exposures produce images that are saturated in the well-lit regions of the scene (the sky and the surface of the clay oven). On the other hand, the two lower exposures produce images that are too dark in dimly lit areas (the inside of the clay oven and the shadowed regions). Brightening (enhancing) the dark regions only results in a grainy appearance as image noise dominates over scene brightness in these regions.

Figure 1: Four images of a scene taken using different exposures.
(click images to view high quality versions)

SPATIALLY VARYING PIXEL EXPOSURES

Prof. Shree Nayar and his visiting scientist Tomoo Mitsunaga (from Sony) have developed a new technique, called spatially varying pixel exposures, for capturing a very wide range of brightness values. Technically, this new method can enhance the dynamic range (range of measurable brightness values) of virtually any imaging system. Here, we describe the concept using a typical digital camera as the imaging system of choice.

At the heart of any digital camera is a small solid-state device (a chip) that converts light energy to electrical signals. The lens of the camera creates a focused optical image of the scene on the surface of the chip. The chip includes a large number (there are millions in today's consumer cameras) of "pixels". Each pixel collects light falling on it and produces an electric charge that is proportional to the intensity of light to which it is exposed. These charges are read out to obtain a digital image of the scene.

While in a conventional imaging chip, all pixels are "equal" in the way they collect light, in the new invention neighboring pixels are differently exposed, as shown in Figure 2. In this example, four neighboring pixels have different exposures and this exposure pattern is repeated over the entire array of pixels on the device.

Figure 2: Spatially varying pixel exposure pattern.
(click image to view high quality version)

Such an exposure pattern can be implemented in many ways. One approach is to place a mask with cells of different optical transparencies adjacent to the detector array. The pattern can also be etched directly on the detector in solid state devices such as charge-coupled devices (CCD's). Alternatively, the sensitivity of the pixels can be preset by using different microlenses for neighboring pixels, or by using different integration times for different pixels, or by embedding different apertures for the potential wells of the pixels. All these implementations result in the same effect, namely, a detector array with spatially varying exposures.

The key observation to be made is that when one pixel is saturated, it is very likely that one or more of its neighboring pixels will produce meaningful brightness measurements. Similarly, when a pixel produces zero brightness due to low exposure, it is likely that one of its neighbors will produce a non-zero brightness measurement. This is the basic concept underlying the new image capture technology.

HANDLING COLOR

The assignment of the different exposures to pixels gets more interesting in the case of a color camera. This problem was recently explored by Shree Nayar and his graduate student Srinivasa Narasimhan.

Typically, color imaging chips use a "mosaic" of color filters (red, green and blue). It turns out that, given this mosaic, a number of different exposure patterns can be used to ensure that, within every small neighborhood of pixels on the detector, each of the three colors is measured under different exposures.

Figure 3 shows one such assignment of exposures and color filters. The color filters are indicated by letters (R, G, B), and the exposures are indicated by different shades of gray. As one can imagine, such an imaging device will produce a rather complex (checkered in brightness and color) looking image.

Figure 3: Spatially varying exposure and color (SVEC) pattern.
(click image to view high quality version)

A prototype camera that incorporates the SVEC imaging technique is currently being pursued by the Computer Vision Laboratory. Meanwhile, initial experiments have been done using SVEC images that are synthesized by appropriately sampling (in space and color) 8-bit images of a scene taken under different exposures.

Figure 4 is the single image that an SVEC camera would produce for the scene shown in Figure 1. This image is complex in terms of its local structure, as it represents a simultaneous sampling of scene radiance along three dimensions: space, color and exposure.

Figure 4: SVEC image, and an enlargement
of its grid pattern, for the scene shown in Figure 1.
(click image to view high quality version)

RECONSTRUCTING HIGH QUALITY IMAGES

Recently, Shree Nayar and Srinivasa Narasimhan have developed a powerful set of algorithms that reconstruct high quality images from a captured SVEC image.

The brightness/color of a scene point is highly correlated with the brightnesses/colors of its neighbors. This fact is exploited by the new algorithms to produce a high dynamic range image. This image is comparable to one produced by a 12 bit (4096 levels) camera, despite the fact that the imaging system used to capture the SVEC image uses a detector that yields only 8-bits (256 levels) of brightness information at each pixel. Even though the correlation between neighboring pixels is exploited by the algorithm, the reduction in spatial resolution is minimal (about 20%, the exact percentage depending on the nature of scene).

Figure 5 shows the high dynamic range image computed by the new algorithms from the SVEC image shown in Figure 4. To show all the details in this computed 12 bit image (which is being displayed on an 8-bit display - your computer monitor), we have used local contrast adjustments.

The richness of information in the image shown in Figure 5 is evident. One can see the subtle shadings of the bright clouds in the sky, the details of the texture on the outside surface of the clay oven, the very dimly lit logs of wood and the picture inside the oven. To a human observer the computed image appears comparable in dynamic range to that produced by an high-end (professional grade) digital camera.

Figure 5: High dynamic range (12-bit) image
computed from the 8-bit SVEC image
shown in Figure 4.
(click image to view high quality version)

OTHER APPLICATIONS

We have described the new technology in the context of digital still cameras. The reconstruction of high quality images from the SVEC image is efficient and hence can also be applied to digital video cameras. A simple signal processing module embedded within the electronics of the video camera can produce high dynamic range images in real time (30 frames per second). Such an implementation is also planned.

It is worth noting that the use of spatially varying pixel exposures has far reaching implications. The basic principles are applicable to any form of electromagnetic radiation and not just the visible light spectrum. Hence, the dynamic ranges of a wide variety of imaging systems can be enhanced using the present invention. These include X-ray, IR (infra-red), SAR (synthetic aperture radar), and MR (magnetic resonance) imaging systems.

ANOTHER RESULT ON DIGITAL IMAGES

Here, we show a another example of our results. We first show four 8-bit images of a scene taken using different exposures. These 8-bit images convey the fact that, for these scenes, no single exposure can capture the entire range of brightness variations; each image includes one or more regions that are either of near-zero brightness or saturated. Next, we show the 8-bit SVEC image for the scene. Finally, we show the 12-bit image computed from the 8-bit SVEC image.



Example 1A: Four images of a scene taken using different exposures. (click images to view high quality versions)

Example 1B: SVEC image, and an enlargement
of its grid pattern, for the scene
shown in Example 1A.
(click image to view high quality version)

Example 1C: High dynamic range (12-bit) image
computed from the 8-bit SVEC image
shown in Example 1A.
(click image to view high quality version)