Real-time Image Processing Using Smart Filters

What is the AOTF camera?

The Acousto-Optical Tunable Filter (AOTF) camera is an implementation of a smart filter. It operates much like an ordinary CCD camera, except that the incoming light is filtered based on its spectrum and polarization before being imaged. The filtering is specified by a computer, which sends control information to the camera. The filtered scene is then captured by an ordinary ccd camera and sent back to the controlling computer via a frame grabber.

The AOTF camera Inside the AOTF camera

The AOTF crystal performs the spectral filtering, and the phase retarder works in conjunction with the AOTF crystal to accomplish polarimetric filtering.

Control strategy selection

An important question that must be answered is "What control information should be sent to the camera?" To a large extent, the choice of this control strategy depends on the application, but some general principles can be noted.

One obvious concept involves applying a narrowband spectral filter to the image while sweeping the center wavelength of the filter over the range of the camera. For example, one might vary the wavelength from 400 to 800 nanometers in increments of 10 nanometers, thereby generating 41 bands. The output of this spectral sweep is an image cube with axes in x, y, and wavelength. It is interesting to note that most objects do not reflect light over just a narrow range of wavelengths. Instead, they reflect light over all wavelengths, perhaps reflecting more at some wavelengths. As a result, a stop sign is visible in all the images, but is brighter in wavelengths in the red area of the spectrum.

560 nm. band of stop sign scene 720 nm. band of stop sign scene

Unfortunately, performing a spectral sweep takes time, which means that objects may move significantly between the instant that the first image is captured and the moment the sweep is completed. At the other extreme is a strategy called band alternation. In this case, two (or a few) bands are rapidly alternated.

The AOTF is not limited to simple narrowband filters, though. More complex waveforms can be generated using an arbitrary waveform generator. With these waveforms, complex filtering operations, such as those performed by the human eye, could be executed in a single frame.

Another possible control strategy is to combine the slow spectral sweep with the fast alternation strategy. The slow sweep can be performed infrequently, and the results analyzed to determine the bands or waveforms to be used in the alternation strategy, which is then executed in a fast loop. In this way, one can adapt the strategy dynamically using all the available spectral information without the penalty of constantly executing a spectral sweep.

Adaptive control strategy block diagram

Example Application: Colorization

Colorization is one interesting application for the AOTF camera. Each image in a spectral sweep is monochrome, but by combining the intensities of all the bands, the original color image can be reconstructed. An interactive sensitivity curve editor allows one to specify the proportion of each band that is used to generate the red, green, and blue components of the final image.

Example colorized image Example colorized image Example colorized image Colorization serves two purposes. First, it allows us to visually verify that the camera is indeed accurately capturing the spectral content of a scene. Merely looking at a series of monochrome images gives no real feel for this content. Second, since we can arbitrarily adjust the color sensitivity curves, we can simulate a variety of real and artificial systems, including ccd cameras and human eyes.

One band of the image stack (570 nm.) Colorized version The trees and grass are enhanced by adjusting the sensitivity curve for green

Terrain Classification

An off-road scene colorized by the AOTF camera Our interest in the AOTF camera lies in its ability to perform terrain classification. At CMU, the Unmanned Ground Vehicle (UGV) project has demonstrated autonomous planning, mapping, and off-road navigation skills using the NavLab II, a modified Army HMMWV. But despite its impressive capabilities, NavLab II is unable to distinguish between rocks and tall grass, trees and hillsides, or even mud and hard ground. As a consequence, the vehicle plans and navigates conservatively, avoiding all objects that may be potential hazards. By identifying and classifying the different types of terrain in a scene, we reduce the number of false positive obstacles, such as tall grass, as well as false negatives, such as water and mud.

Terrain classification is difficult with a monochrome camera because different terrain types may produce the same image intensity. A color camera alleviates this problem somewhat, but the off-road terrain in which we are interested often contains only muted colors, which are difficult to distingish using only the red, green, and blue components of the scene. The AOTF camera provides us with fine-grain measurements over the full visible spectrum as well as the near infrared.

Vegetation Detection

As a first step in terrain classification, we would like to be able to quickly and reliably detect vegetation in an arbitrary scene. Fortunately, vegetation exhibits a rather unique spectrum due to the clorophyll in the leaves.

The unique spectrum of vegetation

By taking the ratios of intensities of two bands, we can discriminate vegetation from non-vegetation. The result of this computation can be used in its raw form as a confidence measure or as a thresholded image. By combining the thresholded image with the colorization algorithm, we can selectively eliminate all vegetation or all non-vegetation from a scene.

Colorized scene of a hillside and houses Raw output of the vegetation detector Thresholded output of the vegetation detector

Everything but the vegetation is masked The vegetation is masked

Classification

The next step beyond detection of vegetation is classification of arbitrary objects. We have developed a classifier that uses a database of spectra to segment an image. At each point in the test image, the spectra is assigned the classification of its nearest neighbor in the database using the Mahalanobis distance metric.

Example of classification using stop sign scene Example of classification using stop sign scene Example of classification using scene with various bushes Example of classification using scene with various bushes

Road Hazard Detection

Example of road hazard detection We have also experimented with the control of the polarimetric filtering capability of the AOTF camera. In one experiment, we tested whether the system could detect regions of high glare on a road. Frequently, glare on a road is due to water or icy regions. By detecting glare, we could provide an early warning system to drivers or an additional input to autonomous vehicles. In the example below, the top two images were obtained by adjusting the retarder for minimum and maximum glare. The regions of glare are then detected simply by subtracting the two images. As with vegetation detection, a binary output can be obtained by thresholding the resulting image. This algorithm demonstrates that our system can accomplish difficult computer vision tasks using simple algorithms. The problem has been made easier by transferring the complexity into the hardware.

Current Directions

Now we are working in two primary areas to extend the functionality of the AOTF camera system. First, we are looking at ways to automatically select a small number of bands to perform arbitrary discrimination or feature enhancement. This is basically an extention of the vegetation detection work (in which the bands were selected empirically). Second, we are working on ways to alleviate the problem of varying illumination. In laboratory situations, the illuminant remains relatively constant throughout an experiment, but for outdoor applications, the illumination varies drastically from sunny to completely overcast. We are working to make our classification algorithm more robust against such effects.

Last modified January 28, 1998
Daniel Huber ( dhuber@cs.cmu.edu)