15-862 Computational Photography
|Project 1: Colorizing the Prokudin-Gorskii Photo Collection - by Frank Palermo|
1. Color Channel Alignment: The algorithm first determines the offset vectors necessary to translate the green and red layers of Prokudin-Gorskii's images into alignment with the blue layer. For "small" images, this is done via a brute-force search over a window of [-20, 20] in each direction; the correctness of the alignment is judged by examining the Normalized Cross-Correlation between the channels at each offset. For "large" images (i.e. > 800px wide), a "multiscale pyramid"/"coarse-to-fine" approach is used (implemented via recursion). The image is repeatedly resized by a factor of 0.5 until it is ~400px across, then the brute-force search is applied as in the base case. This calculated offset is then used as an estimate and is refined at each level as the recursive calls return through progressively larger images. (These "refinements" require a much smaller search window (typically [-2,2] or less), dramatically improving performance).
2. Automatic Cropping: The algorithm next attempts to improve the appearance of the colorized images by several techniques, the first of which is automatic border cropping. It may be observed that the edges of Prokudin-Gorskii images often fall into the following two categories:
The cropping algorithm calculates the standard deviation between the R, G, and B channels at each pixel, and also their mean. It next finds the mean of these measures across each row and down each column. The algorithm scans inward from each of the four edges of the image and crops away those rows or columns which are deemed "too far away" from the overall mean (brightness) or standard deviation (variation between color channels) of the image as a whole.
3. Automatic Color & Contrast: The discussion in lecture regarding the HSV color space inspires the algorithm's final phase: automatic color and contrast adjustment. As we lack detailed information about Prokudin-Gorskii's camera, it is somewhat difficult to suggest reasonable enhancements to the (H)ue information. While various approaches could be used (e.g. the "Gray World Assumption," or attempting to identify a white point in the image), many Prokudin-Gorskii images already have "reasonable" white balance. Therefore, these techniques often result in adjustments that are either too small to be worthwhile, or are extreme to the point of ridiculousness.
The Prokudin-Gorskii images do, however, often appear "faded" or "washed-out" compared to modern photographs. We can change this by converting to the HSV color space, making alterations to the (S)aturation and (V)alue channels, and then converting back to RGB. In both cases, the general approach is to identify "low" and "high" threshold points of the S and V values currently in the image, then apply a mathematical operation to "stretch" this input range to a wider range at the output. The expansion of the (S)aturation channel results in a wider range of colors (from subtle, nearly gray shades to extremely vivid, bright ones), while the expansion of the (V)alue channel increases contrast in the traditional sense. (To reduce the appearance of artificially "overdone" color and contrast, neither adjustment outputs the full [0,1] range of these parameters).
The overall effect of the automatic enhancements described above may be seen in the following comparison, taken from one of the assigned images:
Results (Assigned Images)
Below are results for the assigned images which I consider to be entirely successful. Each of these images was aligned, cropped, and adjusted for color and contrast entirely within Matlab and without any user intervention beyond providing the input and output filename; only the JPEG encoding and resizing for web presentation were accomplished elsewhere (specifically via Adobe Photoshop CS3). Each image is accompanied by the alignment vectors used for the green and red layers, the automatically-calculated cropping rectangle, and data on saturation and brightness. On the current Wean 5336 cluster machines, the entire automated process takes approximately 10s to complete on a "small" image, or 1m10s for a "large" image.
* We can only speculate as to how far Prokudin-Gorskii traveled to capture this image.
A few more examples that turned out nicely, taken from photographs of my own choosing:
It seems that the automated cropping algorithm is the least robust of the three, as it failed to suitably crop two of the assigned images. The likely explanation is that both plates have writing marked in the black border surrounding the image, tricking the border detection (which is based on brightness and color variation) into thinking the borders are a part of the image itself. Still, the alignment, color, and contrast procedures were successful (and the cropping is acceptable, if not quite aggressive enough).
Just For Fun...
Just for fun: what happens if we attempt to reproduce Prokudin-Gorskii's work with a modern digital camera? The camera already contains color filters (a Bayer grid, in most cases), so all that is really needed is a way to access the image sensor data before the camera adjusts it (i.e. RAW mode). Using a camera with RAW mode (here a Canon PowerShot G9) and the open-source DCRAW utility, it is possible to extract the sensor data in a way consistent with our use of Prokudin-Gorskii's plates. Specifically, it is possible to override any color space conversions, white balance adjustments, etc. and simply use the unmodified sensor data for R,G, and B. (This parallels our use of the Prokudin-Gorskii images because we lack specific information on the color properties of his filters, the sensitivity of his glass plate emulsion, etc.). The pertinent command line options for DCRAW are:
A simulated Prokudin-Gorskii "BGR" plate, and the output of the restoration code written for this project, are reproduced below:
Green Alignment Vector: [0 0]
The result is mostly as expected: no offset is required for alignment, the cropping algorithm is too aggressive when there are actually no borders to be cropped, and the S and V channel ranges are expanded. However, the image is surprisingly green (and appears that way when rendered by Photoshop as well).
It turns out that the native output of the G9's sensor under daylight conditions requires significant white balance correction in order to produce a realistic color image. Normally, this correction would be performed either in the camera (when outputting JPEG) or by the RAW conversion utility but in this case it was deliberately disabled. My Prokudin-Gorskii image restoration code does not perform this correction because (as shown by the results above), most Prokudin-Gorskii images do not seem to require it.