15-463 Computational Photography

Fall 2011

Programming Project 1

Images of the Russian Empire:


Colorizing the Prokudin-Gorskii photo collection

Author

Tanachat Nilanon (tnilanon)

Background

Sergei Mikhailovich Prokudin-Gorskii (1863-1944) [Сергей Михайлович Прокудин-Горский] was a man well ahead of his time. Convinced, as early as 1907, that color photography was the wave of the future, he won Tzar's special permission to travel across the vast Russian Empire and take color photographs of everything he saw. And he really photographed everything: people, buildings, landscapes, railroads, bridges... thousands of color pictures! His idea was simple: record three exposures of every scene onto a glass plate using a red, a green, and a blue filter. Never mind that there was no way to print color photographs until much later -- he envisioned special projectors to be installed in "multimedia" classrooms all across Russia where the children would be able to learn about their vast country. Alas, his plans never materialized: he left Russia in 1918, right after the revolution, never to return again. Luckily, his RGB glass plate negatives, capturing the last years of the Russian Empire, survived and were purchased in 1948 by the Library of Congress. The LoC has recently digitized the negatives and made them available on-line.

Overview

The goal of this assignment is to take the digitized Prokudin-Gorskii glass plate images and, using image processing techniques, automatically produce a color image with as few visual artifacts as possible. In order to do this, you will need to extract the three color channel images, place them on top of each other, and align them so that they form a single RGB color image. We will assume that a simple x,y translation model is sufficient for proper alignment. However, the full-size glass plate images are very large, so your alignment procedure will need to be relatively fast and efficient.

Approach

The digitized photographs contain blue-filtered, green-filtered, and red-filtered channels arranged from top to bottom. These channels must be separated first. As the extent of each channel varies between photographs, a crop mask that is bigger than the actual channel size was used to make sure that precious information is not thrown away intentionally. Now these three channels needs to be aligned. One important feature that is essentially preserved across channels is the edges. Sobel filters in X and Y direction were used as edge detector. Edge magnitude is computed by treating the responses from Sobel filters in X and Y direction as vectors, adding them together, and finding the magnitude. Alignment between the channels would be determined by these edge magnitude images from each channel . However, as the crop mask used is bigger than the actual channel size, many artifacts from channel borders appeared in the edge magniture images. To make the alignment result uninfluenced by the channel borders, only the inner region of each edge magniture image was used. Each inner region is comprised of 60.8% of original edge magniture image (10% horizontal border, 12% vertical border). Here, image pyramids were used to speed up the alignment process. After the alignment was found, original channels were composed together to form one single RGB image.

Additional processing was done to crop the RGB image to get rid of strange colors in the borders. The approach is also based on finding the edges in each channel. The top border was found from looking for horizontal edges in the top region of the blue channel. The bottom border was found from looking for horizontal edges in the bottom region of the red channel. The left and right borders were found from looking for vertical edges in the left and right regions of all the channels. The RGB image is then cropped down to the determined borders.

Result

Example set

(given on the project page)

Digitized photo name	Channel offsets	Digitized photo	Original result	Cropped result
00088v.jpg	G: (3, -11) R: (5, -22)
00106v.jpg	G: (1, -10) R: (-1, -17)
00137v.jpg	G: (6, -8) R: (8, -15)
00757v.jpg	G: (3, -12) R: (5, -21)
00888v.jpg	G: (1, -8) R: (1, -14)
00889v.jpg	G: (2, -12) R: (3, -22)
00907v.jpg	G: (0, -12) R: (0, -20)
00911v.jpg	G: (-1, -13) R: (-1, -13)
01031v.jpg	G: (1, -13) R: (2, -22)
01880v.jpg	G: (2, -8) R: (4, -12)
00029u.tif	G: (14, -89) R: (32, -164)
00087u.tif	G: (26, -91) R: (38, -202)
00128u.tif	G: (25, -73) R: (33, -134)
00737u.tif	G: (11, -77) R: (19, -144)
00822u.tif	G: (32, -87) R: (59, -146)
00892u.tif	G: (37, -76) R: (56, -148)
00992u.tif	G: (7, -113) R: (15, -207)
01043u.tif	G: (2, -114) R: (4, -216)
01085u.tif	G: (9, -141) R: (15, -243)
01734u.tif	G: (28, -80) R: (49, -153)

Extra set

(chosen from the Library of Congress website)

Digitized photo name	Channel offsets	Digitized photo	Original result	Cropped result
00099u.tif	G: (4, -66) R: (-1, -125)
00111u.tif	G: (34, -73) R: (60, -131)
00117u.tif	G: (0, -80) R: (-12, -133)
00402u.tif	G: (20, -52) R: (26, -108)
00810u.tif	G: (-31, -73) R: (-55, -129)
00955u.tif	G: (19, -107) R: (30, -198)
01115u.tif	G: (25, -64) R: (36, -116)
01332u.tif	G: (1, -100) R: (-1, -174)
01443u.tif	G: (21, -94) R: (44, -175)
01779u.tif	G: (21, -75) R: (28, -139)

Problem

There is no problem with aligning the color channels. However, the algorithm used to crop RGB images sometimes cropped the images thicker than necessary. Examples of this can be found in 00029u.tif, 00087u.tif, 01043u.tif, 01115u.tif, and 01779u.tif.

Suggestion / Future work

An alternative cropping algorithm could be develop from the idea given on the project page. Another idea is that, since the hues of the filters Prokudin-Gorskii used are not known, it is not precise to associate the color channels to the RGB. The hue differences can be determine from general objects such as sky, grasses, stones, and metals. Then a weighted mapping between the color channels and the RGB can be constructed.