Colorizing the Prokudin-Gorskii Photo Collection

Background
Algorithm
Results
- Small Images
- Large Images

Background

Sergei Mikhailovich Prokudin-Gorskii (1863-1944) [Сергей Михайлович Прокудин-Горский, to his Russian friends] was a man well ahead of his time. Convinced, as early as 1907, that color photography was the wave of the future, he won Tzar's special permission to travel across the vast Russian Empire and take color photographs of everything he saw. And he really photographed everything: people, buildings, landscapes, railroads, bridges... thousands of color pictures! His idea was simple: record three exposures of every scene onto a glass plate using a red, a green, and a blue filter. Never mind that there was no way to print color photographs until much later—he envisioned special projectors to be installed in "multimedia" classrooms all across Russia where the children would be able to learn about their vast country. Alas, his plans never materialized: he left Russia in 1918, right after the revolution, never to return again. Luckily, his RGB glass plate negatives, capturing the last years of the Russian Empire, survived and were purchased in 1948 by the Library of Congress. The LoC has recently digitized the negatives and made them available on-line.

Algorithm

The glass plate negatives contain three color channel images. We split the image into three parts to extract the channels. The first channel is the blue channel, followed by the green channel, and the red channel. We keep the blue channel fixed and align the green and red channels to the blue channel.

To compute the best alignment between two channels, we "slide" one image over the other within a [-15, 15] window while measuring the quality of each alignment. We used two metrics: sum of squared differences (SSD) and normalized cross-correlation (NCC). SSD was faster and produced results indistinguishable from NCC, so we used SSD. After the green and red channels were aligned, we applied the displacements and merged the channels.

The approach thus far worked well for smaller images, but was much too slow for the larger images, which were around 70MB in size and very high resolution. To reduce the runtime, we used an image pyramid (a.k.a. mipmap). Instead of searching a [-15, 15] window on the large images, we downscale both images by .5 and find the best alignment of the downscaled images. This alignment is used as the "center" of a much smaller [-5, 5] window in the alignment of the original scale images. We apply this recursively until the images are smaller than 512 by 512, in which case we use the original algorithm with a window of [-15, 15]. This optimization resulted in a much faster runtime.

However, the results on the large images were terrible. It turns out that the borders were throwing off the alignment. We removed 10% of the image from all four sides to remove the borders. The alignment algorithm was run on the remaining image and the results were very good.

Results

The following are the original color channels, colorized photos without alignment, and colorized photos after alignment along with alignment displacements.