15-463 Project 1: Images of the Russian Empire - Colorizing the Prokudin-Gorskii Photo Collection
Fall 2008
Greg Eden


General notes

In order to align the red and green plates with the blue plate for a given image, I basically tried to find an alignment that would minimize the sum of squared differences between the two shifted images, but I modified the metric in a few ways. Instead of calculating the differences for all places between the shifted images, I only considered the places where pixels in both shifted images overlapped (rather than using circshift or padding the images with zeros for the calculation) so that the metric would be more accurate. To do this, I discarded the data that shifted off the edges of the red or green image, and shifted the blue image in the opposite direction so that both shifted images would have the same dimensions for the calculation. Also, since the number of pixels involved in calculating the metric affects the value of the sum, I decided to divide the sum by the number of pixels involved in the calculation. However, when shifting the images at the end so that they can be combined to form a final colorized image, I pad the images with zeros in the places where the images were shifted from.

To speed up the process of aligning large images, I implemented the image pyramid search procedure, in which I recursively halved the dimensions of a given image using bicubic scaling until the image width was below a certain size. This allowed me to to search a much narrower range of displacements for each image size, since at each level of the pyramid I was able to recursively use the estimated alignment from the half sized image as a central offset for the displacement window. The images you see in the next sections have been aligned using this coarse-to-fine image pyramid method. Also, I have added tweaks involving edge detection and cropping, which were used as well for generating these images. I'll explain these additional features in detail further down the page.


Colorization of Example .jpg Plates

Colorized Plate
Green displacement: x = 2 and y = 6
Red displacement: x = 1 and y = 10
Colorized Plate
Green displacement: x = 2 and y = 4
Red displacement: x = 2 and y = 9
Colorized Plate
Green displacement: x = 3 and y = 7
Red displacement: x = 4 and y = 14
Colorized Plate
Green displacement: x = 0 and y = 5
Red displacement: x = -2 and y = 11
Colorized Plate
Green displacement: x = 1 and y = -3
Red displacement: x = 1 and y = -4
Colorized Plate
Green displacement: x = 1 and y = 3
Red displacement: x = 1 and y = 12
Colorized Plate
Green displacement: x = 3 and y = 5
Red displacement: x = 4 and y = 11
Colorized Plate
Green displacement: x = 0 and y = 5
Red displacement: x = 0 and y = 11
Colorized Plate
Green displacement: x = -1 and y = 6
Red displacement: x = -3 and y = 12
Colorized Plate
Green displacement: x = 0 and y = 8
Red displacement: x = 0 and y = 13



Colorization of Example .tif Plates

These images have been scaled down and saved as .jpg images for the website, but if you click on the image it will take you to the full sized version of the colorized .tif

Colorized Plate
Green displacement: x = 8 and y = 40
Red displacement: x = 28 and y = 84
Colorized Plate
Green displacement: x = -4 and y = 15
Red displacement: x = -12 and y = 133
Colorized Plate
Green displacement: x = 9 and y = -9
Red displacement: x = 14 and y = 10
Colorized Plate
Green displacement: x = 24 and y = 26
Red displacement: x = 31 and y = 73
Colorized Plate
Green displacement: x = 9 and y = 57
Red displacement: x = 13 and y = 120
Colorized Plate
Green displacement: x = 36 and y = 68
Red displacement: x = 62 and y = 146



Colorization of Additional .jpg Plates

These were generated from additional sets of .jpg plates that I picked from the Library of Congress' collection of plates

Colorized Plate
Green displacement: x = 2 and y = 6
Red displacement: x = 4 and y = 13
Colorized Plate
Green displacement: x = 0 and y = 4
Red displacement: x = 0 and y = 10
Colorized Plate
Green displacement: x = 2 and y = 2
Red displacement: x = 3 and y = 6
Colorized Plate
Green displacement: x = 3 and y = 0
Red displacement: x = 3 and y = 2
Colorized Plate
Green displacement: x = 3 and y = 6
Red displacement: x = 3 and y = 13
Colorized Plate
Green displacement: x = 4 and y = 11
Red displacement: x = 5 and y = 23
Colorized Plate
Green displacement: x = 2 and y = 4
Red displacement: x = 1 and y = 10
Colorized Plate
Green displacement: x = 0 and y = 4
Red displacement: x = 0 and y = 11



Use of Edge Detection ("Bells and Whistles")

When I used the original image data without modification, some of the colorizations came out okay, but a lot of them were not aligned well at all. I discovered that if I used the Canny algorithm for edge detection on the blue, green, and red images, and then used only these edges to determine the offsets of the images, I was able to get much more accurate results. Since the 'edge' function outputs a binary image (as in, there is white where the edges are and black everywhere else), I had to convert it to double precision, since otherwise when I later scale the binary image down for the coarse-to-fine pyramid, the scaled down image would have to be binary too, so I wouldn't be able to take advantage of things like bicubic scaling. By calling im2double on the edge detected images, it allows the scaled down versions of the edges to have interpolated grayscale values, preserving the appearance of the edges.

Colorized Plate Without Edge Detection
Without Edge Detection
Colorized Plate With Edge Detection
With Edge Detection



Basic Cropping ("Bells and Whistles")

When the green and red images are shifted in order to align with the blue image, the original borders of the images won't be aligned when they are combined, causing distracting colors to appear near the border of the colorized image. I decided to implement rudimentary cropping based on the size of the colorized image. Since we were given a variety of jpeg plates to work with, I figured that the alignments in that sample would likely be similar to the alignments of other slides in the collection. I experimented with different amounts of cropping, and tried to find an amount that would remove the misaligned borders from most of the images without removing too much of the valid data near the borders. Since the .jpg and .tif files that contained the plates had white space around the joined plates, as well as black borders between the plates, I couldn't simply just crop the image based on the offsets that I found for the red and green components, so I went with this way instead. I also considered using edge detection somehow to find the almost horizontal and vertical edges that would indicate where the bad borders of the colorized image end, but I didn't have time to implement an algorithm to do that.

Colorized Plate Without Cropping
Without Cropping
Colorized Plate With Cropping
With Cropping