15-463: Computational Photography

Project 4: Mosaic

Image rectification

Original Rectified Notes

This was a test image to ensure my code worked.

The ability to unskew pictures taken of papers, books, covers, etc demonstrates a possible application of image rectification.

I attempted to rectify to face the windows in the building on the left hand side.

This was a picture my friend Jason Lee took while in Lancaster, PA.

Image mosaicing

Overview

Using manually defined correspondence points, I recover a homography between two pairs of adjacent images. I warp the left and right photos using their respective homographies to the middle image. I use the minimum and maximum values of the four corners to determine the size of the new warped image. Then I use normalized cross correlation to determine the offset of the left and right image to the middle image, and then I offset each image accordingly.

I also implement blending via multiscale Laplacian Gaussian pyramid. It didn't help out that much, especially on the first mosiac, but it was better than alpha blending.

Standing atop the shoulders of English majors

This was shot from the bridge between Wean and the CIC, around where the English PhD students used to have their cubes. The overexposure on the left scene led to a visible seam (even with blending) near the roof on the right side. Additionally, the warp did not line up the roof exactly, possibly because I didn't have any correspondence points over on the roof. Picking correspondence points on the roof was difficult due to lack of corners. Nevertheless, I think the effect of the sun is pretty awesome. (Originals: 1 2 3)

It's November and there are still green leaves on the trees?!

This was shot from the Randy Pausch bridge towards Doherty. I can't find any stitching seams here, possibly because light levels were pretty constant when I shot these pictures. (Originals: 1 2 3)

In honor of the Donner Ditch, this area shall henceforth be known as the Gates Ditch

This was shot from the "nature preserve" at the stairs that lead from the Cut to the "ditch" between Doherty and Gates. On closer examination, I seemed to have shaken the camera on the right-side image, so the Pausch bridge area is all blurry. I think the blending made the transition from the blurry image to the sharper middle image seem seamless. (Originals: 1 2 3)

Fun fact: I stitched these three mosaics at the same time on three instances of MATLAB on unix13 (8 CPU, 16 GB). I managed to use about 97% of available memory and up to 600% CPU. Totally awesome. The mosaic of Gates took the bulk of time and peaked at 60% of available memory used. I believe this was because of the massive warp on the right side image. That image, when warped, was significantly larger than all the other warped images.

Double fun fact: The Rally to Restore Sanity restored my sanity, but this project destroyed it. Wait. Wait. This was only Part 1?! Noooooooooooooooooooooooooooooo.....

Part 2

Autostitching

In this part, we had to implement feature matching to automatically generate correspondence points for a mosaic. We use the process described in "Multi-Image Matching using Multi-Scale Oriented Patches" by Brown et al. First we use the Harris method to detect corner points. This generates a lot of points (>100,000), so first I filter out corner points with strength less than 0.0001. This typically cuts the number down to about 20,000 points. At this point, it becomes computationally feasible to run adaptive non-maximal suppression (ANMS). (Before filtering, ANMS took about 30 minutes, but after filtering, it ran in minutes.) ANMS selects 500 points of maximal corner strength but spread out over the entire image. Then I extract a descriptor for each point.

Between each two pair of images that needs to be stitched together, I compare each descriptor of the first image with each descriptor of the second image. For each descriptor of the first image, I find the corresponding descriptor in the second image that minimizes the SSD distance. If this distance was less than 0.7, then I kept the pair. There were about 100 points at this point. Then I ran the set of collected pairs through 1000 iterations of RANSAC and filtered out all but about 15 pairs. In each iteration of RANSAC, I select four random pairs of points, calculate a homography transformation between these pairs of points, then apply the homography to the other points in the first image and see how they compare to points in the second image (by SSD). For each pair of points with SSD under 1 is kept as an inlier. The largest group of inliers is kept as the final pair of points.

Here are the results.

Standing atop the shoulders of English majors

Left Middle Right
Warp left to middle Warp right to middle

It's November and there are still green leaves on the trees?!

Left Middle Right
Warp left to middle Warp right to middle

In honor of the Donner Ditch, this area shall henceforth be known as the Gates Ditch

Left Middle Right
Warp left to middle Warp right to middle

Bells and Whistles

Multiscale Laplacian Blending

Basically, at each level of the pyramid, high frequencies have been filtered out. Blending is done along these levels and then merged for the final image.

Look in part 1 for my description of the effects.

Revenge of the Nerds

I decided to have a bit of Carnegie Mellon pride and virtually deface the University of Pittsburgh's Cathedral of Learning.

I defined correspondence points on the corners of the image of the Carnegie Mellon logo to a nice location on the Cathedral of Learning. Then I warp the logo to that orientation. To do the blend, I do two-level Laplacian blending with a mask over the letters of the logo. This blurs the edges of the letters with the stone walls for a nice effect. Additionally, I used a mask with a lesser value (0.7 instead of 1) to allow a little bit of the wall to shine through. This way it looks like the logo was spray painted on.

Source images: Cathedral of Learning Carnegie Mellon