Edward Smongeski
15-463: Computational Photography
Project 6A
November 12, 2012

Background

This project uses homographies to warp images into panoramas, as well as rectifying images to be seen from different viewpoints artificially. The first step in the project was to take pictures of panoramas, which will be shown later. The second step was to recover homographies from two sets of points using least squares. This was achieved by setting up systems of equations in MATLAB, translated into a matrix equation, and solved with '\'. Control points were then selected by hand for images, and homographies were calculated. Each point in the warped image was then inverse warped, and the resulting point was interpolated in the original image to give a pixel value. Finally, the resulting panoramic images were stiched together using linear blending.

Part 1: Panorama stitching

Image Rectification

To test the homographies, images were rectified using only four points. This caused an artificial change in perspective.
curiosity, unwarped curiosity warped
The curiosity rover was warped in an attempt to give a view from above. It gives this illusion somewhat, but mostly reorients the top surface to be directed forward.
fuller tower, unwarped fuller tower, warped
The Fuller tower was warped to give a side view. The result is dramatic, but the side of the tower can be seen clearly.
milk crate, unwarped milk crate, warped
This milk crate was warped such that the frontmost face was square. Notice how it does not actually change the view such that the face is staring directly at us; the rest of the image is merely transformed; the view cannot be fully changed.

Panorama Mosaicing

Once the homographies were tested, panoramas were stitched together.
Carnegie Mellon University Center 1 Carnegie Mellon University Center 2 Carnegie Mellon University Center 3
Carnegie Mellon University Center Panorama
In this example, pictures of the CMU university center were stitched together. We notice that the people walking around are ghosted if they are too close to the seams at which the scene was stitched. This is because they are only in one of the stitching photos.
Architecture Stuff 1 Architecture Stuff 2 Architecture Stuff 3
Architecture Panorama
Someone was kind enough to leave some chunks of wood out by the tennis courts. These arrangements provided many distinct feature points across a broad range to select from. As well, Margaret Morrrison Hall is very detailed. There appears to be an artifact near the seam of the second and third images, although this may just be part of the wooden structure. In all seriousness though, I would like to thank the architecture students for their hard work, and for not littering the fence with any sorts of shantytown.
Oakland Skyline 1 Oakland Skyline 2 Oakland Skyline 3
Oakland Skyline Panorama
This example used pictures of the skyline over Forbes Ave. It was difficult to get some feature points because of the darkness of the image, but the panorama was still successful. There are some artifacts at the bottom; these are also because of the darkness of the image. The alpha calculator for the final image detected these as requiring transparancy. If you click on the image, you can see the version of the image without any alpha channel.
Camelback Mountain 1 Camelback Mountain 2 Camelback Mountain 3
Camelback Mountain 1
They say a picture says a thousand words. If you count the story behind this picture, it's even more. To take this panorama, I decided to fly out to Arizona for a weekend (not at all related to job searches). I took a trip to Camelback Mountain, hoping to climb it and take some nice panoramas from up high. Unfortunately, there was only one tiny parking lot at the foot of the mountain, and it was completely full. All of the side roads were littered with no parking signs (which can be seen in the panorama), but as I drove in and out of the parking lot over and over with no spots opening up, I became impatient, especially since I was flying back to Pittsburgh that afternoon. So, I decided to rebel. I drove down a side street, out of sight of the police officer who was more than likely enforcing the no parking signs, pulled over on the side (blinkers on and all), stepped out of my car, and started taking some photos. Just a few, I thought, I'll be out of here in 30 seconds. Unfortunately, 20 seconds into my 30 second escapade, I head a *clunk*, and turned around to see a tennis ball bouncing off of my car. It was at this point that I decided to stop taking pictures (unfortunately only on the third), and get back into my car, lest some angry dude start chasing me down with a 12 gauge. So I quickly drove away, and found other things to do around town. Luckily, the three pictures I took were enough to form a panorama. Enjoy!

Bells & Whistles

Using homographies, some other special effects were experimented with.

The Traveling CS Major

CS Major 1 CS Major 2 CS Major 3
CS Major Panorama
In this example, I experimented with the idea of a subject moving through a panorama. It was very difficult to define control points for this panorama, as the subject was very close to the camera. The mosaic documents the stages of computer science: thinking, moving a few feet to the computer, and writing code. Clearly there are some issues with the blending. This is mostly attributed to the poor placement of the subject in the original photos, so the blending regions tend to occur such that parts of the subject are cut off. That, or my roommate is the flash.

Chalking

Ground Captain Falcon
Chalking Chalking with no background
Here we see how homographies can be used to create chalk grafiti. By taking a blend of 50% of the warped image and 50% of the background, we obtain a chalky effect. We can obtain the first result by including all of the pixels in the warped image during the blending, or the second result by excluding all of the black pixels in the blending.

Movie Projection

Clint Eastwood Purnell Building
Movie Time!
Homographies can also be used to project a movie scene onto a wall by not blending the two images. Here we see that purnell is having a showing of Gran Torino. Clint Eastwood sure is an angry man.

Street Sign Changing

Speed Limit is 40 This guy is NOT Jackie Chan
The speed limit is NOT 40
We see how we can warp a face into a sign similarly to the movie example above. Without looking at the original, can you tell me what the speed limit is? I'll give you a hint: it's NOT Jackie Chan.

Part 2: Automatic Panorama Stitching

In this portion of the project, I used a series of methods to extract feature points from pairs of images, and then compute an accurate homography to transform one picture to the perspective of the other. The first method used to extract points was harris corner detection, which took out all corners detected in the image. The second method took the harris points, and narrowed them down using adaptive non-maximal supression, which restrained points based on their relative corner strengths and distances from eachother. These points then had feature descriptors associated them by sizing down 40x40 squares centered on each point to 8x8 squares. These descriptors were compared to find matches of points in the images; false positives were supressed by comparing the difference of the first match to that of the second match, and ignoring the match if these differences were too close. It was hypothesized that a value of 0.7 could be used for the cutoff ratio of the first difference to the second difference, but in practice it was found that cutting off at a value of 0.4 still provided a large pool of points to draw from, many of which were correct. Finally, the matches were used in a Random Sampling Consensus (RANSAC) algorithm to robustly determine a set of points that would provide an accurate homography. This homography was used the same as the homography in the first part of the project to stitch the images together in a panorama.

Harris Points

Harris pts 1 Harris pts 2
Here we see two examples for the harris points generated for images to be stiched together. There are a very large number of these points, and as such they are not useful as they currently are to compute a homography.

Adaptive Non-Maximal Suppression

Adaptive Non-Max Supression 1 adaptive Non-Max Supression 2
Here we see two images that have had their harris points suppressed using adaptive non-maximal suppression. This algorithm works by suppressing points based on their relative corner strengths and distances between eachother; higher distances are favored in order to provide an even distribution of points across the image.

Feature Matching

Matched 1 Matched 2
Here we see points that have been matched across the images. We see that the set of points has been cut down even further from the ANMS stage of the program. The matching was achieved by comparing feature descriptors of the points, and only taking descriptor matches that were significantly better than all other matches.

Random Sampling Consensus

ransac 1 ransac 2
The final stage of the homography calculation is Random Sampling Consensus, or RANSAC. The RANSAC algorithm randomly selects sets of four points from the matched feature sets, and computes a homography between them. It then checks how many points are in agreement when the first set of points is transformed to the second set using the calculated homography. The largest set of agreeing points is kept, and a final homography is calculated from this set.

Results

Here we compare the results of the autostitching process to the manually selected control points above.

Manual

Carnegie Mellon University Center Panorama

Automatic

Automatic UC panorama

Manual

Architecture Panorama

Automatic

Automatic Archie Panorama

Manual

Oakland Skyline Panorama

Automatic

Auto Oakland Skyline Panorama

Manual

Camelback Mountain

Automatic

Camelback Mountain Autopanorama

Manual

CS Major Panorama

Automatic

CS Major Autopanorama
This one was put into the autostitcher not expecting the greatest results; surprisingly, it looks better than the original, as it was very difficult to manually define control points in this panorama.