15-463 Project 5a: Guided Panorama Stitching

When people come across beautiful landscapes, they often want to capture the moment in a photograph. Cameras lenses inherently have a limitation in their field of view. Circular fisheye lenses may be able to capture up to a 180 degree FOV, but why spend money on additional hardware when we can reconstruct a panoramic picture from low-FOV images? This project looks at taking several low-FOV images, warping them to a single image plane, and blending them into a single panoramic view.

;

Finding Homographies

If a set of images are taken from the same position, aperture and exposure, then it is possible to remap them all to the same projection plane. Mapping the images to the same projection plane will make it as if they had been captured in the same photograph. For pairs of images in the set, I indicated four corresponding points in the overlapping region between the pair. With these correspondences, we may choose one image as the real projection plane and warp each image to its plane.

A homography relates one projection plane to another. In particular, if is a homography from one projection plane to another, is a point in the source projection plane and is its corresponding location in the second projection plane, then . Note that and are parallel, so . Therefore we can construct a matrix such that where is simply reshaped as a vector.

Once we find the homographies, we may compose them to get from a point in one image's projection plane to any other. Note that coordinates must be renormalized when they are found by composing cordinates with homographies.

;

Warping

The warping step was fairly straightforward. The goal is to take a homography and an image and transform the image to the destination projection plane.

The first step was to create a vector of points that sample every pixel in the destination image. This was done by transforming the bounding box for the input image by the homograph and using meshgrid to make a vector that samples the resulting bounding box. The meshgrid was then transformed by the inverse homography to find the points to sample from the original image. The function interp2 was used to then sample the image.

Secondly, I used interp2 to sample the input photograph and output the destination image.

In the warping step, I also warped a white image in the same way. This created an alpha channel to return with the warped image since not all points in the new image correspond to valid areas in the old image.

OOM

My first panorama was of GHC from the Pausch Bridge. I took many pictures, which effectively spanned an almost 180 degree field of view. Projection planes approach infinite length as field of view approaches 180 degrees, which caused my warped target image to be very large and MATLAB to run out of memory. I therefore discarded the fourth image from the panorama shot as it made reconstructing the panorama impossible and it did not add much information relative to the area it consumed. I actually attempted to reconstruct the panorama with this large fourth image in GIMP to ensure my code was correct, which is how I realized that the last photo should be discarded. For the last two panoramas, I paid more attention to the field of view I was sampling to avoid this problem. When stitching my final panorama, I hit the OOM error again and resolved it by cropping two of the mosiac images.

;

Blending: Averaging

To blend the images together, I used a simple averaging technique. At each pixel, I added the value of that from each warped image weighted by its alpha channel. In addition, I kept an accumulator for the alpha at each pixel. Finally, each pixel is divided by the alpha accumulated at that pixel to obtain the weighted average.

There were some issues with the results when I finished blending. Ghosting was a huge issue because of human error. First, I did not use a tripod for taking pictures. This results in the camera position changing slightly with each shot and making the homographies invalid. Some of the images were also a little blurry, which made it difficult to select control points. Sloppy control point selection resulted in homographies which were slightly off, causing a double-image effect in the final output.

;

Results (Part 1)

I took three panoramas by projecting sets of images to the same plane. The subjects were Gates-Hillman Center, some ceramic farm animals, and a subway scene from Battlefield 3.

Bridge

These photographs were taken from the Pausch Bridge facing Cyert. The Gates-Hillman Center is on the left. The AWB of my phone makes taking panoramas outside difficult.

Ceramic Farm

I bought my roommate some ceramic farm animals for Christmas last year. I thought that they would be a good subject to demonstrate how difficult control points are to define by hand.

Battlefield 3

Since we were required to take a panorama off-campus, I decided to build one from a virtual world. I chose the Operation Metro map from Battlefield 3. These pictures were very clean and easy to define control points for. The main issue with this panorama is that I could not remove the heads-up display or the arms/weapon mesh. This causes a lot of ghosting in many of the images.

;

15-463 Project 5b: Automatic Panorama Stitching

The second half of this project is to remove user interaction by automatically finding homographies between images.

;

Intermediate Steps

Automatically stitching panoramas is composed of five main parts: finding points of interest, suppressing non-maximal interest points, matching features, removing false positive matches, and warping/stitching the images. Considering the ceramic farm example above, I computed Harris interest points, suppressed non-maximal points below a certain suppression radius, and extracted and matched feature descriptors. The following is an image of the left two farm images with their Harris interest points (red), unsuppressed points (blue), and matched features with (yellow).

Each of the components appear to be working very well, except the feature matching. I tried to play with the threshold for the Lowe cutoff, but it returns nearly as many false positives as true positives. The RANSAC algorithm randomly selects points and sees how many of the matches correspond to those four matches. After a few iterations, the largest set is considered to be the set of true positives and a homography with the least L2 norm error is considered to be the true homography. The following image show the control points selected by the feature matching function. Take note of the incredibly large number of false positives.

Once RANSAC has run, a subset of true positives is returned with the homography. The image below shows the set of true positives extracted from this example.

;

Results (Part 2)

The following are my results when attempting to stitch with automatic homography finding.

Bridge

Ceramic Farm

Battlefield 3

These images turned out to be particularly problematic. The gun model and HUD provided too many positive matches, thus creating an incorrect homography. As a result, I attempted to black out the gun models and automatically stitch the panorama with the edited photos.

;