Gradient Domain Fusion

Computational Photography: Project 2G

Matt Mukerjee

(mukerjee at cs)

Overview:

In this project I seek to realistic blend objects by means of solving a least squares equation for the optimal blended image. The constraints of this equation is that the blended image should have an extremely similar gradient to the two input images being blended, as this produces the best blended image according to human perception. However, this process could essentially change the absolute values for each input image (for example, a red hat may become green), but the image will still be easily recognizable (it will still be a hat), within the blended image. Least squares solutions must be used as the partial derivatives in x and y might not combine together to form an integratable surface. Thus, least squares can be used to find the closest integratable surface (in our case, an image).

We were tasked with three simple projects to explore gradient domain operations:
1. Solve a least squares equation that takes an input image and produces the same image as output by means of its gradients.
2. Blend images (a source image and a target image) using Poisson blending via solving a least squares equation. The blended output should use the gradient of the source image when available and the target image elsewhere.
3. Blend images (a source image and a target image) using Mixed Gradient blending via solving a least squares equation. The blended output should use the gradient with the highest magnitude throughout the image.

To phrase these questions as least squares equations, we impose the following constraints:

Problem 1:
    (a). the x-gradient of the output should closely match the x-gradient of the input
    (b). the y-gradient of the output should closely match the y-gradient of the input
    (c). the initial pixel of both images should be the same

Problem 2:
    (a). the x-gradient of the output should closely match the x-gradient of the source image (where available)
             (i). at the edge between the source and target, the "x-gradient" (between output and target) should match the x-gradient in the source
    (b). the y-gradient of the output should closely match the y-gradient of the source image (where available)
             (i). at the edge between the source and target, the "y-gradient" (between output and target) should match the y-gradient in the source
    (c). outside of source, the pixels should equal the target

(a). and (b). for problem 2 are summarized in this equation:

Problem 3:
    (a). the x-gradient of the output should closely match the x-gradient of the image with the highest gradient magnitude (where available)
             (i). at the edge between the source and target, the "x-gradient" (between output and target) should match the x-gradient of the image with the highest gradient magnitude
    (b). the y-gradient of the output should closely match the y-gradient of the image with the highest gradient magnitude (where available)
             (i). at the edge between the source and target, the "y-gradient" (between output and target) should match the y-gradient of the image with the highest gradient magnitude
    (c). outside of source, the pixels should equal the target

(a). and (b). for problem 3 are summarized in this equation:

Results:

Problem 1 (Toy Example):

===>

(minus the white boarder :) )

Discussion:
This shows that the least squares solution using gradients produces the appropriate results. Now let's move onto something more interesting.

Problem 2 (Poisson Blending):

Discussion:
In the first eye-catching example, you can clearly see that simply copying and pasting the source region to the target reason does not provide reasonable results. The background is simply not the right color, and thus the images don't blend well. Instead, by conforming the output image to the constrains imposed above, the output image now matches the target image instead of having that white background around the eye. This would most likely be because the gradient of the background in the eye image is very small, thus at the boundary between the two regions, in order for the gradient to be similar, all of the white in the background needs to be sky color due to the target image. The other images also look quite nice, except for image 3 (Super Mario on a bridge). Surprisingly, the simply copy/paste region version actually seems to be a nicer blend than the Poisson blend version. This is most likely due to the fact that Poisson blending preserves the gradient and not the absolute color values. Because of this, Poisson blends tend to be influenced by the background color information in the target image, which appears to be the case. Mario is very oddly translucent, especially in front of the water, giving him a ghost-like appearance.

Problem 3 (Mixed Gradient Blending):

Discussion:
For these examples, I employed Mixed Gradient blending following the least squares constraints as explained above. Mixed blending provides a nice way to remove flat-shaded regions in your source image, forcing them to take color information from the target image in their place. In the first image, its very easy to see the difference between a simple copy/past region solution and the Mixed Gradient blending approach. The results are quite nice, giving the source image a rather "graffiti" like quality. However, in the second image, we don't see as nice results. This would appear to be due to two concerns. First, it could be that the target area that's trying to be blended is black (with some weird reflections). Second (and much more likely) is that the area with high gradient magnitude in the source image is change from black to green (ie. the black outline of the ghost), and my assumption is that trying to blend a black outline to a black background is probably not going to work, leading to these mediocre results. The same situation is true for the third set of images.