Hugh Cover – September 2011

The aim of this project is to add an object from one image into a scene in a second image. If we were to cut out the object and paste it directly into the scene it would not look correct, especially around the borders of the cut. A few techniques to fix this issue are explored here.

Toy Problem

This toy problem is a demonstration that you can recreate an entire image with only a single pixel’s intensity and the original gradient information. A linear system is created that says the 2D gradient of the output image at each pixel should be equal to the gradient of the input image at the same pixel. A final constraint is added that says the top left pixel intensity should be equal in the input and output images. By finding a least squares solution to the linear system we generate the output image, which in this case should be identical to the input.

The left image is the original input image, the right is the output. The error recoded was zero (if Matlab’s small imprecision error is ignored) indicating the image was perfectly recreated.

Poisson Blending

The idea of Poisson blending is to cut an object out of a “source” image and paste it into a “scene” image while making it look like it belongs. It is related to the toy problem above in that it uses gradient information and some initial intensity information to generate the final result. A linear system is created that says the 2D gradient at each pixel of the patch being inserted should be equal to the gradient of the equivalent pixel in the source image. At the edges of the patch the gradient will include a pixel from the scene image which has a fixed intensity/colour, this gives the initial intensity information used to rebuild a patch that will fit in well with the scene. The least squares solution to the linear system is found giving an object patch that hopefully looks like it belongs in the scene.

Backyard Invasion

The source and target images

The object patch naively pasted into the scene

Poisson blending allows for a smooth backyard invasion.

Kangaroos* in America

*It’s actually a Wallaby

Source, target and naive combination

After blending the Aussies are truly invading

Failure Case

The left image is the source patch pasted onto the target image; the right image is after Poisson blending. This case has failed because the background of the source image does not match closely enough with the target image. The Poisson method can’t deal with the matching of complex backgrounds.

Mixed Gradients

This is a variation on the Poisson blending method. Instead of trying to make the image patch have the same gradient as the source image the maximum of the source and target gradient at each pixel is used. Basically the gradient of the most interesting image at each pixel location is used. This is particularly good if the background of the source image is blank, i.e. a single colour, as it will take the gradient of the scene for the blank part and hopefully the gradient of the object in the actual object.

UFO over Sydney

Source, target and naive combination

Poisson blend with mixed gradient

A Special Case

When trying to put an alien on the moon during one of the Apollo missions I first tried vanilla Poisson blending which didn’t work well because the alien’s background has no gradient and so just stretches the pixel intensities at the boundary over the black section.

I then tried mixed gradient which correctly handled the object background but also put the moon’s texture on the alien. This is because the moon surface has lots happening and therefore high gradients, where as the alien is very smooth.

To get a good result I used the moon’s gradient only when the object patch gradient is zero. This ensures that the moon gradient is used for the black background sections but not in the alien itself.

The final result is a friendly face to welcome the Apollo astronauts.