Assignment: Seam Carving

Ronit Slyper

Objective

I implemented the basic idea of "Seam Carving for Content-Aware Image Resizing" by Shai Avidan and Ariel Shamir.
The code to do this consisted of three files:

energy_function.m computes an energy function on an image. Separating this function out let me try different functions easily.
seams.m computes the seams and stores them in a matrix
seams_call.m removes the matrix of seams from the image

Energy Function

I began with a gradient computed using nlfilter; this tack was way too slow.
I tried a laplacian, gaussian, and unsharp filters on both the images and energy, but none proved better that the gradient: the gaussian removed high frequencies, so the result wasn't useful; the laplacian and unsharp emphasized edges too much.
I also tried computing the gradient, then eroding the result to widen the ``don't go here'' borders between objects, in hopes that this would prevent the failure case where seams would break the structure of objects. Artifacts of the strel shape were too great, however, and I did not want to blur the gradient.
In a similar vein, I used a canny edge detector on the image, imdilated the resulting edges, and added it to the gradient energy-matrix. The hope was this would prevent edge crossings by making edges unappealing, and thus prevent distortion of structures. Results were

and

.
Why so bad? The cause is an imperfection in the scanned image that results in many of the seams heading for the middle, which has a zero gradient patch:

The next image worked better:

It had no such imperfections, so the edge detection helped more. Using a simple gradient energy function:

,
and a gradient with enhanced edges:

The suits of the men show less distortion.

In all following images I used a gradient computed via sums of convolutions.

GUI

I learned matlab's gui-creation toolkit (GUIDE); it is similar to Visual Basic and thus easy to get a grasp of. Getting the axes object to resize with the image was a long and mentally anguishing trip. Here is the GUI in action:


Viewing the energy function	Viewing the image	Viewing a larger image.	Object selection for removal

two sliders for changing the image size. Seams are computed in one direction (horizontal or vertical), and the image/energy-matrix gets resized by the user. When the user switches direction, seams get recomputed on the resized energy matrix. The limitation of this is that you cannot, e.g., shrink v seams, shrink h seams, then enlarge v seams back to the original width.
load and save buttons
a checkbox to see the resized energy picture instead of the actual picture (very useful)
in-house object selection for removal using the roipoly function. The mask returned by roipoly is added into the energy matrix and the seams recomputed.
By uncommenting a few lines in seams_call.m, you can press spacebar many times to see every single seam superimposed on the image as it's removed.

Bells & Whistles

Make it fast; Compose an image and compare to seam carving; Remove objects (using seam carving, without seam insertion); Matlab GUI for viewing and object removal.

Results

I resized images in both directions, sequentially. (However, if an image was an obvious failure in the first direction, I often didn't take the time to continue resizing in the other direction). All images are shown at actual size (no standard scaling of results!), with the original on the left and resized version on the right.

Failures


Maurice Denis, MoMA.When important objects are smooth, they end up looking rather emaciated.

flickr + matlab fun. This picture is both a success and a failure. The seam carving is flawless; however, the gradient fails to reflect a human notion of importance, and thus large stretches of high-frequency beach are preserved.

flickrfun. The duck is very smooth compared to the high-frequency water, and thus ends up doing a strange dance move.

MC Escher. Seams pass through lines, removing structure.

Felix del Marle, MoMA. This image is both a success and a failure. It has the same problem as the previous Escher work. However, new art is created!.

George Grosz, MoMA. I was surprised this came out so bad, with the same problem as above. The smooth regions are still noisy, colorwise, so seams pass through edges.

flickr. High frequency background means the lady's head is seam carved.

Leopold Survate, MoMA. Too noisy.

Lyubov Popova, MoMA. So much for being clever.

Successes


Kupka, MoMA. Great picture for a gradient energy function.

MC Escher. Self similarity! A great place for scene carving. The results look suspiciously like scaling, though.

MC Escher. Large monochrome blocks are easy to carve.

MC Escher. This distorts the structure somewhat, but you don't notice it at first, since Escher paintings are disorienting anyway.

Matisse, MoMA. Weight loss via seam carving.

Van Gogh, MoMA. Landscaping via seam carving.

Object Removal

It is difficult to find images where object removal will work well.


MC Escher. I roughly painted out one of the fish, resized the image, then refined it by painting out the tail in the bottom, and resizing again.

self. I roughly delineated the man blocking my view, and seam-carved him out in both directions.

self. At left, I perform the same task in the Gimp, using the clone brush. At right, I duplicate one of the cuter occupants to replace the removed guy. I found that attention to the shadows results in an astounding leap in the images' credibility.