Greg Eden and Heather Strong
15-463 Fall 2008
Final Project: Painting Without Paint

Goal

Painting and photography are two very different art forms, but they share a common goal: to present a rendition of a scene. While photography is able to give a much more scientifically accurate representation, absolute accuracy is not always what is most desired. Our society cherishes the works of both bygone and modern-day painters, who, through their impressive abilities, are able to produce beautiful and memorable works of art. Our goal has been to examine various paintings and to use what we can learn about their styles in order to render a photograph that looks as though it had been painted by the same artist.

We believe that there are numerous uses for a tool that could do this automatically. The most evident use would be to allow someone with minimal artistic ability to take a picture of a memorable event or an attractive scene, and then produce a painting of the experience for display. Another possibility would be for a person who likes the scene depicted in a painting, but would like to view a similar scene in the same or a different style. A scene could also be rendered in several different painting styles in order to clearly illustrate the similarities and differences between the styles of two artists.

Methodology

In order to render a photo in the style of a given painting, we found it was necessary to examine and reproduce that painting's local properties (i.e. brush strokes) as well as its global properties (i.e. color palette, degree of realism, etc.). We chose to approach each of these aspects of imitation via the following algorithms:

  • In their paper, Image Analogies, Hertzmann et al. describe a way to use their algorithm (which, given an unfiltered image A, filtered image A', and unfiltered image B, will produce B' such that B' is to B as A' is to A) in order to learn how to perform various image filters, including artistic filters. Their algorithm operates at a very local level, examining blocks of five pixels at a time, which allows them to imitate the styles of paintings that are characterized by very local properties, such as visible brush strokes. However, their method uses only the colors already existing in the photograph. So even when imitating the works of the impressionists, for example, who are known for both their use of highly visible (almost obtrusive) brush strokes as well as very vibrant colors, the results from this algorithm alone may not be quite sufficient.

    While we found this algorithm to be very powerful, it employs the use of an algorithm for finding the Approximate Nearest Neighbor to a pixel. This is significantly faster than finding the nearest neighbor, but it is still a very slow process, and the processing time for a single photo increases exponentially with the size of the photo. So while our output images from this algorithm may seem somewhat small, they all took several hours to generate.

  • In Color Transfer in Correlated Color Space, Xiao and Ma explain an algorithm they developed for transferring the colors of a source image to a target image which depicts a similar scene. They achieve this by taking the 3D RGB color space of a first image and translating, rotating, and scaling it to line up as closely as possible with the color space of a second image. In cases where the two scenes are drastically different, this will often yield non-optimal results. (i.e. If the source scene contains a lot of sky but the target scene doesn't, the resulting image is likely to have a blue cast in non-sky areas.) In this case, they suggest implementing the transfer on different parts of the image separately, then blending the results together. They call this Swatch-Based Transfer. Their work is based off of Color Transfer between Images by Reinhard et al., who were able to attain similar results, but needed to perform computation in lαβ color space.

Results

By applying different combinations of Image Analogies and Color Transfer, we found we were able to imitate several different art styles with varying degrees of believability. In some cases, we found additional minor tweaks to our photographs helped to improve our output. Any such tweaks that we used have been documented below on a case-by-case basis.

Some of the photographs below were taken by us for use in other projects in this class. For all photos and art which we did not produce, clicking on a given image will take you to its original source. Some photos have also been shrunken so as to fit more nicely on the webpage.

Andy Warhol

Warhol's signature style is recognizable from its global properties, and so we found that it wasn't even necessary to run the Image Analogies code in order to imitate his work. On the other hand, Swatch-Based Color Transfer by itself was not quite sufficient. Heather's picture below has been contrast enhanced in Adobe Photoshop in order to get our rendering to more closely resemble how Warhol portrayed Marilyn Monroe. (See original photo here.) Still, the shading in our result image isn't quite drastic enough. Nevertheless, it is interesting to note that the target photograph still looks very natural, but running a simple Color Transfer with Warhol's image was able to yield Warhol-like shading (seen in the skin and hair) in the result image.

The viewer may be surprised to see areas of green and purple in the result image, when there was no green or purple perceivable in the source. The Color Transfer algorithm only attempts to shape the 3D color space of the target image to resemble that of the source image; that is, it cannot force the target image's color space to be identical to the source image's, and it has difficulty when executed on two drastically different scenes. Therefore, in such cases as these, where the algorithm is pushed to its limit, it is possible for the results to appear somewhat unexpected.

Source Image
Target Image
End Result


Edouard Manet vs. Claude Monet

We thought it would be interesting to see what differences cropped up in rendering a single photograph using the colors and painting styles of Manet and Monet, two French painters commonly confused with one another. We started with the image below, and ran Color Transfer then Image Analogies.

Color Transfer

We attempted to choose a Manet and a Monet painting with similar color composition to use in executing Color Transfer. Although the paintings below are very different, they were chosen because they both contain large amounts of blue and green, similar to the original image. The resulting images are not very surprising. Manet's contains lots of brown and is much more subdued in comparison with the vibrant Monet result. This is consistent with modern-day interpretation of these painters' works. The melancholy of Manet's style in contrast with the jubilance of Monet's style is often illustrated through the comparison of Manet's Rue Mosnier Decked with Flags vs. Monet's La Rue Montorgueil, both painted on a patriotic festival on June 30, 1878.

Again, the viewer may be surprised to see the yellow flowers turn blue as a result of the color transfers. As was previously noted, this is because the Color Transfer algorithm translates, rotates, and scales the target image's entire color space in order to line it up with the source image's color space as closely as possible. It makes no effort to try and preserve any colors in the target image.

Manet Source
Monet Source
Manet Result
Monet Result

Image Analogies

Hertzmann et al. provided a Manet analogy for use in their paper, which meant that we only needed to create a similar Monet analogy. By experimenting on the Manet A' image, we found that Adobe Photoshop's noise reduction filter set to maximal noise reduction and minimal detail preservation allowed us to most closely approximate the Manet A image. We then found a close-up of a Monet painting to use as our Monet A' image, and ran the noise reduction filter with those same parameters in order to obtain the Monet A image.

The output images are half the size of the original photograph. Each of them took about 5 hours to produce, so executing Image Analogies on the full size images (which have 4 times as many pixels) would have taken over 20 hours.

A
A'
A
A'
Manet Analogy
Monet Analogy
Manet End Result
Monet End Result


Raphael Sanzio

Next, we attempted to render a photograph in the style of a Renaissance artist. Since these artists focused on perfecting their technique in representing perspective, we thought it would be appropriate to use a scene Greg had photographed that had strong perspective lines and a clear vanishing point.

Color Transfer

For Color Transfer, we chose to use Raphael's School of Athens, which appeared to have a similar color composition to Greg's photo of Baker Hall. But even though the two do contain similar colors, Color Transfer yielded a much brighter, warmer looking version of the photo.

Source Image
Target Image
Color Transfer Result

Image Analogies

We found a close-up of Plato and Aristotle in the School of Athens to use in creating an Image Analogy. We took the A' image and ran Photoshop's noise reduction filter with the same parameters used to generate the Monet analogy above. Unfortunately, our result seems to be a bit more texturized than would have been desirable. Nevertheless, the algorithm successfully yielded an image that has a somewhat Renaissance-like feel to it. The output is very different from that of either of the impressionists.

A
A'
End Result


Henri Matisse

As a challenge, we decided to attempt to imitate a work of modern, abstract art. Since we weren't quite sure exactly how beautiful the result would turn out, we decided to use a photograph of someone we wouldn't feel bad about making look a little ridiculous...

Image Warping

As Matisse uses a very abstract style of painting, in hopes of better approximating his work, we started by first warping Bush to The Green Line's geometry using face warping code from project 3.

Image with Desired Geometry
Image to Warp
Result of Warping

Color Transfer

Next, we used Swatch-Based Color Transfer (with MANY swatches) to attempt to transfer Matisse's colors in The Green Line to the warped photo. In spite of repeated efforts, we could not get this to work well. This is one case in which the two images are simply too different for the transfer algorithm to be able to do an acceptable job. But although the resultant image does not look very Matisse-like, we did feel that it created an interesting effect.

Result of Color Transfer

Image Analogies

The Image Analogies were able to create a nice blotchy paint effect, most easily perceptible in the hair. However, it would have been nice if either this or Color Transfer had been able to create some heavy outlines in the image.

A
A'
End Result

Conclusions

The combination of Image Analogies and Color Transfer was able to yield many very interesting results, but the limitations of this approach are clear; it is very unlikely that any of our output images would actually be mistaken for a long-lost work by one of these artists. Based on our results, Image Analogies seems to be an accurate way of examining and reproducing an artist's style of brush strokes. On the other hand, this Color Transfer algorithm did not prove to be very well-suited for our general purposes. While it yielded good results for the impressionists and Raphael, it very quickly broke down when used in conjunction with less realistic looking works. We would like to see an algorithm that could perform well in both cases. Overall, we feel that further research is needed to find algorithms that can examine a painting and reproduce both its local and global properties more accurately.