Computational Photography Final Project

By: Timothy Farkas

Project Description

In my project I will be synthesizing textures by using methods similar to the technique outlined by Heeger and Bergen. The Heeger and Bergen algorithm synthesises textures by contructing a steerable pyramid. A steerable pyramid is similar to a laplacian and gaussian pyramid with the exception that the frequency band at each level is split into 4 orientations by using oriented filters similar to gabor filters. A texture is synthesized by constructing the steerable pyramid for an exmplar texture, then matching the histograms of the exemplar texture at each level and orientation to the histograms of random noise. Link to paper here.

Texture Synthesis Algorithm 1

The first algorithm I implemented was a simplified version of the Heeger and Bergen algorithm. This algoirthm did not take into account different orientations in the texture and simply matched the histograms at each level of the laplacian pyramid.

Results

original	synthetic
original	synthetic
original	synthetic
original	synthetic
original	synthetic
original	synthetic

Weaknesses

This method does not account for different orientations in a texture, so it produces anisotropic textures.

Texture Synthesis Algorithm 2

This algorithm is the Heeger and Berger algorithm, which matches the histograms of different orientation bands.

Results

original	synthetic
original	synthetic
original	synthetic
original	synthetic
original	synthetic
original	synthetic

Weaknesses

It only accounts for 4 orientations. And you can only do 5 or so iterations of histogram matching due to aliasing.

Texture Synthesis Algorithm 3

This algorithm is similar to the Heeger and Berger algorithm except that it does not select frequency bands by convolving with filters, instead frequency bands are selected directly in the frequency domain. This is done by taking the fourier transform of an examplar texture, zeroing out all the frequencies you don't want, and then taking the inverse fourier transform. This gives you greater control over the frequencies you want, and also allows you to completely avoid aliasing if you decide to decimate the image. In addition to directly selecting frequencies in the fourier domain, way more than 4 orientations were used.

Examples of selected orientation bands.

After selecting a frequency band in the fourier domain, the histograms are iteratively matched as usual.

Results

original	algorithm 3	algorithm 2 (Heeger, Berger)
original	algorithm 3	algorithm 2 (Heeger, Berger)
original	algorithm 3	algorithm 2 (Heeger, Berger)
original	algorithm 3	algorithm 2 (Heeger, Berger)
original	algorithm 3	algorithm 2 (Heeger, Berger)
original	algorithm 3	algorithm 2 (Heeger, Berger)

Weaknesses

This didn't work as well as I expected and seems to create more sinisoidal artifacts than algorithm 2 (Heeger and Berger), especially on the last image above.

I thought this might be due frequencies leaking when the histograms are matched for a particular frequency band. This problem is illustrated in the following figures.

Noisy image's frequency band before histogram matching.

Source image's frequency band.

Noisy image's frequency band after histogram matching.

As can be seen above, matching the histograms for a frequency band introduces other frequencies.

Texture Synthesis Algorithm 4

I tried to fix the problem described with algorithm 3 by simply iterating the histogram matching for the frequency band until the energy in frequencies outside of the frequency band is negligible. Then the frequencies outside of the frequency band are thrown away.

The following figures illustrate the process.

Noisy image's frequency band after 1 histogram matching iteration.

Noisy image's frequency band after 10 histogram matching iterations.

Noisy image's frequency band after throwing away undesired frequencies.

Results

original	algorithm 4	algorithm 3
original	algorithm 4	algorithm 3
original	algorithm 4	algorithm 3
original	algorithm 4	algorithm 3
original	algorithm 4	algorithm 3

Weaknesses

Removing the spectral leakage didn't really improve the results that much. But then I noticed that matching the histograms for a frequency band did not shape the fourier of the synthetic texture to look like the fourier transform of the the examplar texture.

FFT of synthetic texture.

FFT of original texture.

You can see in the figures above that the energy appears to be distributed uniformly within each rectangular frequency band in the synthetic texture.

Texture Synthesis Algorithm 5

I tried to fix the problem described with algorithm 4 by matching the histograms of the magnitudes of frequencies within each frequency band as well as the histograms of the frequency bands in the spatial domain. However, matching the histograms of the magnitudes of frequencies did not make the FFT of the synthetic texture look like the FFT of the exemplar texture.

FFT of synthetic texture.

FFT of original texture.

Results

No improvement.

original	algorithm 5	algorithm 4
original	algorithm 5	algorithm 4

Texture Synthesis Algorithm 6

I tried to make the fourier transform of the synthetic texture look more like the fourier transform of the exemplar texture by multiplying the fourier transform of the initial noise by the magnitude of the the fourier transform of the exemplar texture. Then histograms are matched as usual.

FFT of noise.

FFT of original texture.

Result of multiplying the two.

Results

original	algorithm 6	algorithm 4
original	algorithm 6	algorithm 4
original	algorithm 6	algorithm 4

Strengths

The results are slightly better.

Texture Synthesis Algorithm 7

I tried to see if algorithm 2 (Heeger and Berger) could be improved by multiplying the fourier transform of the initial noise by the magnitide of the fourier transform tof the exemplar texture.

original	algorithm 7	algorithm 2 (Heeger and Berger)
original	algorithm 7	algorithm 2 (Heeger and Berger)
original	algorithm 7	algorithm 2 (Heeger and Berger)
original	algorithm 7	algorithm 2 (Heeger and Berger)
original	algorithm 7	algorithm 2 (Heeger and Berger)

The results are slightly better.

Conclusion

Matching histograms of different frequency bands is not enough to create a good complex textures. This is because there are many spatial arrangements of pixels that satisfy the histogram constraint for a particular frequency band. Additionally, matching histograms in the spatial domain for a particular frequency band does significantly change the distribution of energy within that frequency band. So if energy is evenly distributed in a frequency band before histograms are matched, it will be uniformly distributed after histograms are matched. So, assuming that fourier transforms of similar textures are similar, histogram matching in the spatial domain is not enough.