Project 2 for grad students - Eulerian Video Magnification

Yan Gu

Overview

In this project, I finished a MATLAB framework of the SIGGRAPH 2012 paper "Eulerian Video Magnification for Revealing Subtle Changes in the World" from MIT CSAIL. This project is named as Eulerian Video Magnification to reveal temporal image variations that tend to be difficult to visualize, that one can amplify blood flow and low-amplitude motion with Eulerian Magnification without needing to perform image segmentation or computing optical flow.

The algorithm implementation is described in the Framework part in detail.

Framework

The main process of the framework is:

1. Build the Laplacian pyramid;

2. Proceed a temporal filtering on each level (and maybe some) of the Laplacian pyramid;

3. Do pixel change magnification on the result given by the temporal filtering;

4. Reconstruct the image by changing the value of pixels on the Laplacian pyramid..

pseudocode for recursive main process:

function image = recur_call(a, framesNum, height, width, loFeq, hiFeq, samRate, cutoff, alpha, motion)

//framesNum: number of frame

//height & width: image size

//loFeq & hiFeq: the frequence range we want to manipulate

//samRate: sample rate

//cutoff: lambda_c mentioned in the artical

//alpla & motion: coefficient (number that) we want to amplified of color and motion

    if the level reach the threashold

        image = manipulate(a, frame, height, width, alpha, loFeq, hiFeq, samRate);

    else

        downSamplingImage = DownSampling(a);

        Get The Laplacian Pyramid Lap;

        if the level reach the minimum Feq require

            image = manipulate(Lap, frame, height, width, motion, loFeq, hiFeq, samRate);

        downSamplingImageRes = recur_call(downSamplingImage, FramesNum, height/2, width/2, loFeq, hiFeq, samRate, cutoff, alpha, motion);

        image = image + UpSampling(downSamplingImageRes);

In the Laplacian pyramid builing process, I used the function impyramid(), with , 'reduce' or 'expand' on the second parameter to get the down-sampling and up-sampling result. Then the Laplacian pyramid can be generated by using the difference between orginal image and the image taking both sampling process.

In the temporal filtering process, I tried several filters. The result is that the ideal window function get the best visual effort, even if side lobes will cause some minor artifact that leaking to other frequency. In order to save time, I use FFT to transform the vector in equal length. So the high cutoff frequency will be the "ceil( hiFeq * frame / samRate ) + 1" -th element of the frequency domain vector. Similarly, the low cutoff frequency is the "floor( loFeq * frame / samRate ) + 1" -th element of the frequency domain vector.

Pixel change magnification is very easy. I just get the filtered function from above, and then multiply it with the coefficient I gave, add it back to the orginal pixel value.

Reconstruction of the image is straight-forward: just use the new image value to reconstuct the pyramid and get the new value of the image.

Parameters:

cutoff: this is the correspondence parameter of lambda_c. However, this is an easier achievement: based on the paper, we just need to manipulate low frequency signals. This is the threashold of the number of layer on the pyramid that we start to do temporal filtering process.

alpla & motion: These are the coefficients that multiply to the filtered function. I gave two parameters in order to better control the effort. Alpha is worked on color amplify, and motion is worked on the Laplacian pyramid.

I have faced so many difficulties in this project. I appriciate Marynel a lot for her help.

Display

Basic blood flow amplified results:

result of "face.mp4"

Parameters: ( [loFeq, hiFeq, sameRate, cutoff, alpha, motion] )

[0.8 1 30 5 100 1]

 

result of "baby2.mp4"

Parameters:

[2 3 25 5 150 1]

 

"leg.mp4"

 

color amplified result of "leg.mp4"

Parameters: ( [loFeq, hiFeq, sameRate, cutoff, alpha, motion] )

[1.5 2.5 25 5 20 1]

 

motion amplified result of "leg.mp4"

Parameters: ( [loFeq, hiFeq, sameRate, cutoff, alpha, motion] )

[1.5 2.5 25 1 10 10]

Bells & Whistles

Extra slow-amplitude motion results (10pts):

slow-amplitude motion aplified result of "wrist.mp4"

Parameters:

[0.8 1.2 30 2 10 20]

 

Parameter testing (15pts):

Instead of giving two additional set of parameters on each of the processed videos, I tried many addtional sets of parameters to the sample image "face.mp4" to see what is the effort of each parameter.

result of "face.mp4"

Parameters:

[0.8 1 30 5 100 1]

 

If I add the loFeq and hiFeq to [2,3], which does not fix to the frequency with the blood flow of face, it will receive a ramdom jumping result on each spuerpixel, and they are not correspondence.

failure case of "face.mp4"

Parameters:

[2 3 30 5 100 1]

 

This is the result of motion amplified result using the parameters given in the paper.

motion amplified result of "face.mp4"

Parameters:

[0.8 1 30 2 1 20]

 

If I change the cutoff layer into 0, the details are much amplified, and cause minor artifacts. This proofs the reason that it is unnecessary to amplify those very detailed signals.

another motion amplified result of "face.mp4"

Parameters:

[0.8 1 30 0 1 20]

 

If I change the amplified coefficient into very large scale, it will cause clearly artifacts on the result.

failure case of motion amplified result of "face.mp4"

Parameters:

[0.8 1 30 0 1 100]