Programming Project #3
Face Morphing
Author: Josip Djolonga

Introduction

The purpose of this project was to implement a face morphing algorithm. That is, given two faces and control points we had to write an algorithm that would continuously morph one face into the other. The immediate faces that were obtained can be used to create a video. On the right you can see a video of how two faces are continuously morphed.

General Overview

First, a set of corresponding points have to be specified on both pictures. To make the transition look more natural, is helpful if these points coincide with some characteristics of the face like: eyes, mouth ends, eyebrows etc. We created a data set with pictures from every student in the class and corresponding labels. For consistency, we labeled them following this template:


Then, using this data we can compute the mean face and use this in various ways to caricaturize the faces, as described below.

Algorithms

Face Morphing

Triangulation

First we create a Delaunay triangulation on the set of control points. There are many choices on which points to compute the triangulation, I chose to compute it on the average of the two control points sets.

Morphing

One parameter t from [0,1] is used to control the morphing. Basically, the final structure to which we morph is a linear interpolation F = (1-t)A + tB where A and B denote the control points of the first and second picture respectively. Note that the same triangulation is used for F, A and B. Now, we want to map each triangle in A and B to the corresponding triangle in F. For every par of non-degenerate triangles there exists an invertible affine mapping that maps one triangle to the other. These mappings are easy to compute as the conditions result in a linear system of equations. Now, we go over each pixel in the resulting picture (procedure also known as inverse warping), we see which pixel in A and B map to this pixel (or interpolate if the inverse is not on the pixel grid) and cross-dissolve these two intensities. The cross-dissolve is regulated by another parameter w from [0,1]. In the following sections, we usually use the same value for both t and w.

Movie Generation

Once we have the morphing algorithm, creating the video is trivial. While we increase the parameter t from 0 to 1 by uniform discrete steps, we also cross dissolve the pictures by the same value w=t.

Mean Face

Given all the faces from the students, we first take the mean of the control points C. Then they are all morphed towards C and the average over these morphs is taken. This produces the mean face, and this is the result of the dataset with the students.

Bells & Whistles

Using the Mean Face for Fun and Profit

Now that we have computed the mean control points and faces (the colors) which was shown above, what we can do is to morph to face towards the mean face. These are some results.

Original face	Completely warped to the mean face
	Completely warped from the mean face

Original face	Partially warped with the mean face

Also, we can compute the mean male and female control points and faces. Using these, we can create masculinizing and feminizing transforms. To masculinize a face, we morph it towards the mean male control points. Also, using the mean male face we can change the colors so that it would appear more masculine. Similarly, we can feminize faces. These are the mean male and female faces.

Mean male face	Mean female face

Note that the female picture is not as smooth, and this is because only 6 pictures were used. All these pictures were from different datasets, so there is a lot of variability between them. They were all resized to a common size. Here are some interesting results when we use these ideas.

Original face	Feminized

Original face	Feminized

Original face	Feminized

Original face	Masculinized

Original face	Masculinized

Original face	Masculinized

PCA Basis

I have computed the PCA basis for the male and female pictures. These were computed only on the red channel, and the pictures were resized to 104x125 (my computer couldn't handle the original sized pictures). Unfortunately, as there were not many pictures in the dataset, I do not think that the first several pictures represent the essence of the male and female faces. These are the computed basis images:

Principal components for the male faces. Leading one is shown first.

Principal components for the female faces. Leading one is shown first.

I have tried adding the first few principal components to a face to make it look more feminine/masculine, but the results were not satisfactory. An example of this is the following picture: