A study on face morphing algorithms

FROM：https://ccrma.stanford.edu/~jacobliu/368Report/index.html

Abstract

    An automatic face morphing algorithm is proposed. The algorithm automatically extracts feature points on the face, and basedon these feature points images are partitioned and face morphing is performed. The algorithm has been used to generate morphing between images of facesof different people as well as between different images of the face ofan individual.  The results of both inter- and intra-personal morphingare subjectively satisfactory.

---

I.Introduction

    Morphing applications are everywhere. Hollywood filmmakers use novel morphing technologies to generate special effects, andDisney uses morphing to speed up the production of cartoons. Among so manymorphing applications, we are specifically interested in face morphingbecause we believe face morphing should have much more important applicationsthan other classes of morphing.

    To do face morphing, feature points are usually specifiedmanually in animation industries [2].  To alleviate the demand forhuman power, M. Biesel [1] proposed an algorithm within Bayesian frameworkto do automatic face morphing.  However, his approach involved computationof 3N dimensional probability density function, N being the number of pixelsof the image, and we thought the approach was too much computation-demanding.

    Therefore, we would like to investigate how featurefinding algorithms can help us achieve automatic face morphing.  Withinthe scope of this project, we built up a prototypical automatic animationgenerator that can take an arbitrary pair of facial images and generatemorphing between them.

Back to top

---

II. Algorithms

Outline of our Procedures

    Our algorithm consists of a feature finder and aface morpher. The following figure illustrates our procedures.
 

The details for the implementations will be discussed in the followingparagraphs.

Pre-Processing

    When getting an image containing human faces, itis always better to do some pre-processing such like removing the noisybackgrounds, clipping to get a proper facial image, and scaling the imageto a reasonable size.  So far we have been doing the pre-processingby hand because we would otherwise need to implement a face-finding algorithm. Due to time-limitation, we did not study automatic face finder.

Feature Finding
    Our goal was to find 4 major feature points, namelythe two eyes, and the two end-points of the mouth.  Within the scopeof this project, we developed an eye-finding algorithm that successfullydetect eyes at 84% rate.  Based on eye-finding result, we can thenfind the mouth and hence the end-points of it by heuristic approach.
 
1. Eye-finding
    The figure below illustrates our eye-finding algorithm. We assume that the eyes are more complicated than other parts of the face. Therefore, we first compute the complexity map of the facial image by slidinga fixed-size frame and measuring the complexity within the frame in a "totalvariation" sense.  Total variation is defined as the sum of differenceof the intensity of each pair of adjacent pixels.  Then, we multiplythe complexity map by a weighting function that is set a priori. The weighting function specifies how likely we can find eyes on the faceif we don't have any prior information about it.  Afterwards, we findthe three highest peaks in the weighted complexity map, and then we decidewhich two of the three peaks, which are our candidates of eyes, reallycorrespond to the eyes.  The decision is based on the similarity betweeneach pair of the candidates, and based on the location where these candidatesturn out to be.  The similarity is measured in the correlation-coefficientsense, instead of the area inner-product sense, in order to eliminate thecontribution from variation in illumination.

2. Mouth-finding
    After finding the eyes, we can specify the mouthas the red-most region below the eyes.  The red-ness function is givenby
 

Redness = ( R > G * 1.2 ? ) * ( R > R_th ? ) *  { R/ (G + epsilon ) }
 

 
    where R_th is a threshold, and epsilonis a small number for avoiding division by zero.  Likewise, we can define the green-ness and blue-ness functions.  The followingfigure illustrate our red-ness, green-ness, and blue-ness functions. Note that the mouth has relatively high red-ness and low green-ness comparingto the surrounding skin.  Therefore, we believe that using simplesegmentation or edge detection techniques we would be able to implementan algorithm to find the mouth and hence its end points automatically,if time permitting.

 

Image Partitioning

    Our feature finder can give us the positions of theeyes and the ending points of the mouth, so we get 4 feature points. Besidethese facial features, the edges of the face also need to be carefullyconsidered in the morphing algorithm.  If the face edges do not matchwell in the morphing process, the morphed image will look strange on theface edges. We generate 6 more feature points around the face edge, whichare the intersection points of the extension line of the first 4 facialfeature points with the face edges.  Hence, totally we have 10 featurepoints for each face.  In the following figure, the white dots correspondto the feature points.

 

    Based on these 10 feature points, our face-morpherpartitions each photo into 16 non-overlapping triangular or quadrangularregions.  The partition is illustrated in the following two pictures. Ideally, if we could detect more feature points automatically, we wouldbe able to partitioned the image into finer meshes, and the morphing resultwould have been even better.

           Image 1                                Image 2
__________
 

    Since the feature points of images 1 and 2 are, generallyspeaking, at different positions, when doing morphing between images, theimages have to be warped such that their feature points are matched. Otherwise, the morphed image will have four eyes, two mouths, and so forth.It will be very strange and unpleasant that way.
    Suppose we would like to make an intermediate imagebetween images 1 and 2, and the weightings for images 1 and 2 are alphaand (1-alpha), respectively. For a feature point A in image 1, and thecorresponding feature point B in image 2, we are using linear interpolationto generate the position of the new feature point F:

    The new feature point F is used to construct a pointset which partitions the image in another way different from images 1 and2. Images 1 and 2 are warped such that their feature points are moved tothe same new feature points, and thus their feature points are matched. In the warping process, coordinate transformations are performed for eachof the 16 regions respectively.
 

Coordinate Transformations

    There exist many coordinate transformations for themapping between two triangles or between two quadrangles. We used affineand bilinear transformations for the triangles and quadrangles, respectively. Besides, bilinear interpolation is performed in pixel sense.

1. Affine Transformation
 

    Suppose we have two triangles ABC and DEF. An affinetransformation is a linear mapping from one triangle to another. For every pixel p within triangle ABC, assume the position of p is a linearcombination of A, B, and C vectors.  The transformation is given bythe following equations,

    Here, there are two unknowns, Lambda1 and Lambda2,and two equations for each of the two dimensions.  Consequently, Lambda1and Lambda2 can be solved, and they are used to obtain q.  I.e., theaffine transformation is a one-to-one mapping between two triangles.

2. Bilinear Transformation

 

    Suppose we have two quadrangles ABCD and EFGH. TheBilinear transformation is a mapping from one quadrangle to another. For every pixel p within quadrangle ABCD, assume that the position of pis a linear combination of vectors A, B, C, and D.  Bilinear transformationis given by the following equations,
 

    There are two unknowns u and v. Because this is a2D problem, we have 2 equations. So, u and v can be solved, and they areused  to obtain q.  Again, the Bilinear transformation is a one-to-onemapping for two quadrangles.
 

Cross-Dissolving

    After performing coordinate transformations for eachof the two facial images, the feature points of these images are matched.i.e., the left eye in one image will be at the same position as the lefteye in the other image. To complete face morphing, we need to do cross-dissolvingas the coordinate transforms are taking place. Cross-dissolving is describedby the following equation,

 

where A,B are the pair of images, and C is the morphing result.

    This operation is performed pixel by pixel, and eachof the color components RGB are dealt with individually.
 

The following example demonstrates a typicalmorphing process.

1. The original images of Ally and Lion, scaled to the same size. Please note that the distance between the eyes and the mouth is significantlylonger in the lion's picture than in Ally's picture.

__________
 

2. Perform coordinate transformations on the partitioned images to matchthe feature points of these two images. Here, we are matching the eyesand the mouths for these two images.  We can find that Ally's facebecomes longer, and the lion's face becomes shorter.

__________
 

3. Cross-dissolve the two images to generate a new image.
The morph result looks like a combination of these two wrapped faces. The new face has two eyes and one mouth, and it possesses the featuresfrom both Ally's and the lion's faces.

_______________

Back to top

---

III.Results

1. Feature-finding results
    We first fine-tuned the parameters of the eye-finderso that it can successfully detect the eyes in the photos of 13 differentcelebrities.  Afterwards, we evaluated the performance of the eye-finderby applying it to 160 of properly scaled photos of1999-2000EE new graduate students at Stanford.  The eye-finder successfullydetected both eyes from 113 of the 160 students, one of the two eyes of42 students, and none of the two eyes of 5 students.  Being able todetect 268 out of 320 eyes, the eye-finder had a detection rate of 84%.
    The following pictures are some of the successfulexamples.  Note that in the training set, some faces in the photoswere either tilting or not looking into the front directions, but our eye-finderwas still able to correctly detect the eyes.  Also note that the trainingset has people of different skin-colors and of both sexes.

    Among those of the pictures for which the eye-finderfailed, all sorts of other features such like ears, mouths, noses, rimsof glasses, etc, were detected instead.  Also, the eye-detection ratefor people wearing glasses are lower than people not wearing glasses. Below we illustrate some examples of wrong detection.
 

 

2. Morphing Results

(1) Morphing between faces ofdifferent people

Here are some non-animated morphing examples. We performed face morphingfor several different cases

- human and animal (lion)
- man and man
- man and woman

In the following, the very left and very right images of each row areoriginal images, and the intermediate ones are synthesized morphed images.

 

(2) Morphing between differentimages of the same person

The following are morphing examples for the faces of a person with differentexpressions or poses.
We want to interpolate the intermediate expressions or poses by morphing.

     Serious<===                                                  ===>   Smiling

Looking forward  <===                                    ===>  Facing another way

     Happy <===                                                   ===>   Angry

     Straight <===                                             ===> Swinging head

 

Animation of all the above examples can be found here

Back to top

---

IV.Conclusion
    An automatic face morphing algorithm is proposed. The algorithm consists of a feature finder followed by a face-morpher thatutilizes affine and bilinear coordinate transforms.
    We believe that feature extraction is the key techniquetoward building entirely automatic face morphing algorithms.  Moreover,we believe that the eyes are the most important features of human faces. Therefore, in this project we developed an eye-finder based on the ideathat eyes are, generally speaking, more complicated than the rest of theface.  We hence achieved an 84% of eye detection rate.  Also,we proposed red-ness, green-ness and blue-ness function and illustratedhow we would be able to find the mouth based on these functions.
    We demonstrated that a hybrid image of two humanfaces can be generated by morphing, and the hybrid face we generated indeedresembles each of the two "parent" faces.  Also, we demonstrated thatface morphing algorithms can help generate animation.
    Ideally speaking, the more feature points we canspecify on the faces, the better morphing results we can obtain. If wecan specify all the important facial features such as the eyes, the eyebrows,the nose, the edge points of the mouth, the ears, and some specific pointsof the hair, we are confident that we can generate very smooth and realisticallylooking morphing from one image to another.

Back to top

---

V. References

[1] Martin Bichsel, "Automatic Interpolation and Recognition of FaceImages by Morphing",     proceedings of the 2nd international conference on automatic face and gesture recognition, pp128-135
[2] Jonas Gomes et al. "Warping and morphing of graphical objects ",Morgan Kaufmann Publishers (1999).

Back to top

---

Appendix:who did what?
 
 Morphing algorithm: Yu-Li
 Feature finding algorithm: Yi-Wen
 Obtaining images from the internet: Yi-Wen
 Preprocessing of images: Yu-Li
 Presentation preparation: together
 Report writing: together