Project Description

The face is a rich source of information about human behavior. Facial displays indicate emotion, pain, brain function and pathology, and regulate social behavior. Manual methods of coding facial behavior are labor intensive, semi-quantitative, and difficult to standardize across laboratories or over time. With few exceptions, current approaches to automated analysis focus on a small set of prototypic expressions (e.g., anger or joy), which facilitates analysis. In daily life, prototypic expressions occur relatively infrequently, and emotion more often is communicated by change in one or two discrete features, such as tightening the lips in anger. To capture the subtlety of human emotion and non-verbal communication, our interdisciplinary team of computer scientists and psychologists developed the first version of Automated Face Analysis. Automated Face Analysis quantifies subtle changes in facial motion and demonstrates concurrent validity with human observers using the Facial Action Coding System. Continuing system development is part of a larger goal of developing computer systems that can detect human activity, recognize the people involved, understand their behavior, and respond appropriately.

We developed an automatic expression analysis system, including both facial feature extraction, representation, and expression recognition, that automatically discriminates among subtly different facial expressions based on Facial Action Coding System (FACS) action units (AUs) using neural network. To detect qualitative changes in facial expression, we develop a multi-state model based system for tracking facial features that uses convergent methods of feature analysis. We define the different head orientations and different component appearances as different states. For different head states, different face components are used. For each face component, there are different states also. For each different state, a description and extraction method should be different.

Multi-state facial component models are proposed for tracking and modeling both permanent (e.g. mouth, eye, and brow) and transient (e.g. furrows and wrinkles) facial features. Based on these multi-state models, and without artificial enhancement, we detect and track the subtle changes of the facial features, including mouth, eyes, brow, cheeks, and their related wrinkles and facial furrows. Motivated by FACS action units, these changes are represented as a collection of mid-level feature parameters. Then, we employ a neural network to recognize the action units after the facial features are correctly extracted and suitably represented. Eleven basic lower face action units and combinations (Neutral, AU9, AU 10, AU 12, AU 15, AU 17, AU 20, AU 25, AU 26, AU 27, and AU23+24) and seven basic upper face action units (Neutral, AU1, AU2, AU4, AU5, AU6, AU7) are identified by a single neural network for lower face and upper face separately.