Exploiting Sparsity and Co-occurrence for Action Unit Recognition

Presented at IEEE FG 2015

Yale Song^*1, Daniel McDuff^*2, Deepak Vasisht³, Ashish Kapoor⁴
¹Yahoo Labs, ²Affectiva, ³MIT CSAIL, ⁴Microsoft Research
(* Y. Song and D. McDuff have equally contributed to this work while both were at MIT.)

Figure 1. Facial action units have strong sparsity and co-occurrence structure. Even for complex expressions such as disgust and surprise, less than only five out of 45 action units are activated. Further, groups of action units tend to co-occur in similar expressions. We exploit these two properties for facial action unit recognition.

Abstract

We present a novel Bayesian framework for facial action unit recognition. The first key observation behind this work is sparsity: out of possible 45 (and more) facial action units, only very few are active at any moment. The second is the strong statistical co-occurrence structure: most facial expressions are made by common combinations of facial action units, so knowing the presence of one can act as a strong prior for inferring the presence of others. We developed a novel Bayesian graphical model that encodes these two natural aspects of facial action units via compressed sensing and group-wise sparsity inducing priors. One crucial aspect of our approach is the allowance of overlapping group structures, which proves useful in dealing with action units that occur frequently across multiple groups. We derive an efficient inference scheme and show how such sparsity and co-occurrence can be automatically learned from data. Experiments on three standard benchmark datasets show superiority over the state-of-the-art.

Publication

FG 2015 paper [pdf][bibtex]

Downloads

MATLAB code [github]
Precomputed features of CK+ dataset [2] [mat]
AU co-occurrence statistics from the Kassam dataset [3] [mat]

References

[1] Ashish Kapoor, Raajay Viswanathan, and Prateek Jain. "Multi-label classification using bayesian compressed sensing." NIPS 2012.
[2] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews. "The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression." In CVPRW, 2010.
[3] K. S. Kassam. "Assessment of emotional experience through facial expression." PhD thesis, Harvard, 2010.

Last update: Apr 30, 2015