Tracking Face Orientation Kentaro Toyama Vision–Based Interaction Group Microsoft Research

2/14/2001Vision for Graphics2 Applications Puppeteering of graphical avatars Chat rooms/online games/video conf. Online customer service Performance-driven animation Enhanced accessibility Input for novel UI User monitoring Avatar animation Hands-free cursor control Active- window control Gaze-adjusted Video conferencing Automated cameraman

2/14/2001Vision for Graphics3 Related Work Templates/EKF (Jebara & Pentland, MIT Media Lab) JET/Elastic bunch graphs (von der Malsburg et al., Eyematic Interfaces) Active Appearance Models (Cootes & Taylor, Univ. of Manchester)

2/14/2001Vision for Graphics4 Overview Coarse OrientationFine 3D PoseHead Position Face Detection + Attention Detection

2/14/2001Vision for Graphics5 3D Pose Tracking Points tracked by multi- scale sum-of-absolute- difference template matching Estimate 6-DOF pose of known 3D points with Levenberg-Marquardt optimization

2/14/2001Vision for Graphics6 Algorithm Assume N=9 known points, in head-centered frame: Find best-fit 6-DOF pose: Track N points in image: Project model points:

2/14/2001Vision for Graphics7 Coarse Orientation Estimation Extract wavelet-based edge density features from known head location Project feature vectors onto trained ellipsoidal model Find maximum- likelihood 3D rotation 3D ellipsoidal model feature vectors detected face Joint work with Ying Wu edge density templates

2/14/2001Vision for Graphics8 Algorithm Overview: Training Cropped Input Image Prepro- cessing Feature Extraction Ellipsoid Model Annotated Pose

2/14/2001Vision for Graphics9 Preprocessing Cropped Input Image Grayscale and Resizing Histogram Equalization Masking

2/14/2001Vision for Graphics10 Feature Extraction Preprocessed Image Feature Kernels Output Feature Vectors

2/14/2001Vision for Graphics11 Algorithm Overview: Estimation Predictor Motion Model Final Pose Estimator Cropped Input Image Prepro- cessing Feature Extraction Ellipsoid Model

2/14/2001Vision for Graphics12 Coarse Head Pose Estimation

2/14/2001Vision for Graphics13 Coarse Head Pose Estimation

2/14/2001Vision for Graphics14 Coarse Head Pose Estimation

2/14/2001Vision for Graphics15 Bootstrap Initialization Final Pose Estimator Cropped Input Image Prepro- cessing Feature Extraction Boostrapped Ellipsoid Model Generic Ellipsoid Model

2/14/2001Vision for Graphics16 Attention Detection Head PositionFace Detection +

2/14/2001Vision for Graphics17 Head Position Estimation Bayesian fusion of low- level information Observable indicators of component reliability influence weighting Joint work with Eric Horvitz skin coloredge motion final estimate color reliability rel. indicator em reliability rel. indicator

2/14/2001Vision for Graphics18 Components Reliability indicators Skin-color blobEllipse contour tracking - Bounding box aspect ratio - Fraction of pixels classified as skin - Ellipse-tracking residual - Fraction of pixels exhibiting interframe difference Tracking algorithm

2/14/2001Vision for Graphics19 Quick & Dirty Face Detection Compute edge density and average intensity in predefined regions Graph match with relational template over range of positions and scales edge density image relational template detected face

2/14/2001Vision for Graphics20 Bibliography F. Pighin, J. Hecker, D. Lischinski, D. H. Salesin, and R. Szeliski. Synthesizing realistic facial expressions from photographs. In SIGGRAPH'98 Proceedings, pages , Orlando, July Z. Liu, Z. Zhang, C. Jacobs, and M. Cohen. Rapid modeling of animated faces from video. Technical Report MSR-TR , Microsoft Research, February B. Guenter et al. Making faces. Proceedings of SIGGRAPH 98, pages , July V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. Proceedings of SIGGRAPH 99, pages , August K. Toyama. Prolegomena for robust face tracking. Technical Report MSR-TR-98-65, Microsoft Research, November F. Pighin, D. H. Salesin, and R. Szeliski. Resynthesizing facial animation through 3D model-based tracking. In Seventh International Conference on Computer Vision (ICCV'99), pages , Kerkyra, Greece, September 1999.

2/14/2001Vision for Graphics21 Bibliography I. Buck et al. Performance-driven hand-drawn animation. In Symposium on Non Photorealistic Animation and Rendering, pages , Annecy, June ACM SIGGRAPH. D. A. Rowland and D. I. Perrett. Manipulating facial appearance through shape and color. IEEE Computer Graphics and Applications, 15(5): , September M. Turk and A. Pentland. Face recognition using eigenfaces. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'91), pages , Maui, Hawaii, June IEEE Computer Society Press. P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman. Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7): , July A. Lanitis, C. J. Taylor, and T. F. Cootes. Automatic interpretation and coding of face images using flexible models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7): , July 1997.