1. Tracking Face Orientation Kentaro Toyama Vision–Based Interaction Group Microsoft Research

2. 2/14/2001Vision for Graphics2 Applications Puppeteering of graphical avatars Chat rooms/online games/video conf. Online customer service Performance-driven animation Enhanced accessibility Input for novel UI User monitoring Avatar animation Hands-free cursor control Active- window control Gaze-adjusted Video conferencing Automated cameraman

3. 2/14/2001Vision for Graphics3 Related Work Templates/EKF (Jebara & Pentland, MIT Media Lab) JET/Elastic bunch graphs (von der Malsburg et al., Eyematic Interfaces) Active Appearance Models (Cootes & Taylor, Univ. of Manchester)

4. 2/14/2001Vision for Graphics4 Overview Coarse OrientationFine 3D PoseHead Position Face Detection + Attention Detection

5. 2/14/2001Vision for Graphics5 3D Pose Tracking Points tracked by multi- scale sum-of-absolute- difference template matching Estimate 6-DOF pose of known 3D points with Levenberg-Marquardt optimization

6. 2/14/2001Vision for Graphics6 Algorithm Assume N=9 known points, in head-centered frame: Find best-fit 6-DOF pose: Track N points in image: Project model points:

7. 2/14/2001Vision for Graphics7 Coarse Orientation Estimation Extract wavelet-based edge density features from known head location Project feature vectors onto trained ellipsoidal model Find maximum- likelihood 3D rotation 3D ellipsoidal model feature vectors detected face Joint work with Ying Wu edge density templates

8. 2/14/2001Vision for Graphics8 Algorithm Overview: Training Cropped Input Image Prepro- cessing Feature Extraction Ellipsoid Model Annotated Pose

9. 2/14/2001Vision for Graphics9 Preprocessing Cropped Input Image Grayscale and Resizing Histogram Equalization Masking

10. 2/14/2001Vision for Graphics10 Feature Extraction Preprocessed Image Feature Kernels Output Feature Vectors

11. 2/14/2001Vision for Graphics11 Algorithm Overview: Estimation Predictor Motion Model Final Pose Estimator Cropped Input Image Prepro- cessing Feature Extraction Ellipsoid Model

12. 2/14/2001Vision for Graphics12 Coarse Head Pose Estimation

13. 2/14/2001Vision for Graphics13 Coarse Head Pose Estimation

14. 2/14/2001Vision for Graphics14 Coarse Head Pose Estimation

15. 2/14/2001Vision for Graphics15 Bootstrap Initialization Final Pose Estimator Cropped Input Image Prepro- cessing Feature Extraction Boostrapped Ellipsoid Model Generic Ellipsoid Model

16. 2/14/2001Vision for Graphics16 Attention Detection Head PositionFace Detection +

17. 2/14/2001Vision for Graphics17 Head Position Estimation Bayesian fusion of low- level information Observable indicators of component reliability influence weighting Joint work with Eric Horvitz skin coloredge motion final estimate color reliability rel. indicator em reliability rel. indicator

18. 2/14/2001Vision for Graphics18 Components Reliability indicators Skin-color blobEllipse contour tracking - Bounding box aspect ratio - Fraction of pixels classified as skin - Ellipse-tracking residual - Fraction of pixels exhibiting interframe difference Tracking algorithm

19. 2/14/2001Vision for Graphics19 Quick & Dirty Face Detection Compute edge density and average intensity in predefined regions Graph match with relational template over range of positions and scales edge density image relational template detected face

20. 2/14/2001Vision for Graphics20 Bibliography F. Pighin, J. Hecker, D. Lischinski, D. H. Salesin, and R. Szeliski. Synthesizing realistic facial expressions from photographs. In SIGGRAPH'98 Proceedings, pages 75--84, Orlando, July 1998. Z. Liu, Z. Zhang, C. Jacobs, and M. Cohen. Rapid modeling of animated faces from video. Technical Report MSR-TR-2000-11, Microsoft Research, February 2000. B. Guenter et al. Making faces. Proceedings of SIGGRAPH 98, pages 55- -66, July 1998. V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. Proceedings of SIGGRAPH 99, pages 187--194, August 1999. K. Toyama. Prolegomena for robust face tracking. Technical Report MSR-TR-98-65, Microsoft Research, November 1998. F. Pighin, D. H. Salesin, and R. Szeliski. Resynthesizing facial animation through 3D model-based tracking. In Seventh International Conference on Computer Vision (ICCV'99), pages 143--150, Kerkyra, Greece, September 1999.

21. 2/14/2001Vision for Graphics21 Bibliography I. Buck et al. Performance-driven hand-drawn animation. In Symposium on Non Photorealistic Animation and Rendering, pages 101--108, Annecy, June 2000. ACM SIGGRAPH. D. A. Rowland and D. I. Perrett. Manipulating facial appearance through shape and color. IEEE Computer Graphics and Applications, 15(5):70- -76, September 1995. M. Turk and A. Pentland. Face recognition using eigenfaces. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'91), pages 586--591, Maui, Hawaii, June 1991. IEEE Computer Society Press. P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman. Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):711-- 720, July 1997. A. Lanitis, C. J. Taylor, and T. F. Cootes. Automatic interpretation and coding of face images using flexible models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):742--756, July 1997.