DDDAS: Stochastic Multicue Tracking of Objects with Many Degrees of Freedom PIs: D. Metaxas, A. Elgammal and V. Pavlovic Dept of CS, Rutgers University C. Neidle Linguistics, Boston Univ. C. Vogler, Gallaudet University

Goals and Objectives Develop vision-based system for the automated detection and analysis of nonverbal communication: –ASL Analysis –Biometrics Applications –Medical Applications

DDDAS: Technical Approach Approach: a) Analysis of video from CCD cameras b) Creation of database for face, upper body and gait c) Stochastic and Deformable model-based algorithms for face and upper body analysis d) Combination of Continuous and Discrete Trackers e) Data Driven Model Adaptability (co-Training Methods for Model refinement and Data Improvement) f) Data Driven error analysis for Model Switching

Technical Approach: Our System Developed System PC Face and body analysis Gait Analysis Images

1) Dynamic Data-Driven 3D Hand Tracking 2D features: –Edges –Color –Contour (boundaries) –Optical flow Continuous approach: –Edges, optical flow –Model-based, 2D =>3D forces, model fitting, articulation and refinement –Use of a physics-based dynamic estimation approach Discrete approach: –Skin color, edges, contour, and integration of them into multi- frame descriptors. –Appearance-based: find the best matching hand configuration in a database and use it as solution to the current (input) frame

Track “continuously” for fast results and track “discretely” for model re-initialization –We don’t lose track when continuous tracking fails under strong rotations and occlusions –We can still track the hand faster than any discrete approach Dynamic Data-Driven detection of switching between trackers –Learn the mapping between 2D-3D (actual) error using a database with continuous tracking results and Support Vector Regression. –Given the result of the continuous tracking for an input frame, extract the 2D error and make a conclusion for the 3D error (from SV Regression) –The system uses the error of fit to the data and makes dynamically the correct decision –The system updates its mapping between 2D-3D based on new data (co-training methods) ie updates the results of SV regression) –Lagrangian Dynamics Inverse Nonlinear Problems Dynamic Data-Driven 3D Hand Tracking (cont.)

DDDAS: 3D Hand Tracking Examples

2) Dynamic Data Driven 3D Face Tracking Face detection –Skin color database to learn the color distribution (data driven) –Facial pixels detection based on the learning –Extract the face bounding box Facial features extraction in 2D –ASM + KLT = discrete + continuous tracking in 2D = fast and never loses track Dynamic Data Driven Facial Model-based tracking and Refinement –Previous 3D solution + current 2D solution = current 3D solution –Based on the current 2D features, refine the face model –Based on the current 2D features decide what is the right ASM model to use.

DDDAS: 3D Face Tracking Results

Data Collection and Applications so Far ASL Data Collection based on collaboration with Linguists (Carol Neidle BU) –Based on single and Multiple cameras –Manual Annotation of ASL data Automated Detection of Faces Automated Model Initialization (faces and hands) Automated Tracking and Annotation of events (e.g raising eyebrows) Validation based on annotated data

Future Work ASL –Automated segmentation of finger vs non finger spelling –Linguistically important relationships between hand movement and facial expressions –Eventual attempt to deal with analysis involving large vocabularies Medical –Automated Analysis of Stress in people –Analysis of movement in Autistic kids Theory –Develop novel stochastic, learning and deformable modeling methods which can deal with bad data, moving backgrounds and very fast motions. Basically DDDAS models than can deal with nonlinear and complex movements