1. M4 meeting@Prague 22-23.01.2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber Face Tracking and Person Action Recognition - Update

2. Sascha Schreiber 2/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Recapitulation of methodology for action recognition Face tracking with I-Condensation Recognition performance comparison on actions from the m4 dataset Kalman filtering of occluded gestures Outlook Overview

3. Sascha Schreiber 3/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Person Action Recognition Extraction of person locations Temporal segmentation Feature calculation Classification of segments Face detection/Blob tracking Global Motion Features Bayesian Information Criterion Hidden Markov Models Actions, timestamps

4. Sascha Schreiber 4/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Person Action Recognition Extraction of person locations Temporal segmentation Feature calculation Classification of segments Face detection/Blob tracking Global Motion Features Bayesian Information Criterion Hidden Markov Models Actions, timestamps

5. Sascha Schreiber 5/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Face Tracking Particle filtering with ICondensation N weighted particles Updating using their likelihood Sampling from prediction density - Standard Condensation sampling - Sampling from importance function for reinitialisation - Importance sampling with weighting correction factor  Introduction of importance function: skin color distribution Automatic initialization by pyramid sampling and MLP classification

6. Sascha Schreiber 6/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Performance of Face Tracking Standard CondensationICondensation Demonstration of difference between:

7. Sascha Schreiber 7/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Person Action Recognition Extraction of person locations Temporal segmentation Feature calculation Classification of segments Face detection/Blob tracking Global Motion Features Bayesian Information Criterion Hidden Markov Models Actions, timestamps

8. Sascha Schreiber 8/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 IDIAP training data (TRN 01-30), IDIAP test data (TST 01-30) Continuous HMMs (6 states, 3 mixtures) Sit downStand upNodding Shaking head WritingPointing Score Sit down90000190% Stand up19001364% Nodding21215586874% Shaking head 0022164137% Writing0038204682485% Pointing002007097% Overall80% Recognition Performance m4

9. Sascha Schreiber 9/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 IDIAP training data (TRN 01-30), IDIAP test data (TST 01-30) Discrete HMMs (6 states, codebook 1500) Sit downStand upNodding Shaking head WritingPointing Score Sit down90000190% Stand up310000171% Nodding13251233087% Shaking head 003021105% Writing00280515794% Pointing414065779% Overall86% Recognition Performance m4

10. Sascha Schreiber 10/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Occluded Gestures Classification result Classification Compensation of occlusion Stream- segmentation Feature- extraction Smoothed featurestream Featurestream Segmented Featurestream Video- stream Occlusion Scenario: Person walking on front of a tracked object

11. Sascha Schreiber 11/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Occluded Gestures Application for Kalman filtering: Calculation of an estimate Time update equationMeasurement update equation Discrete-time process:

12. Sascha Schreiber 12/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Occluded Gestures Kalman- filter Kalman- filter Kalman- filter N action-specialized Kalman-Filters, each trained for a special gesture to be recognized by the HMM Improving featurestream by smoothing with : Kalman- filter One general Kalman-Filter for the disturbed featurestream

13. Sascha Schreiber 13/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Performance of Kalman filtering Score Featurestream unoccluded & unfiltered79,86% Featurestream occluded & unfiltered56,75% Featurestream occluded & filtered (general)57,98% Featurestream occluded & filtered (specialized)60,12% IDIAP training data (TRN 01-30), IDAP test data (TST 01-30) Continuous HMMs (6 states, 3 mixtures)

14. Sascha Schreiber 14/14 Institute for Human-Machine Communication Munich University of Technology M4 meeting@Prague 22-23.01.2004 Implementation of extended Kalman filter Head orientation tracking Integration of face recognition into particle filter Further improvement of action detection on m4 data Connection to Meeting Segmentation / Multimodal Recognizer Outlook

15. M4 meeting@Prague 22-23.01.2004 Institute for Human-Machine Communication Munich University of Technology Face Tracking and Person Action Recognition - Update Sascha Schreiber