M Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber Face Tracking and Person Action Recognition - Update

Sascha Schreiber 2/14 Institute for Human-Machine Communication Munich University of Technology M Recapitulation of methodology for action recognition Face tracking with I-Condensation Recognition performance comparison on actions from the m4 dataset Kalman filtering of occluded gestures Outlook Overview

Sascha Schreiber 3/14 Institute for Human-Machine Communication Munich University of Technology M Person Action Recognition Extraction of person locations Temporal segmentation Feature calculation Classification of segments Face detection/Blob tracking Global Motion Features Bayesian Information Criterion Hidden Markov Models Actions, timestamps

Sascha Schreiber 4/14 Institute for Human-Machine Communication Munich University of Technology M Person Action Recognition Extraction of person locations Temporal segmentation Feature calculation Classification of segments Face detection/Blob tracking Global Motion Features Bayesian Information Criterion Hidden Markov Models Actions, timestamps

Sascha Schreiber 5/14 Institute for Human-Machine Communication Munich University of Technology M Face Tracking Particle filtering with ICondensation N weighted particles Updating using their likelihood Sampling from prediction density - Standard Condensation sampling - Sampling from importance function for reinitialisation - Importance sampling with weighting correction factor  Introduction of importance function: skin color distribution Automatic initialization by pyramid sampling and MLP classification

Sascha Schreiber 6/14 Institute for Human-Machine Communication Munich University of Technology M Performance of Face Tracking Standard CondensationICondensation Demonstration of difference between:

Sascha Schreiber 7/14 Institute for Human-Machine Communication Munich University of Technology M Person Action Recognition Extraction of person locations Temporal segmentation Feature calculation Classification of segments Face detection/Blob tracking Global Motion Features Bayesian Information Criterion Hidden Markov Models Actions, timestamps

Sascha Schreiber 8/14 Institute for Human-Machine Communication Munich University of Technology M IDIAP training data (TRN 01-30), IDIAP test data (TST 01-30) Continuous HMMs (6 states, 3 mixtures) Sit downStand upNodding Shaking head WritingPointing Score Sit down % Stand up % Nodding % Shaking head % Writing % Pointing % Overall80% Recognition Performance m4

Sascha Schreiber 9/14 Institute for Human-Machine Communication Munich University of Technology M IDIAP training data (TRN 01-30), IDIAP test data (TST 01-30) Discrete HMMs (6 states, codebook 1500) Sit downStand upNodding Shaking head WritingPointing Score Sit down % Stand up % Nodding % Shaking head % Writing % Pointing % Overall86% Recognition Performance m4

Sascha Schreiber 10/14 Institute for Human-Machine Communication Munich University of Technology M Occluded Gestures Classification result Classification Compensation of occlusion Stream- segmentation Feature- extraction Smoothed featurestream Featurestream Segmented Featurestream Video- stream Occlusion Scenario: Person walking on front of a tracked object

Sascha Schreiber 11/14 Institute for Human-Machine Communication Munich University of Technology M Occluded Gestures Application for Kalman filtering: Calculation of an estimate Time update equationMeasurement update equation Discrete-time process:

Sascha Schreiber 12/14 Institute for Human-Machine Communication Munich University of Technology M Occluded Gestures Kalman- filter Kalman- filter Kalman- filter N action-specialized Kalman-Filters, each trained for a special gesture to be recognized by the HMM Improving featurestream by smoothing with : Kalman- filter One general Kalman-Filter for the disturbed featurestream

Sascha Schreiber 13/14 Institute for Human-Machine Communication Munich University of Technology M Performance of Kalman filtering Score Featurestream unoccluded & unfiltered79,86% Featurestream occluded & unfiltered56,75% Featurestream occluded & filtered (general)57,98% Featurestream occluded & filtered (specialized)60,12% IDIAP training data (TRN 01-30), IDAP test data (TST 01-30) Continuous HMMs (6 states, 3 mixtures)

Sascha Schreiber 14/14 Institute for Human-Machine Communication Munich University of Technology M Implementation of extended Kalman filter Head orientation tracking Integration of face recognition into particle filter Further improvement of action detection on m4 data Connection to Meeting Segmentation / Multimodal Recognizer Outlook

M Institute for Human-Machine Communication Munich University of Technology Face Tracking and Person Action Recognition - Update Sascha Schreiber