Head Tracking in Meeting Scenarios Sascha Schreiber

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 2/18 Overview Video processing system for person action recognition Single person head tracking using elliptical structures Multiple person tracking Summary and outlook

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 3/18 System Setup 1)Face/Head Tracking: Face Detection & Particle Filtering 2)Feature extraction: Global Motion Features 3)Kalman filtering: necessary for disturbed feature streams 4)Temporal segmentation: Bayesian Information Criterion 5)HMM classification: 6 models, one for each gesture Video streamTracked personsKalman filteringTemporal segmentationHMM classification  Standing up  Sitting down  Pointing  Nodding  Shaking head  Writing Gesture inventory:

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 4/18 Single Person Tracking Former system: –Initialization by pyramid sampling  slow, only frontal to half-profile views of faces can be detected –Weighting of particles by the output of a neural network  Loss of track, if no face visible (e.g. person leans forward)  Particles have to be placed quite exactly

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 5/18 Actual system: –Initialization by skin colored regions Single Person Tracking Former system: –Initialization by pyramid sampling  slow, only frontal views of faces can be detected 1) Skin color blob detection 2) Search for valid blobs  ratio major axis/minor axis

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 6/18 Former system: –Weighting of particles by the output of a neural network  Loss of track, if no face visible (e.g. person leans forward)  Particles have to be placed quite exactly Actual system –Weighting of particles using ellipses Single Person Tracking 1) Compute gradient image 2) Search best fitting gradient along normal:  3) Calculate cost function: 

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 7/18 Single Person Tracking Comparision of tracking results: Neural netEllipses

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 8/18 Multiple Person Tracking Problem: -Particles concentrate on location with highest head likelihood Solution: -Introduction of „Super particles“ representing the number of persons in scenario

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 9/18 Multiple Person Tracking Weighting of Superparticles 1) Counting blobs in binary skin color mask  n blobs 2) Testing all possible combinations

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 10/18 Multiple Person Tracking Weighting of Superparticles Configuration coverage Configuration compactness Particle likelihood 1) Counting blobs in binary skin color mask  n blobs 2) Testing all possible combinations 3) Measurement / Cost function

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 11/18 Multiple Person Tracking Weighting of Superparticles 1) Counting blobs in binary skin color mask  n blobs 2) Testing all possible combinations 3) Measurement / Cost function

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 12/18 Multiple Person Tracking Demovideo „Multiple Person Tracking“

24/09/2004 Institute for Human-Machine Communication Munich University of Technology Sascha Schreiber 13/18 Summarization & Outlook Robust head tracking  Improving tracking algorithm Features computed relative to the head position  Tracking hands as additional feature Temporal segmentation facilitated by tracking Combined tracking and recognition

Head Tracking in Meeting Scenarios Sascha Schreiber