1. Using Multi-Modality to Guide Visual Tracking Jaco Vermaak Cambridge University Engineering Department Patrick Pérez, Michel Gangnet, Andrew Blake Microsoft Research Cambridge Paris, December 2002

2. Introduction  Visual tracking difficult: changes in pose and illumination, occlusion, clutter, inaccurate models, high-dimensional state spaces, etc.  Tracking can be aided by combining information in multiple measurement modalities  Illustrated here on head tracking using:  Sound and contour measurements  Colour and motion measurements

3. General Tracking

4. Tracking Equations  Objective : recursive estimation of the filtering distribution:  General solution:  Prediction step:  Filtering / update step:  Problem : generally no analytic solutions available

5. Particle Filter Tracking  Monte Carlo implementation of general recursions.  Filtering distribution represented by samples / particles with associated importance weights:  Proposal step : new particles proposed from a suitable proposal distribution:  Reweighting step: particles reweighted with importance weights:  Resampling step : multiply particles with high importance weights and eliminate those with low importance weights.

6. Particle Filter Building Blocks  Sampling from conditional density  Resampling  Reweighting with positive function

7. Particle Filter Implementation Requires specification of:  System configuration and state space  Likelihood model  Dynamical model for state evolution  State proposal distribution  Particle filter architecture

8. Head Tracking using Sound and Contour Measurements

9. Problem Formulation  Objective : track the head of a person in a video sequence using audio and image cues  Audio : time delay of arrival ( TDOA ) measurements at microphone pair orthogonal to optical axis of camera  Image : edge events along normal lines to a hypothesised contour  Complimentary modalities: audio good for (re)initialisation ; image good for fine localisation

10. System Configuration image plane camera microphone pair

11. Model Ingredients  Low-dimensional state space : similarity transform applied to a reference template  Dynamical prior : integrated Langevin equation, i.e. second-order Markov kernel  Multi-modal data likelihoods :  Sound based likelihood: TDOA at mic. pair  Contour based likelihood: edge events

12. Contour Likelihood  Input : maxima of projected luminance gradient along normals ( such events on normal)

13. Contour Likelihood  Advantages  Low computational cost  Robust to illumination changes  Drawbacks  Fragile because of narrow support (especially with only similarity transform on a fixed shape space)  Sensitive to background clutter  Extension  Multiply gradient by inter-frame difference to reduce influence of background clutter

14. Inter-Frame Difference Without frame differenceWith frame difference

15. Audio Likelihood  Input : positions of peaks in generalised cross-correlation function ( GCCF )  Reverberation leads to multiple peaks TDOA GCCF TDOA

16. Audio Likelihood  Deterministic mapping from Time Delay of Arrival ( TDOA ) to bearing angle ( microphone calibration ) to X-coordinate in image plane ( camera calibration )  Audio likelihood follows in similar manner to contour likelihood  Likelihood assumes a uniform clutter model

17. Particle Filter Architecture  Layered sampling : first X-position and sound likelihood; then rest  X-position proposal : mixture of diffusion dynamics and sound proposal:  To admit “ jumps ” from proposal X-dynamics have to be augmented with an uniform component :

18. Examples  Effect of inter-frame difference:  Conversational ping-pong:

19. Examples  Conversational ping-pong and sound based reinitialisation:

20. Head Tracking using Colour and Motion Measurements

21. Problem Formulation  Objective : detect and track the head of a single person in a video sequence taken from a stationary camera  Modality fusion:  Motion and colour measurements are complementary  Motion : when the object is moving colour is unreliable  Colour : when the object is stationary motion information disappears  Automatic object detection and tracker initialisation using motion measurements  Individualisation of the colour model to the object:  Initialised with a generic skin colour model  Adapted to object colour during periods of motion: motion model acts as “anchor”

22. Object Description and Motion  Head modelled as an ellipse that is free to translate and scale in the image  Binary indicator variable to signal whether object is present in the image or not, so object state becomes:  State components assumed to have independent motion models  Indicator: discrete Markov chain  Position and scale: Langevin motion with uniform initialisation:

23. Image Measurements  Measurements taken on a regular filter grid :  Measurement vector: hue image saturation image frame-difference image isotropic Gaussian filters

24. Observation Likelihood Model  Measurements at gridpoints assumed to be independent  Unique background (object absent) likelihood model for each gridpoint  All gridpoints covered by the object share the same foreground likelihood model:  At each gridpoint the measurements are also assumed to be independent:  Note that the background motion model is shared by all the gridpoints

25. Colour Likelihood Model  Normalised histograms for both foreground and background colour likelihood models:  Background models trained on a sequence without objects  Foreground models trained on a set of labelled face images  Histogram models supplied with a small uniform component to prevent numerical problems associated with empty bins

26. Motion Likelihood Model  Background frame-difference measurements empirically found to be gamma distributed:  Foreground frame-difference depends on magnitude of motion, number and orientation of foreground edges, etc.  Modelling these effects accurately is difficult  In general: if the object is moving foreground frame-difference measurements are substantially larger than those for background  Thus a two-component uniform distribution is adopted for the foreground frame-difference measurements ( outlier model )

27. Particle Proposal  Three stages of operation:  Birth : object first enters scene; proposal should detect object and spawn particles in the object region  Alive : object persists in scene; proposal should allow object to be tracked, whether it is stationary or moves around  Death : object leaves scene; proposal should kill particles associated with the object  Form of particle proposal: empirical probability of object being alive

28. Particle Proposal  Indicator proposal:  Birth only allowed if there is no object currently in the scene  All particles alive are subjected to a fixed death probability  State proposal:  Langevin dynamics if object is alive  Gaussian birth proposal: parameters from detection module

29. Object Detection  Object region detected by probabilistic segmentation of the horizontal and vertical projections of the frame-difference measurements:  Region location and size determine parameters for birth proposal distribution

30. Colour Model Adaptation  Why:  Generic skin colour model may be too broad for accurate localisation  Model sensitive to colour changes due to changes in pose and illumination  When:  Object present and moving : largest variations in colour expected  Motion likelihood “anchors” particles around moving object  How:  Gradual: avoid fitting to the background: enforced with prior  Stochastic EM: contribution of particles proportional to likelihood

31. Colour Model Adaptation  Unknown parameters: normalised bin values for object hue and saturation histograms  EM Q-function for MAP estimation:  No analytic solution but particle approximation yields:  Monte Carlo approximation only performed over particles that are currently alive

32. Colour Model Adaptation  Dirichlet prior used for parameter updates:  Prior centred on old parameter values  Variance controlled by multiplicative constant  Update rule for normalised bin counts becomes:

33. What Happens? particle histograms weighted average histogram

34. Implementation  Colour model adaptation iterations occur between particle prediction and particle reweighting in standard particle filter  Stochastic EM algorithm initialised with parameters from previous time step  A single stochastic EM iteration is sufficient at each time step  Number of particles is fixed to 100  Non-optimised algorithm runs at 15fps on standard desktop PC

35. Examples No adaptation: tracker gets stuck on skin- coloured carpet in the background Adaptation: tracker successfully adapts to changes in pose and illumination and lock is maintained No motion likelihood: tracker fails, illustrating need for “anchor” likelihood

36. Examples Tracking is successful despite substantial variations in pose and illumination and the subject temporarily leaving the scene Particles are killed when the subject leaves the scene; upon re-entering the individualised colour model allows lock to be re-established within a few frames

37. The End