1. Multi-Group Tracking with Adaptive Target Model Loris Bazzani, PhD student (XXIV cycle) University of Verona Department of Computer Science PhD thesis proposal

2. 2 Objectives Compare existing Multi-Target Tracking methods, studying the sampling technique Propose a new tracking method: Multi- Group Tracking Model robustly and adaptively the target Integrate target model with Multi-Group tracking

3. 3 Introduction Multi-Target Tracking Multi-Group Tracking Target Modeling Conclusions Outline State of the art Open issues Proposed ideas

4. Introduction (1) Tracking: spatial and temporal localization of a mobile object in an environment monitored by sensor(s) Multi-target (MTT): keeping the identity of different targets Reliable: insensible to noise and occlusions Application to Automated Surveillance

5. 5 Introduction (2) Multi-Group Tracking (MGT): Spatial and temporal localization of groups of objects Motivations: Humans prefer to stay in group rather than alone High-level representation of the relations among the targets MGT is simpler than MTT in a crowded scenario MGT can help MTT when occlusions occur

6. 6 Introduction (3) Multi-Group Tracking (MGT): why is MGT a hard task?

7. 7 Introduction (4) Target Model: A general and representative example that summarizes any possible changing of the target intrinsic variations: pose variation and shape deformation extrinsic variations: illumination changes, camera movement, and occlusions Not considering the above variation causes the failure of the tracking [Ross08] Fit with the re-identification problem

8. 8 Multi-Target Tracking (1) Filtering We observe the real world events as a state by the measurement process Objective: estimating the state of the system at each instant given the measurements Abstract Formulation [Arulampalam02] State Space Approach for modeling discrete-time dynamic systems State: abstract nature of the target Measurement: “visible” dimensions of the state space

9. 9 Multi-Target Tracking (2) Data Association [Bar-Shalom87] The observer has at his disposal a huge amount of measurements Finding the correct correspondences between measurements and states of the system

10. 10 Multi-Target Tracking (3)

11. 11 Multi-Target Tracking (4) Particle Filter (PF) [Isard01]

12. Multi-Target Tracking (5) State Space Conformation (+) Efficient sampling (-) No interaction modeling (+) Implicit interaction modeling (-) Curse of dimensionality (+) Efficient sampling (+) Implicit interaction modeling [Isard01] [MacCormick00] [Lanz06]

13. Multi-Target Tracking (6) - HJS vs. MHT [Bazzani09] - MHT cons: 1) Multiple tracks cause proliferation in the number of tracks 2) Occlusions generate new tracks 3) Not robust to non-linear people motion HJS pros: 1) One track is kept for each target 2) Partial occlusions are handled; 3) Deal with non-linearity of people motion.

14. 14 Multi-Target Tracking (7) Open issues of PF-based MTT (and MGT): Sampling Method Dynamic Model Linear-Gaussian model Observation Model State Estimation Maximum-A-Posteriori or Weighted Mean

15. 15 Multi-Target Tracking (6) Sequential importance sampling/re- sampling, Regularized PF [Arulampalam02] Markov Chain Monte Carlo (MCMC)[Andrieu03] Reversible-Jump MCMC[Khan05] Rao-Blackwellizing PF[Schindler05] Sampling Independent spaces[MacCormick00] Joint space[Isard01] Hybrid joint-separable (HJS) spaces[Lanz06] State Space

16. Multi-Target Tracking (8.1) Sequential importance sampling/re-sampling [Arulampalam02] : classical PF + degeneracy problem avoiding by re-sampling Regularized PF [Arulampalam02] : resamples applying a Kernel to the particles MCMC [Andrieu03] : defines a Markov chain over the state space, such that the stationary distribution of the chain is equal to the sought posterior Reversible-Jump MCMC [Khan05] : switches between variable dimensional state spaces Rao-Blackwellizing PF [Schindler05] : analytically computes a portion of the distribution other the state space Sampling

17. 17 Multi-Target Tracking (8.2) Likelihood: compare an observation z given a hypothesis of state of the system x Usually defined in the Gibbs form: where d is a metric x and z MUST be represented in the same feature space ISSUES: feature space (for x and z) and metric definitions occlusion handling Observation model

18. 18 Multi-Target Tracking (9) Occlusion Handling Study and implement RJ-MCMC particle filter Propose a set of jumps in order to cope with tracking a variable number of objects Propose an observation model in RJ-MCMC framework - Proposed Research -

19. 19 Multi-Group Tracking (1) defines a group as the moving regions, extracted from a foreground analysis infers the MGT from the tracks estimation carried out by the MOT (e.g. tracks clustering) uses the foreground information and MOT to detect groups, but then tracks them as different entities (-) Social interactions cannot be caught by the foreground analysis (-) The inter-group dynamic yields a loss of appearance informations (-) Direct dependence from MOT (-) The MOT estimation is not reliable when occlusions occur (+) Cancel out the above problems (-) Model creation is a hard task

20. 20 Multi-Group Tracking (2) Foreground-based MGT Tracking at three levels of abstraction [McKenna00] : Regions: connected component that have been tracked for T frames People: one or more regions grouped together Groups: one or more people grouped together, if they share a region (+) Simplicity (-) heuristic FG analysis

21. 21 Multi-Group Tracking (4) MGT from MOT Clustering particles in PF-based tracking [Kembhavi06] time t-1time t particle setgroups proximity & appearance graph new groups Objective: minimize the proximity and the appearance distance if i and j are grouped cost to track i into the group g

22. 22 Multi-Group Tracking (3) MGT from MOT MCMC PF for group tracking [Pang07]: Track groups as ensemble of targets analyzing : group variable Treated as Bayesian estimation problem Group structure model: captures the relations among objects

23. 23 Multi-Group Tracking (4) - Proposed Research - Problems: Definition of “Group”: an entity containing targets with similar characteristics (e.g. motion, interactions,...) Deterministic/formal definition as an ensemble of objects Add non-deterministic component into the tracking method Intra-group occlusions: if we know that the objects hasn’t left the group, we infer that it is still into the group Inter-group occlusions: tracking of groups

24. 24 Multi-Group Tracking (5) - Proposed Research -

25. 25 Multi-Group Tracking (6) - Proposed Research - Use a MOT method Create a MGT method (track only groups) Definition of group dynamics -> sociological studies Definition of a group observation model Define a collaborative probabilistic framework in order to share MGT and MOT informations

26. 26 Target Modeling (1) Fixed target models Single-feature color histogram[Sebastian08] color correlogram[Huang99] spatiogram[Birchfield05] multi-resolution histogram[Hadjidemetriou01] SIFT feature[Lowe99] Multiple-features covariance descriptor[Porikli06] ensemble of localized features[Gray08] bag of features[Nowak06] Template and Active Contours [Blake98] Subspace methods [Ross08] Part-based models [Ramanam07]

27. 27 Target Modeling (1) Train the model using the appearance data available before tracking begins Adapt the model to account for its changes in appearance, using an on- line learning method Open Issues: Representation of the target: feature space Leaning technique

28. 28 Target Modeling (2) Feature Color Histogram: which is the best color space for tracking? [Sebastian08] (-) No spatial information Color correlogram [Huang99], spatiogram [Birchfield05], multi-resolution histogram [Hadjidemetriou01] (+) Add the spatial information (-) Increase the computational burden

29. 29 Target Modeling (3) Feature Covariance descriptors [Porikli06] Spatial and appearance attributes (+) Natural way of fusing multiple features (-) Computationally expensive -> integral images

30. 30 Target Modeling (4) Ensemble of localized features [Gray08] Define a feature space and let machine learning approach find the best representation AdaBoost extracts the object representation: the most discriminative set of features, and the similarity measures: the most discriminative set of likelihood ratio test Used for re-identification problem Fixed target models

31. 31 Target Modeling (5) Adaptive target models Incremental learning of Covariance-based descriptors [Porikli06] Principal Component Analysis (PCA) incremental learning [Ross08] Convex combination of models using a learning rate Feature-based model Parametric model

32. 32 Target Modeling (6) Patch-based (local) updating [Kwon09] Evolve photometric and geometric appearance Local Patches can be added, deleted or moved to different position Examining the patches by landscape analysis Bad patches are modified on-line: background patches and patches in regions with high density of patch are deleted Good patches are moved Appearance model is updated with a convex combination

33. 33 Target Modeling (7) - Proposed Research - Part-based multiple-features: Maximally Stable Color Regions HS(V) histograms Recurrent high-structured patches Temporal updating: Delete the non-stable features Cover the Variability of the stable features Global Local view point, partial occlusions illumination, view point, deformations, partial occlusions partial occlusions, illumination Invariances:

34. 34 Conclusions Compare existing Multi-Target Tracking methods, studying the sampling technique Propose a new tracking method: Multi- Group Tracking Model robustly and adaptively the target Integrate target model with Multi-Group tracking, using HJS and RJ-MCMC

35. 35 The beginning Thanks for attention Now, we “just” put the proposed ideas into practice Questions?

36. 36 References [Ross08] D.A. Ross, J. Lim, R.S. Lin, and M.H. Yang. Incremental learning for robust visual tracking. International Journal of Computer Vision, 77(1):125–141, 2008. [Arulampalam02] M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp. A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Trans. on Signal Processing, 50(2):174–188, 2002. [Bar-Shalom87] Y. Bar-Shalom. Tracking and data association. Academic Press Professional, Inc., San Diego, CA, USA, 1987. [Isard01] M. Isard and J. MacCormick. Bramble: A bayesian multipleblob tracker. In IEEE Int. Conf. on Computer Vision, 2001. [MacCormick00] John MacCormick and Andrew Blake. A probabilistic exclusion principle for tracking multiple objects. Int. J. Comput. Vision, 39(1):57–71, 2000. [Lanz06] O. Lanz. Approximate bayesian multibody tracking. IEEE Trans. on Pattern Analysis and Machine Intelligence, 28(9):1436–1449, 2006. [Andrieu03] Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. An introduction to mcmc for machine learning. Machine Learning, 50(1):5–43, January 2003.

37. 37 References [Khan05] Z. Khan, T. Balch, and F. Dellaert. Mcmc-based particle filtering for tracking a variable number of interacting targets. IEEE Trans. on Pattern Analysis and Machine Intelligence, 27(11):1805–1819, 2005. [Schindler05] G. Schindler and F. Dellaert. A Rao-Blackwellized Parts-Constellation Tracker. In ICCV Workshop on Dynamical Vision; International Conference on Computer Vision, 2005. Springer [Mckenna00] Stephen J. Mckenna, Sumer Jabri, Zoran Duric, Harry Wechsler, and Azriel Rosenfeld. Tracking groups of people. Computer Vision and Image Understanding, 2000. [Pang07] Sze Kim Pang, Jack Li, and Simon Godsill. Models and algorithms for detection and tracking of coordinated groups. In Symposium of image and Signal Processing and Analisys, 2007. [Sebastian08] Sebastian, P.; Yap Vooi Voon; Comley, R., "The effect of colour space on tracking robustness," Industrial Electronics and Applications, 2008. ICIEA 2008. 3rd IEEE Conference on, vol., no., pp.2512-2516, 3-5 June 2008 [Huang99] J. Huang, S. Ravi Kumar, M. Mitra, W.J. Zhu, and R. Zabih. Spatial color indexing and applications. International Journal of Computer Vision, 35(3):245–268, 1999. [Birchfield05] ST Birchfield and S. Rangarajan. Spatiograms versus histograms for region-based tracking. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, volume 2, 2005.

38. 38 References [Hadjidemetriou01] E. Hadjidemetriou, MD Grossberg, and SK Nayar. Spatial information in multiresolution histograms. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, volume 1, 2001. [Porikli06] F. Porikli, O. Tuzel, and P. Meer. Covariance tracking using model update based on lie algebra. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, 2006. [Gray08] D. Gray and H. Tao. Viewpoint Invariant Pedestrian Recognition with an Ensemble of Localized Features. In Proceedings of the 10th European Conference on Computer Vision: Part I, pages 262–275. Springer, 2008. [Kwon09] J.S. Kwon and K.M. Lee. Tracking of a non-rigid object via patch-based dynamic appearance modeling and adaptive basin hopping monte carlo sampling. pages 1208–1215, 2009.