1. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 1 ‘’ Pedestrian Tracking Using DCM and Image Correlation ’’ G.Antonini S.Venegas JP.Thiran and M.Bierlaire IM2-2004

2. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 2 Outline Introduction ( motivations and objectives ) DCM for pedestrian dynamic - Pedestrian behavior modeling (overview) - DCM specification - DCM calibration - DCM estimation results Pedestrian detection using DCM and image correlation Pedestrian tracking using DCM and image correlation Results Conclusions and future works

3. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 3 Introduction Motivation: new research project conducted with the aim to integrate state-of-the-art tracking algorithms with behavioral models for pedestrian dynamic for video surveillance applications (IM2.SA). Objectives: our goal is to provide a tool for the computation of pedestrian trajectories in real, complex scenarios. These trajectories could then be used to build statistical density maps and land-use maps for scene analysis.

4. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 4 DCM for pedestrian behavior

5. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 5 Pedestrian behavior : overview Previous approaches are mainly physical-based models: people as particles (microscopic models) subjected to forces; people with fluid-like properties (macroscopic models, Navier-Stokes or Boltzmann-like equations). Our approach: - microscopic model (time-based behavior of each pedestrian); - walking is a sequence of choices: where to put the next step? (DCM); - dynamical and individual-based spatial discretization.

6. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 6 Pedestrian behavior : overview MicroscopicMacroscopic Two methodologies cellular automata social force mixed discrete-continuous time –space behavior of individual pedestrians Macroscopic Henderson,L. (1971) Helbing,D. (1992) pedestrians with fluid-like properties

7. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 7 P RoI Static pedestrian area The space model

8. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 8 Pedestrian behavior : our approach..a sequence of choices: where to put the next step? discrete choice model..a microscopic approach.. agent-based..dynamical and flexible.. dynamic individual-based spatial discretization

9. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 9 The space model Constant speed Accelerated Decelerated 10 15 20 25 Dynamic and individual-based We discretize the space discretizing changes in speed module and direction

10. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 10 Behavioral model DCM are disaggregate behavioral models designed to forecast the behavior of individuals in choice situations DCM Choice Set Alternative’s attributes Socio-economic attributes Random variable Choice Set Alternative’s attributes dm attributes Random variable

11. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 11 Behavioral model Choice Set Alternative’s attributes dm attributes Random variable 111 12 22 2333 i j

12. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 12 Behavioral model Choice Set Alternative’s attributes dm attributes Random variable destination D current direction d O CjCj

13. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 13 Behavioral model Choice Set Alternative’s attributes dm attributes Random variable Cj d_1j d_2j d_3jd_kj k 3 2 1 i

14. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 14 Behavioral model Choice Set Alternative’s attributes dm attributes Random variable Dummy variables capturing the attractiveness of acc / dec. We postulate they vary with the current speed of dm

15. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 15 Behavioral model : the CNL formulation Choice Set Alternative’s attributes dm attributes Random variable ACC DEC CONST Correlation based on speed C NOT C Correlation based on direction

16. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 16 Behavioral model : the mixed NL formulation Choice Set Alternative’s attributes Socio-economic attributes Random variable ACC DEC CONST Correlation based on the speedCorrelation based on the direction NL Error structure Mixed Nested Logit Mixed Nested Logit Unconditional choice probability Joint density of the factors Smooth and unbiased estimator Draw from factor’s distribution

17. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 17 Data collection Data are collected from a training sequence with a frame rate of 10 fps We have manually tracked 36 pedestrians using a monocular calibrated camera, storing the top view positions at each observation. We have globally 1675 position observations, with a time interval of 3 frames (0.3 seconds). The observed choice has been measured three steps forward in time (0.9 seconds). Observations corresponding to static pedestrians and to observed choices out of the choice set have been removed. Finally, we calibrate the model on 1410 observations.

18. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 18 Data collection (Tanaboriboon,Y.,Hwa,S. and Chor,C.(1986).Pedestrian characteristic study in Singapore, Journal of Transportation Engineering 112:229-235) Trajectory lengths8 – 98 pts

19. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 19 …looking inside collected data… We manually track 36 pedestrians for a total of 1410 position observations The frequency of choices 1 2 … 10 11,,, 1222 2333 6 17 28

20. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 20 Estimation results: CNL Variable number Variable name Coefficients estimates Asymptotic std error t-test 0 t-test 1 1 -1.5345 e-00 +5.2903 e-01 - 2.9005 e+00 2 -7.9454 e-02 +5.2485 e-03 - 1.5138 e+01 3 - 5.3839 e-02 +1.9406 e-08 - 2.7742 e+06 4 - 2.4751 e+01 + 4.7804 e+00 - 5.1775 e+00 5 - 2.5543 e-01 + 5.8682 e-02 - 4.3528 e+00 6 +1.6859 e+00 + 1.3083 e-01 + 1.2886 e+01 7 - 8.9047 e-01 + 1.3054 e-01 - 6.8212 e+00 8 +1.0805 e+00 +1.3879 e-01 +7.7848 e+00 +5.8025 e-01 9 +2.5210 e+00 +6.0733 e-01 +4.1509 e+00 +2.5044 e+00 10 +1.4130 e+00 +9.0226 e-02 +1.5660 e+01 +4.5775 e+00 Init log_lik. = - 4979.03Final log_lik. = - 2566.05 Rho-square = - 0.484629 11 +2.4714 e+01 +8.0513 e-02 +3.0695 e+02 +2.9453 e+00 12 +1.3271 e+00 +1.8426 e-01 +7.2024 e+00 +1.7754 e+00

21. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 21 Estimation results: mixed NL Variable number Variable name Coefficients estimates Asymptotic std error t-test 0 t-test 1 Utility parameters: 1 - 1.5051e-01 +5.6931e-02 - 2.6438e+00 2 - 5.2491e-02 +9.6579e-03 - 5.4350e+00 3 - 4.0572e-02 +5.0527e-03 - 8.0298e+00 4 - 3.0922e+01 +7.1640e+00 - 4.3163e+00 5 - 6.5567e-01 +1.1817e-01 - 5.5481e+00 6 +1.7525e+00 +1.7028e-01 +1.0292e+01 7 - 7.9586e-01 +9.3527e-02 - 8.5094e+00 Init log_lik. = - 4930.08Final log_lik. = - 3384.94 Rho-square = - 0.313411..

22. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 22 Estimation results: mixed NL Variable number Variable name Coefficients estimates Asymptotic std error t-test 0 t-test 1 Utility parameters: 8 +1.8573e+00 +3.8892e-01 +4.7757e+00 9 - 1.5691e+00 +5.5359e-01 - 2.8345e+00 10 - 1.0134e+00 +4.8586e-01 - 2.0858e+00 11 +6.6238e-01 +1.8646e-01 +3.5523e+00 12 +5.9938e-01 +2.6174e-01 +2.2899e+00 13 +1.0150e+00 +2.6239e-01 +3.8684e+00 14 +2.6667e+00 +7.4026e-01 +3.6024e+00 Init log_lik. = - 4930.08Final log_lik. = - 3384.94 Rho-square = - 0.313411 15 +2.5289e+00 +4.9287e-01 +5.1308e+00 Model parameters: 16 +1.4235e+00 +1.7582e-01 +8.0963e+00 +2.4087e+00

23. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 23 Pedestrian simulator Developed by Mats Weber in the context of the CTI project SIMBAD Is initialized with a time-dependent origin-destination matrix. An itinerary is associated with each agent. At each step the utilities and probabilities are calculated (red=high utility, blue=low utility)

24. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 24 Application of DCM to pedestrian detection and tracking

25. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 25 Pedestrian detection An hypothetical moving object Grid on top-view plane camera Projection of hypothetical moving objects (first frame) We assume an average target height of 1.70m Detection of new moving objects in the active zones (each refresh period)

26. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 26 Pedestrian detection Original image Background imageForeground image An hypothetical moving object Grid on top-view plane camera Projection of hypothetical moving objects (first frame) We assume an average target height of 1.70m

27. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 27 Pedestrian detection Next frame target correlation Correlation matrix

28. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 28 Pedestrian detection Pre-filtering: simple thresholding on the visual displacements projected on the top-view plane. An activation value (starting score) is given to each hypthesis. Each bad step consist in a penalty. Filtering: we use the model’s probabilities to give scores to the trajectories over a period of T frames

29. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 29 Trajectories filtering and detection results Accepted trajectoriesUnfiltered trajectories:Filtered trajectories:All trajectories Frame 20 Frame 35Frame 50Frame 65

30. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 30 Trajectories filtering and detection results

31. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 31 Pedestrian tracking The first approach is to treat tracking as a sequence of detection cycles: deterministic template matching and behavioral filtering.

32. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 32 Pedestrian tracking We use the model as a prior and the normalized image correlation as likelihood ( at each step the model is propagated from a MAP estimation on the previous posterior )

33. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 33 Pedestrian tracking In typical tracking problems, at each frame we have a model for the target’s motion and a measurement from the image, represented by the likelihood term. Normalized correlation between the current template and the target image DCM probabilities ( at each step the dynamic model is propagated from a MAP estimation on the previous posterior )

34. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 34 Results

35. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 35 Conclusions & future works DCM are flexible and efficient for pedestrian modeling; The use of behavioral models is usefull both for detection and tracking. Can help to solve occlusion and illumination condition related problems. To do: - Better representations for the posterior distribution and the likelihood term; - DCM has to be extended to high density scenarios with an explicit model for fixed and moving obstacles; - We are currently working on a post-clustering of trajectories to integrate at the end of each detection step. Interesting preliminary results for pedestrians’ calculation.

36. Signal Processing Institute Swiss Federal Institute of Technology, Lausanne 36 Future works The probabilistic approach to tracking has to be improved including better representations for the posterior distribution and the likelihood term ( multimodality in the correlation distribution ). We are currently working on a post-clustering of trajectories to integrate at the end of each detection step. Interesting preliminary results for pedestrians’ calculation. We are working on a scale-adaptive head-detector, based on statistical properties of edges’ curvature, to use in the initialization step and/or for the validation of the likelihood modes. DCM has to be extended to high density scenarios with an explicit model for fixed and moving obstacles.