1. 1 Schedule. May 4, 2009 6.869 Monday, May 4: –Lecture on tracking objects and people. Wednesday, May 6: –Lecture on writing papers and giving talks. Monday, May 11: –The intersection of vision and graphics: motion magnification, shapetime photography, motion magnification, and infinite images. Wednesday, May 13: –Final project presentations. Friday, May 15, 5:00pm: –Final written reports accepted until then. Reading on tracking: Forsyth and Ponce, chapter 17.

2. Tracking framework 2 x1x1 x2x2 x3x3 y1y1 y2y2 y3y3 Gauss

3. 3 Kalman filter homepage http://www.cs.unc.edu/~welch/kalman/ (kalman filter demo applet) Kevin Murphy’s Matlab toolbox: http://www.ai.mit.edu/~murphyk/Software/Kalman/ kalman.html Resources

4. Embellishments for tracking Richer models of P(y n |x n ) Richer model of probability distributions 4

5. 5 Jepson, Fleet, and El-Maraghi tracker

6. 6 Wandering, Stable, and Lost appearance model Introduce 3 competing models to explain the appearance of the tracked region: –A stable model—Gaussian with some mean and covariance. –A 2-frame motion tracker appearance model, to rebuild the stable model when it gets lost –An outlier model—uniform probability over all appearances. Use an on-line EM algorithm to fit the (changing) model parameters to the recent appearance data.

7. 7 Jepson, Fleet, and El-Maraghi tracker for toy 1-pixel image model Red line: observations Blue line: true appearance state Black line: mean of the stable process

8. 8 Mixing probabilities for Stable (black), Wandering (red) and Lost (green) processes

9. Non-toy image representation Phase of a steerable quadrature pair (G2, H2). Steered to 4 different orientations, at 2 scales. 9

10. 10 The motion tracker Motion prior,, prefers slow velocities and small accelerations. The WSL appearance model gives a likelihood for each possible new position, orientation, and scale of the tracked region. They combine that with the motion prior to find the most probable position, orientation, and scale of the tracked region in the next frame. Gives state-of-the-art tracking results.

11. 11 Jepson, Fleet, and El-Maraghi tracker

12. 12 Add fleet&jepson tracking slides Jepson, Fleet, and El-Maraghi tracker Far right column: the stable component’s mixing probability. Note its behavior at occlusions.

13. 13 Show videos Play freeman#results.mpg

14. Embellishments for tracking Richer model of P(y n |x n ) Richer model of probability distributions –Particle filter models, applied to tracking humans 14

15. 15 (KF) Distribution propagation [Isard 1998] prediction from previous time frame Noise added to that prediction Make new measurement at next time frame

16. 16 Distribution propagation [Isard 1998]

17. 17 6.869 Computer Vision Recursive filtering for tracking--Kalman filtering and particle filtering. April 11, 2011 Bill Freeman and Antonio Torralba 17

18. 18 See slide notes for material to be presented on board. 18

19. 17 Representing non-linear Distributions

20. 18 Representing non-linear Distributions Unimodal parametric models fail to capture real- world densities…

21. 19 Discretize by evenly sampling over the entire state space Tractable for 1-d problems like stereo, but not for high-dimensional problems.

22. 20 Representing Distributions using Weighted Samples Rather than a parametric form, use a set of samples to represent a density:

23. 21 Representing Distributions using Weighted Samples Rather than a parametric form, use a set of samples to represent a density: Sample positionsProbability mass at each sample This gives us two knobs to adjust when representing a probability density by samples: the locations of the samples, and the probability weight on each sample.

24. 22 [Isard 1998] Representing distributions using weighted samples, another picture

25. 23 Sampled representation of a probability distribution You can also think of this as a sum of dirac delta functions, each of weight w:

26. Tracking, in particle filter representation 24 x1x1 x2x2 x3x3 y1y1 y2y2 y3y3 Prediction step Update step

27. 25 Particle filter Let’s apply this sampled probability density machinery to generalize the Kalman filtering framework. More general probability density representation than uni-modal Gaussian. Allows for general state dynamics, f(x) + noise

28. 26 Sampled Prediction = ? ~= Drop elements to marginalize to get

29. 27 Sampled Correction (Bayes rule) Prior  posterior Reweight every sample with the likelihood of the observations, given that sample: yielding a set of samples describing the probability distribution after the correction (update) step:

30. 28 Naïve PF Tracking Start with samples from something simple (Gaussian) Repeat But doesn’t work that well because of sample impoverishment… – Predict – Correct s Take each particle from the prediction step and modify the old weight by multiplying by the new likelihood Run every particle through the dynamics function and add noise.

31. 29 10 of the 100 particles, along with the true Kalman filter track, with variance: Sample impoverishment time

32. 30 In a sampled density representation, the frequency of samples can be traded off against weight: These new samples are a representation of the same density. I.e., make N draws with replacement from the original set of samples, using the weights as the probability of drawing a sample. Resample the prior s.t. …

33. 31 Resampling concentrates samples

34. 32 A practical particle filter with resampling

35. 33 Pictorial view [Isard 1998]

36. 34

37. 35

38. 36 Animation of condensation algorithm [Isard 1998]

39. 37 Tracking –hands –bodies –Leaves What might we expect? Reliable, robust, slow Applications

40. 38 Contour tracking [Isard 1998]

41. 39 Head tracking [Isard 1998] Picture of the states represented by the top weighted particles The mean state

42. 40 Leaf tracking [Isard 1998]

43. 41 Hand tracking [Isard 1998]

44. Probabilistic Tracking and Reconstruction of 3D Human Motion in Monocular Video Sequences Presentation of the thesis work of: Hedvig Sidenbladh, KTH Thesis opponent: Prof. Bill Freeman, MIT

45. Thesis supervisors Prof. Jan-Olof Eklundh, KTH Prof. Michael Black, Brown University Dr. David Fleet, Xerox PARC Prof. Dirk Ormoneit, Stanford University Collaborators

46. Models of Human Dynamics Action-specific model - Walking –Training data: 3D motion capture data –From training set, learn mean cycle and common modes of deviation (PCA) Mean cycleSmall noiseLarge noise

47. Walking Person Walking model 2500 samples ~10 min/frame #samples from 15000 to 2500 by using the learned likelihood

48. No likelihood * how strong is the walking prior? (or is our likelihood doing anything?)

49. 42 Go to hedvig slides

50. 43 Desired operations with a probability density Expected value Marginalization Bayes rule

51. 44 Computing an expectation using sampled representation using

52. 45 Marginalizing a sampled density If we have a sampled representation of a joint density and we wish to marginalize over one variable: we can simply ignore the corresponding components of the samples (!):

53. 46 Marginalizing a sampled density

54. 47 Sampled Bayes rule k p(V=v 0 |U)p(U)