2. Stanford CS223B Computer Vision, Winter 2007 Lecture 12 Tracking Motion Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens

3. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

4. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Tracking Problem n Can we estimate the position of the object? n Can we estimate its velocity? n Can we predict future positions? Image 4Image 1Image 2Image 3

5. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Tracking Problem n Given Sequence of Images n Find center of moving object n Camera might be moving or stationary n We assume: We can find object in individual images. n The Problem: Track across multiple images. n Is a fundamental problem in computer vision

6. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Methods Bayes Filter Particle Filter Uncented Kalman Filter Kalman Filter Extended Kalman Filter

7. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Further Reading…

8. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example: Moving Object

9. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

10. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Slow Down! Example of Bayesian Inference ? Environment prior p(staircase) = 0.1 Bayesian inference p(staircase | image) p(image | staircasse) p(staircase) p(im | stair) p(stair) + p(im | no stair) p(no stair) = 0.7  0.1 / (0.7  0.1 + 0.2  0.9) = 0.28 Sensor model p(image | staircase) = 0.7 p(image | no staircase) = 0.2 p(staircase) = 0.28 Cost model cost(fast walk | staircase) = $1,000 cost(fast walk | no staircase) = $0 cost(slow+sense) = $1 Decision Theory E[cost(fast walk)] = $1,000  0.28 = $280 E[cost(slow+sense)] = $1 =

11. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filter Definition n Environment state x t n Measurement z t n Can we calculate p(x t | z 1, z 2, …, z t ) ?

12. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters Illustrated Localization problem

13. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters: Essential Steps n Belief: Bel(x t ) n Measurement update: Bel(x t )  Bel(x t ) p(z t |x t ) n Time update: Bel(x t+1 )  Bel(x t )  p(x t+1 |u t,x t ) 

14. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 x = state t = time z = observation u = action  = constant Bayes Filters Markov Bayes Markov

15. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters x = state t = time z = observation u = action

16. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters Illustrated

17. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters n Initial Estimate of State n Iterate –Receive measurement, update your belief (uncertainty shrinks) –Predict, update your belief (uncertainty grows)

18. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Methods Bayes Filter Particle Filter Uncented Kalman Filter Kalman Filter Extended Kalman Filter

19. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

20. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters: Basic Idea (equality for ) set of n particles X t

21. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filter Explained

22. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Basic Particle Filter Algorithm Initialization: X 0  n particles x 0 [i] ~ p(x 0 ) particleFilters( X t  1 ){ for i=1 to n x t [i] ~ p(x t | x t  1 [i] ) (prediction) w t [i] = p(z t | x t [i] ) (importance weights) endfor for i=1 to n include x t [i] in X t with probability  w t [i] (resampling) }

23. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Weight samples: w = f / g Importance Sampling

24. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filter By Frank Dellaert

25. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Case Study: Track moving objects from Helicopter 1. Harris Corners 2. Optical Flow (with clustering) 3. Motion likelihood function 4. Particle Filter 5. Centroid Extraction David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

26. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 More Particle Filter Tracking David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

27. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Some Robotics Examples n Tracking Hands, People n Mobile Robot localization n People localization n Car localization n Mapping

28. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Examples Particle Filter Siu Chi Chan McGill University

29. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Another Example Mike Isard and Andrew Blake

30. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Tracking Fast moving Objects K. Toyama, A.Blake

31. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters (1)

32. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters (2)

33. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particles = Robustness

34. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters for Tracking Cars With Anya Petrovskaya

35. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

36. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Tracking with KFs: Gaussians! updateinitial estimate x y x y prediction x y measurement x y

37. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filters

38. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filters prior Measurement evidence posterior

39. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 A Quiz prior Measurement evidence posterior?

40. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Gaussians --   Univariate  Multivariate

41. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Properties of Univariate Gaussians

42. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Measurement Update Derived

43. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 n We stay in the “Gaussian world” as long as we start with Gaussians and perform only linear transformations. Properties Multivariate Gaussians Essentially the same as in the 1-D case, but with more general notation

44. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Linear Kalman Filter Estimates the state x of a discrete-time controlled process that is governed by the linear stochastic difference equation with a measurement

45. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Components of a Kalman Filter Matrix (n  n) that describes how the state evolves from t to t+1 without controls or noise. Matrix (n  i) that describes how the control u t changes the state from t to t+1. Matrix (k  n) that describes how to map the state x t to an observation z t. Random variables representing the process and measurement noise that are assumed to be independent and normally distributed with covariance R t and Q t respectively.

46. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Algorithm 1. Algorithm Kalman_filter(  t-1,  t-1, u t, z t ): 2. Prediction: 3. 4. 5. Correction: 6. 7. 8. 9. Return  t,  t

47. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates in 1D old belief measurement belief new belief belief measurement

48. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates in 1D old belief new belief measurement

49. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates in 1D old belief new belief measurement new belief newest belief

50. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates belief latest belief belief measurement belief

51. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Prediction-Correction-Cycle Prediction

52. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Prediction-Correction-Cycle Correction

53. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Prediction-Correction-Cycle Correction Prediction

54. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Summary n Highly efficient: Polynomial in measurement dimensionality k and state dimensionality n: O(k 2.376 + n 2 ) n Optimal for linear Gaussian systems! n Most robotics systems are nonlinear!

55. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

56. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Let’s Apply KFs to Tracking Problem Image 4Image 1Image 2Image 3

57. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter with 2-Dim Linear Model n Linear Change (Motion) n Linear Measurement Model What is A, B, Q? What is C, D, R?

58. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Algorithm 1. Algorithm Kalman_filter(  t-1,  t-1, u t, z t ): 2. Prediction: 3. 4. 5. Correction: 6. 7. 8. 9. Return  t,  t

59. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter in Detail n Measurements n Change n Prediction n Measurement Update updateinitial position x y x y prediction x y measurement x y

60. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Can We Do Better?

61. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman, Better! initial positionpredictionmeasurement next prediciton update

62. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 We Can Estimate Velocity! past measurements prediction

63. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter For 2D Tracking n Linear Measurement model (now with 4 state variables) n Linear Change

64. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Putting It Together Again n Measurements n Change n Prediction n Measurement Update

65. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary Kalman Filter n Estimates state of a system –Position –Velocity –Many other continuous state variables possible n KF maintains –Mean vector for the state –Covariance matrix of state uncertainty n Implements –Time update = prediction –Measurement update n Standard Kalman filter is linear-Gaussian –Linear system dynamics, linear sensor model –Additive Gaussian noise (independent) –Nonlinear extensions: extended KF, unscented KF: linearize n More info: –CS226 –Probabilistic Robotics (Thrun/Burgard/Fox, MIT Press)

66. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary Kalman Filter n Estimates state of a system –Position –Velocity –Many other continuous state variables possible n KF maintains –Mean vector for the state –Covariance matrix of state uncertainty n Implements –Time update = prediction –Measurement update n Standard Kalman filter is linear-Gaussian –Linear system dynamics, linear sensor model –Additive Gaussian noise (independent) –Nonlinear extensions: extended KF, unscented KF: linearize n More info: –CS226 –Probabilistic Robotics (Thrun/Burgard/Fox, MIT Press) Particle and discrete set of particles (example states) = predictive sampling = resampling, importance weights fully nonlinear easy to implement

67. Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters