Stanford CS223B Computer Vision, Winter 2007 Lecture 12 Tracking Motion Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Tracking Problem n Can we estimate the position of the object? n Can we estimate its velocity? n Can we predict future positions? Image 4Image 1Image 2Image 3

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Tracking Problem n Given Sequence of Images n Find center of moving object n Camera might be moving or stationary n We assume: We can find object in individual images. n The Problem: Track across multiple images. n Is a fundamental problem in computer vision

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Methods Bayes Filter Particle Filter Uncented Kalman Filter Kalman Filter Extended Kalman Filter

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Further Reading…

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Example: Moving Object

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Slow Down! Example of Bayesian Inference ? Environment prior p(staircase) = 0.1 Bayesian inference p(staircase | image) p(image | staircasse) p(staircase) p(im | stair) p(stair) + p(im | no stair) p(no stair) = 0.7  0.1 / (0.7   0.9) = 0.28 Sensor model p(image | staircase) = 0.7 p(image | no staircase) = 0.2 p(staircase) = 0.28 Cost model cost(fast walk | staircase) = $1,000 cost(fast walk | no staircase) = $0 cost(slow+sense) = $1 Decision Theory E[cost(fast walk)] = $1,000  0.28 = $280 E[cost(slow+sense)] = $1 =

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filter Definition n Environment state x t n Measurement z t n Can we calculate p(x t | z 1, z 2, …, z t ) ?

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters Illustrated Localization problem

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters: Essential Steps n Belief: Bel(x t ) n Measurement update: Bel(x t )  Bel(x t ) p(z t |x t ) n Time update: Bel(x t+1 )  Bel(x t )  p(x t+1 |u t,x t ) 

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 x = state t = time z = observation u = action  = constant Bayes Filters Markov Bayes Markov

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters x = state t = time z = observation u = action

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters Illustrated

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Bayes Filters n Initial Estimate of State n Iterate –Receive measurement, update your belief (uncertainty shrinks) –Predict, update your belief (uncertainty grows)

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Methods Bayes Filter Particle Filter Uncented Kalman Filter Kalman Filter Extended Kalman Filter

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters: Basic Idea (equality for ) set of n particles X t

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filter Explained

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Basic Particle Filter Algorithm Initialization: X 0  n particles x 0 [i] ~ p(x 0 ) particleFilters( X t  1 ){ for i=1 to n x t [i] ~ p(x t | x t  1 [i] ) (prediction) w t [i] = p(z t | x t [i] ) (importance weights) endfor for i=1 to n include x t [i] in X t with probability  w t [i] (resampling) }

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Weight samples: w = f / g Importance Sampling

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filter By Frank Dellaert

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Case Study: Track moving objects from Helicopter 1. Harris Corners 2. Optical Flow (with clustering) 3. Motion likelihood function 4. Particle Filter 5. Centroid Extraction David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 More Particle Filter Tracking David Stavens, Andrew Lookingbill, David Lieb, CS223b Winter 2004

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Some Robotics Examples n Tracking Hands, People n Mobile Robot localization n People localization n Car localization n Mapping

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Examples Particle Filter Siu Chi Chan McGill University

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Another Example Mike Isard and Andrew Blake

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Tracking Fast moving Objects K. Toyama, A.Blake

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters (1)

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters (2)

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particles = Robustness

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Particle Filters for Tracking Cars With Anya Petrovskaya

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Tracking with KFs: Gaussians! updateinitial estimate x y x y prediction x y measurement x y

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filters

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filters prior Measurement evidence posterior

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 A Quiz prior Measurement evidence posterior?

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Gaussians --   Univariate  Multivariate

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Properties of Univariate Gaussians

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Measurement Update Derived

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 n We stay in the “Gaussian world” as long as we start with Gaussians and perform only linear transformations. Properties Multivariate Gaussians Essentially the same as in the 1-D case, but with more general notation

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Linear Kalman Filter Estimates the state x of a discrete-time controlled process that is governed by the linear stochastic difference equation with a measurement

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Components of a Kalman Filter Matrix (n  n) that describes how the state evolves from t to t+1 without controls or noise. Matrix (n  i) that describes how the control u t changes the state from t to t+1. Matrix (k  n) that describes how to map the state x t to an observation z t. Random variables representing the process and measurement noise that are assumed to be independent and normally distributed with covariance R t and Q t respectively.

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Algorithm 1. Algorithm Kalman_filter(  t-1,  t-1, u t, z t ): 2. Prediction: Correction: Return  t,  t

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates in 1D old belief measurement belief new belief belief measurement

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates in 1D old belief new belief measurement

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates in 1D old belief new belief measurement new belief newest belief

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Updates belief latest belief belief measurement belief

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Prediction-Correction-Cycle Prediction

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Prediction-Correction-Cycle Correction

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 The Prediction-Correction-Cycle Correction Prediction

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Summary n Highly efficient: Polynomial in measurement dimensionality k and state dimensionality n: O(k n 2 ) n Optimal for linear Gaussian systems! n Most robotics systems are nonlinear!

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Overview n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Let’s Apply KFs to Tracking Problem Image 4Image 1Image 2Image 3

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter with 2-Dim Linear Model n Linear Change (Motion) n Linear Measurement Model What is A, B, Q? What is C, D, R?

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter Algorithm 1. Algorithm Kalman_filter(  t-1,  t-1, u t, z t ): 2. Prediction: Correction: Return  t,  t

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter in Detail n Measurements n Change n Prediction n Measurement Update updateinitial position x y x y prediction x y measurement x y

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Can We Do Better?

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman, Better! initial positionpredictionmeasurement next prediciton update

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 We Can Estimate Velocity! past measurements prediction

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Kalman Filter For 2D Tracking n Linear Measurement model (now with 4 state variables) n Linear Change

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Putting It Together Again n Measurements n Change n Prediction n Measurement Update

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary Kalman Filter n Estimates state of a system –Position –Velocity –Many other continuous state variables possible n KF maintains –Mean vector for the state –Covariance matrix of state uncertainty n Implements –Time update = prediction –Measurement update n Standard Kalman filter is linear-Gaussian –Linear system dynamics, linear sensor model –Additive Gaussian noise (independent) –Nonlinear extensions: extended KF, unscented KF: linearize n More info: –CS226 –Probabilistic Robotics (Thrun/Burgard/Fox, MIT Press)

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary Kalman Filter n Estimates state of a system –Position –Velocity –Many other continuous state variables possible n KF maintains –Mean vector for the state –Covariance matrix of state uncertainty n Implements –Time update = prediction –Measurement update n Standard Kalman filter is linear-Gaussian –Linear system dynamics, linear sensor model –Additive Gaussian noise (independent) –Nonlinear extensions: extended KF, unscented KF: linearize n More info: –CS226 –Probabilistic Robotics (Thrun/Burgard/Fox, MIT Press) Particle and discrete set of particles (example states) = predictive sampling = resampling, importance weights fully nonlinear easy to implement

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007 Summary n The Tracking Problem n Bayes Filters n Particle Filters n Kalman Filters n Using Kalman Filters