Computer Vision Linear Tracking Jan-Michael Frahm COMP 256 Some slides from Welch & Bishop

Computer Vision 2 Tracking Tracking is the problem of generating an inference about the motion of an object given a sequence of images. The key technical difficulty is maintaining an accurate representation of the posterior on object position given measurements, and doing so efficiently.

Computer Vision 3 Examples of tracking

Computer Vision 4 Model for tracking Object has internal state –Capital indicates random variable –Small represents particular value Obtained measurements in frame i are –Value of the measurement

Computer Vision 5 General Steps of Tracking 1.Prediction: What is the next state of the object given past measurements 2.Data association: Which measures are relevant for the state? 3.Correction: Compute representation of the state from prediction and measurements.

Computer Vision 6 Tracking predict correct

Computer Vision 7 Only immediate past matters Measurements depend only on current state Important simplifications Fortunately it doesn’t limit to much! Independence Assumptions

Computer Vision 8 Linear Dynamic Models State is linear transformed plus Gaussian noise Relevant measures are linearly obtained from state plus Gaussian noise Sufficient to maintain mean and standard deviation

Computer Vision 9 A really simple example We are on a boat at night and lost our position We know: star position

Computer Vision 10 Constant Velocity p is position of boat, v is velocity of boat state is We only measure position so

Computer Vision 11 Marc makes a measurement , Conditional Density Function

Computer Vision 12 Jan makes a measurement Conditional Density Function ,

Computer Vision 13 Combine measurements & variances Conditional Density Function Online weighted average!

Computer Vision 14 Rudolf Emil Kalman Born 1930 in Hungary BS and MS from MIT PhD 1957 from Columbia Filter developed in Now retired

Computer Vision 15 Kalman filter Just some applied math. A linear dynamic system: f(a+b) = f(a) + f(b) Noisy data in  hopefully less noisy out. But delay is the price for filtering...

Computer Vision 16 Predict  Correct KF operates by 1.Predicting the new state and its uncertainty 2.Correcting with the new measurement predict correct

Computer Vision 17 What is it used for? Tracking missiles Tracking heads/hands/drumsticks Extracting lip motion from video Fitting Bezier patches to point data Lots of computer vision applications Economics Navigation

Computer Vision 18 A really simple example We are on a boat at night and lost our position We know: move with constant velocity star position

Computer Vision 19 But suppose we’re moving Not all the difference is error. Some may be motion KF can include a motion model Estimate velocity and position

Computer Vision 20 Process Model Describes how the state changes over time The state for the first example was scalar The process model was “nothing changes” A better model might be constant velocity motion

Computer Vision 21 Measurement Model “What you see from where you are” not “Where you are from what you see”

Computer Vision 22 Constant Velocity p is position of boat, v is velocity of boat state is We only measure position so

Computer Vision 23 State and Error Covariance First two moments of Gaussian process Error Covariance Process State (Mean)

Computer Vision 24 The Process Model Uncertainty over interval State transition Process dynamics Difficult to determine

Computer Vision 25 Measurement Model Measurement uncertainty Measurement matrix Measurement matrix Measurement relationship to state

Computer Vision 26 Predict (Time Update) a priori state, error covariance, measurement

Computer Vision 27 Measurement Update (Correct) Kalman gain a posteriori state and error covariance Minimizes posteriori error covariance

Computer Vision 28 The Kalman Gain Weights between prediction and measurements to posteriori error covariance For no measurement uncertainty State is deduced only from measurement

Computer Vision 29 The Kalman Gain Simple univariate (scalar) example a posteriori state and error covariance

Computer Vision 30 Summary PREDICT CORRECT

Computer Vision 31 Estimating a Constant The state transition matrix The measurement matrix Prediction

Computer Vision 32 Measurement Update

Computer Vision 33 Setup/Initialization

Computer Vision 34 State and Measurements = 0.1

Computer Vision 35 Error Covariance

Computer Vision 36 State and Measurements = 1

Computer Vision 37 State and Measurements = 0.01

Computer Vision 38 Example camera pose estimation

Computer Vision 39 Kalman Filter Web Site Electronic and printed references –Book lists and recommendations –Research papers –Links to other sites –Some software News

Computer Vision 40 Java-Based KF Learning Tool On-line 1D simulation Linear and non-linear Variable dynamics

Computer Vision 41 KF Course Web Page ( ) Java-Based KF Learning Tool KF web page

Computer Vision 42 Closing Remarks Try it! –Not too hard to understand or program Start simple –Experiment in 1D –Make your own filter in Matlab, etc. Note: the Kalman filter “wants to work” –Debugging can be difficult –Errors can go un-noticed

Computer Vision 43 Relevant References Azarbayejani, Ali, and Alex Pentland (1995). “Recursive Estimation of Motion, Structure, and Focal Length,” IEEE Trans. Pattern Analysis and Machine Intelligence 17(6): Dellaert, Frank, Sebastian Thrun, and Charles Thorpe (1998). “Jacobian Images of Super- Resolved Texture Maps for Model-Based Motion Estimation and Tracking,” IEEE Workshop on Applications of Computer Vision (WACV'98), October, Princeton, NJ, IEEE Computer Society.

Computer Vision 44 Example: Constant Velocity