1. Computer Vision Linear Tracking Jan-Michael Frahm COMP 256 Some slides from Welch & Bishop

2. Computer Vision 2 Tracking Tracking is the problem of generating an inference about the motion of an object given a sequence of images. The key technical difficulty is maintaining an accurate representation of the posterior on object position given measurements, and doing so efficiently.

3. Computer Vision 3 Examples of tracking

4. Computer Vision 4 Model for tracking Object has internal state –Capital indicates random variable –Small represents particular value Obtained measurements in frame i are –Value of the measurement

5. Computer Vision 5 General Steps of Tracking 1.Prediction: What is the next state of the object given past measurements 2.Data association: Which measures are relevant for the state? 3.Correction: Compute representation of the state from prediction and measurements.

6. Computer Vision 6 Tracking predict correct

7. Computer Vision 7 Only immediate past matters Measurements depend only on current state Important simplifications Fortunately it doesn’t limit to much! Independence Assumptions

8. Computer Vision 8 Linear Dynamic Models State is linear transformed plus Gaussian noise Relevant measures are linearly obtained from state plus Gaussian noise Sufficient to maintain mean and standard deviation

9. Computer Vision 9 A really simple example We are on a boat at night and lost our position We know: star position

10. Computer Vision 10 Constant Velocity p is position of boat, v is velocity of boat state is We only measure position so

11. Computer Vision 11 Marc makes a measurement 14 121086420-2, Conditional Density Function

12. Computer Vision 12 Jan makes a measurement Conditional Density Function 14 121086420-2,

13. Computer Vision 13 Combine measurements & variances 14 121086420-2 Conditional Density Function Online weighted average!

14. Computer Vision 14 Rudolf Emil Kalman Born 1930 in Hungary BS and MS from MIT PhD 1957 from Columbia Filter developed in 1960-61 Now retired

15. Computer Vision 15 Kalman filter Just some applied math. A linear dynamic system: f(a+b) = f(a) + f(b) Noisy data in  hopefully less noisy out. But delay is the price for filtering...

16. Computer Vision 16 Predict  Correct KF operates by 1.Predicting the new state and its uncertainty 2.Correcting with the new measurement predict correct

17. Computer Vision 17 What is it used for? Tracking missiles Tracking heads/hands/drumsticks Extracting lip motion from video Fitting Bezier patches to point data Lots of computer vision applications Economics Navigation

18. Computer Vision 18 A really simple example We are on a boat at night and lost our position We know: move with constant velocity star position

19. Computer Vision 19 But suppose we’re moving Not all the difference is error. Some may be motion KF can include a motion model Estimate velocity and position 14 121086420-2

20. Computer Vision 20 Process Model Describes how the state changes over time The state for the first example was scalar The process model was “nothing changes” A better model might be constant velocity motion

21. Computer Vision 21 Measurement Model “What you see from where you are” not “Where you are from what you see”

22. Computer Vision 22 Constant Velocity p is position of boat, v is velocity of boat state is We only measure position so

23. Computer Vision 23 State and Error Covariance First two moments of Gaussian process Error Covariance Process State (Mean)

24. Computer Vision 24 The Process Model Uncertainty over interval State transition Process dynamics Difficult to determine

25. Computer Vision 25 Measurement Model Measurement uncertainty Measurement matrix Measurement matrix Measurement relationship to state

26. Computer Vision 26 Predict (Time Update) a priori state, error covariance, measurement

27. Computer Vision 27 Measurement Update (Correct) Kalman gain a posteriori state and error covariance Minimizes posteriori error covariance

28. Computer Vision 28 The Kalman Gain Weights between prediction and measurements to posteriori error covariance For no measurement uncertainty State is deduced only from measurement

29. Computer Vision 29 The Kalman Gain Simple univariate (scalar) example a posteriori state and error covariance

30. Computer Vision 30 Summary PREDICT CORRECT

31. Computer Vision 31 Estimating a Constant The state transition matrix The measurement matrix Prediction

32. Computer Vision 32 Measurement Update

33. Computer Vision 33 Setup/Initialization

34. Computer Vision 34 State and Measurements = 0.1

35. Computer Vision 35 Error Covariance

36. Computer Vision 36 State and Measurements = 1

37. Computer Vision 37 State and Measurements = 0.01

38. Computer Vision 38 Example camera pose estimation

39. Computer Vision 39 Kalman Filter Web Site http://www.cs.unc.edu/~welch/kalman/ Electronic and printed references –Book lists and recommendations –Research papers –Links to other sites –Some software News

40. Computer Vision 40 Java-Based KF Learning Tool On-line 1D simulation Linear and non-linear Variable dynamics http://www.cs.unc.edu/~welch/kalman/

41. Computer Vision 41 KF Course Web Page http://www.cs.unc.edu/~tracker/ref/s2001/kalman/index.html ( http://www.cs.unc.edu/~tracker/ )http://www.cs.unc.edu/~tracker/ Java-Based KF Learning Tool KF web page

42. Computer Vision 42 Closing Remarks Try it! –Not too hard to understand or program Start simple –Experiment in 1D –Make your own filter in Matlab, etc. Note: the Kalman filter “wants to work” –Debugging can be difficult –Errors can go un-noticed

43. Computer Vision 43 Relevant References Azarbayejani, Ali, and Alex Pentland (1995). “Recursive Estimation of Motion, Structure, and Focal Length,” IEEE Trans. Pattern Analysis and Machine Intelligence 17(6): 562-575. Dellaert, Frank, Sebastian Thrun, and Charles Thorpe (1998). “Jacobian Images of Super- Resolved Texture Maps for Model-Based Motion Estimation and Tracking,” IEEE Workshop on Applications of Computer Vision (WACV'98), October, Princeton, NJ, IEEE Computer Society. http://mac-welch.cs.unc.edu/~welch/COMP256/

44. Computer Vision 44 Example: Constant Velocity