1. Object Tracking - Slide 1 Object Tracking Computer Vision Course Presentation by Wei-Chao Chen April 05, 2000

2. Object Tracking - Slide 2 Overview What is object tracking?What is object tracking? –Camera or/and objects are moving with respect to each other. –Using knowledge of current state of objects and new input image(s), calculate the new position of objects. –Real-time performance requirements.

3. Object Tracking - Slide 3 Overview Applications of Object TrackingApplications of Object Tracking –Behavioral Analysis of objects or human, such as gesture analysis, facial expression analysis, etc. –Robotics such as autonomous vehicle. –Smart Camera such as video surveillance, etc.

4. Object Tracking - Slide 4 Overview Tracking with automatic model switching Facial expression tracking separated from pose.

5. Object Tracking - Slide 5 Overview Tracking using Hermes curve evolution and PDE for incremental updates. Video Surveillance with behavioral model and Bayesian networks.

6. Object Tracking - Slide 6 Overview Research focuses on tracking:Research focuses on tracking: –Feature detection: »Contour detection, »Energy Minimization. –Recursion algorithm. –Adaptive modeling for enhanced prediction. –Modeling techniques for behavioral analysis.

7. Object Tracking - Slide 7 Related Topics Camera Position EstimationCamera Position Estimation –With moving camera with known internal parameters, extract the camera motion. static object C t1 C t2 E 1,2 p t1 1 p t1 2 p t2 1 p t2 2 For each i, we have: Find more than 8 points and the essential matrix E 1,2 is solved.

8. Object Tracking - Slide 8 Related Topics Optical Flow:Optical Flow: –The velocity field of each points on the images. –Can be used as input for object tracking. –Search for similar flow vectors to locate the tracked objects.

9. Object Tracking - Slide 9 Mathematics for Object Tracking Problem Statement:Problem Statement: –Given certain measurements in the system, solve for the system states. »Example: In HiBall tracker, given the orientation of beacon related to the tracker (measurements), calculate the orientation/location of the tracker (system states). –Both measurements and system states might contain error.

10. Object Tracking - Slide 10 Mathematics for Object Tracking System Measuring Device uncertainty measurement error aiai vivi xixi w i-1 g(..) previous state(s) f(..)

11. Object Tracking - Slide 11 Mathematics for Object Tracking Notations & Definitions:Notations & Definitions: –a –a i : state vector of the system at instance i. – –x i : measurement vector at instance i. – –f i (x i, a i, v i ) = 0 : measurement equation, where v i is the measurement uncertainty. – –a i = g(a i-1, w i-1 ) : state model of the system, where w i represents the uncertainty of the model at instance i. a.k.a. “plant equation”.

12. Object Tracking - Slide 12 Mathematics for Object Tracking Deterministic least-squares methodDeterministic least-squares method –Given enough measurements, over-constraint the system to find the unique state solution. Recursive least-squares methodRecursive least-squares method –Given under-constraint measurements and previous states, find the next system state. Kalman filtering methodKalman filtering method –Also a recursive method. –Optimal solution of given constraints.

13. Object Tracking - Slide 13 Deterministic Methods Example:Example: –For linear measurements with constant state: »f(x, a)= x - Ha = 0, »f(x, a) = x - Ha = 0, –Goal: optimize the function »f T (x, a) f (x, a). –The solution of state is found by: »â = (H T H) -1 H T x. –The pseudo-inverse of H –The pseudo-inverse of H results in the least- square solution of a.

14. Object Tracking - Slide 14 Deterministic Method Advantage:Advantage: –The method is conceptually easy. Disadvantage:Disadvantage: –Need to keep records of enough numbers of measurements. –Difficult to solve in practice »large set of linear equations in the above example need to be solved.

15. Object Tracking - Slide 15 Recursive Method Address the problem of deterministic method.Address the problem of deterministic method. –We want to have incremental method. –We don’t want to keep lots of earlier states. –We don’t want to wait until all measurements arrives.

16. Object Tracking - Slide 16 Recursive Method Again, let f i (x i, a)be the measurement function.Again, let f i (x i, a) be the measurement function. = x i - H i a i = 0. Assume f is linear and let f i = x i - H i a i = 0. Let We have H iT H i = H (i-1)T H i-1 + H i T H i Let P i = (H iT H i ) -1

17. Object Tracking - Slide 17 Recursive Method Optimize the following equation: We shall eventally get: The above equation shows the updated state â i using only previous state â i-1 and measurement matrix H i.

18. Object Tracking - Slide 18 Kalman Filter Method How about noise in the system and measurements?How about noise in the system and measurements? –There will always be errors in measurements: »Imprecise instrumentation, »Ambiguity, »Finite resolution of measurements. –Model of the system g() might not be accurate: »Imprecise modeling of system. »Uncertainty of previously calculated state(s) due to imprecise measurements. Solution: tolerate the noise.Solution: tolerate the noise.

19. Object Tracking - Slide 19 Kalman Filter Method Kalman filter is “optimal” because it incorporates:Kalman filter is “optimal” because it incorporates: –knowledge of the system and measurement device dynamics, –the statistical description of the system noises, measurement errors, and uncertainty in the dynamics models, –any available information about initial conditions of the variables of interest. Peter S. Maybeck, “ Stochastic models, estimation, and control”

20. Object Tracking - Slide 20 Kalman Filter Method Peter S. Maybeck, “ Stochastic models, estimation, and control” Fig: Typical Kalman filter application

21. Object Tracking - Slide 21 Kalman Filter Method Linear Kalman Filter EquationsLinear Kalman Filter Equations Assume we have the following plant and measurement equations: Error w i, v i are both assumed to be a normal-distribution random variable.

22. Object Tracking - Slide 22 Kalman Filter Method - Define R i, Q i : R i : measurement error covariance cov(v i ) Q i : state model error covariance cov(w i ) Both R i and Q i can be calculated incrementally. - Define P i : P i : State estimation error covariance. P i can be calculated by E[(a i -â i )(a i -â i ) T ]

23. Object Tracking - Slide 23 Kalman Filter Method Linear Kalman Equations are found by optimizing P i : Cool... but what do they mean?

24. Object Tracking - Slide 24 Kalman Filter Method Prediction Phase: Correction Phase:

25. Object Tracking - Slide 25 Kalman Filter Method Prediction Phase Correction Phase QiQi â’ P’ K RiRi state error measurement error â P

26. Object Tracking - Slide 26 Kalman Filter Method Startup of recursion requires: 1. Estimated initial states b 0. 2. Weights on estimation of b 0, denoted as M 0. 3. Initial error covariances R 1 and Q 1. Kalman filter starts by calculating: K 0 = M 0 H(HM 0 H T + R 1 ) -1 and P 0 = (M 0 -1 + H T (R 1 ) -1 H) -1

27. Object Tracking - Slide 27 2D Token Tracking Input: A sequence of images with known temporal order.Input: A sequence of images with known temporal order. Goal: Given a set of 2D lines in one image, track their movements on the images.Goal: Given a set of 2D lines in one image, track their movements on the images. Assumption:Assumption: –There exist some edge-detector to extract edges. –Both camera & object can move in the world. –No internal camera parameters are required.

28. Object Tracking - Slide 28 2D Token Tracking

29. Object Tracking - Slide 29 2D Token Tracking Problem Abstraction:Problem Abstraction: –Identify the “system states” »Find a representation of line » The collection of line position, size, orientation, their derivatives are the “states”. »Error in the modeling is allowed. –Find the “measurement” equation »Found by doing edge detection and match with “predicted” line location.

30. Object Tracking - Slide 30 2D Token Tracking Represent a line with [a, b, x 0, l] where x+ay+b = 0 x 0 : Ordinate of midpoint of line a, b: Orientation of line l: Length of line The “plant” equation can be written as followed:

31. Object Tracking - Slide 31 2D Token Tracking Measurement function can be written as below: x i = r i = r i-1 ’ = Ha i + v i where r 0 ; is the predicted measurement, and H extracts a subset of the system state as the measurements. To complete the modeling, we need covariance matrices Q i and R i. R i is explicitly defined with the representation of line segment, and Q i is chosen by the user, according to the precision of edge detector.

32. Object Tracking - Slide 32 2D Token Tracking Initialization of the process: 1. Initialize state b 0 and M 0 (estimated weights of b 0 ), assuming segments have no motion/acceleration: 2. Calculate K 0 and P 0 and the recursion starts.

33. Object Tracking - Slide 33 3D Token Tracking Similar to 2D, except the 3D information of objects need to be known.Similar to 2D, except the 3D information of objects need to be known. –This implies calibrated stereo rig of cameras. –Representation of lines and state of the system are directly extended from 2D to 3D. –Edge detectors are changed to feature-based 3D reconstructor.

34. Object Tracking - Slide 34 3D Token Tracking

35. Object Tracking - Slide 35 References Faugera, O. “Three-Dimensional Computer Vision - Chapter 8: Tracking Tokens over Time”.Faugera, O. “Three-Dimensional Computer Vision - Chapter 8: Tracking Tokens over Time”. ICCV ‘98ICCV ‘98 Welch, G. and Bishop G. “An Introduction to the Kalman Filter”.Welch, G. and Bishop G. “An Introduction to the Kalman Filter”.