1. Tracking
Many slides adapted from Kristen Grauman, Deva Ramanan

2. Feature tracking
So far, we have only considered optical flow estimation in a pair of images
If we have more than two images, we can compute the optical flow from each frame to the next
Given a point in the first image, we can in principle reconstruct its path by simply “following the arrows”

3. Tracking challenges Ambiguity of optical flow Large motions
Find good features to track
Large motions
Discrete search instead of Lucas-Kanade
Changes in shape, orientation, color
Allow some matching flexibility
Occlusions, disocclusions
Need mechanism for deleting, adding new features
Drift – errors may accumulate over time
Need to know when to terminate a track

4. Handling large displacements
Define a small area around a pixel as the template
Match the template against each pixel within a search area in next image – just like stereo matching!
Use a match measure such as SSD or correlation
After finding the best discrete location, can use Lucas-Kanade to get sub-pixel estimate

5. Tracking over many frames
Select features in first frame
For each frame:
Update positions of tracked features
Discrete search or Lucas-Kanade
Terminate inconsistent tracks
Compute similarity with corresponding feature in the previous frame or in the first frame where it’s visible
Start new tracks if needed

6. Shi-Tomasi feature tracker
Find good features using eigenvalues of second-moment matrix
Key idea: “good” features to track are the ones that can be tracked reliably
From frame to frame, track with Lucas-Kanade and a pure translation model
More robust for small displacements, can be estimated from smaller neighborhoods
Check consistency of tracks by affine registration to the first observed instance of the feature
Affine model is more accurate for larger displacements
Comparing to the first frame helps to minimize drift
J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.

7. J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.
Tracking example
J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.

8. Tracking with dynamics
Key idea: Given a model of expected motion, predict where objects will occur in next frame, even before seeing the image
Restrict search for the object
Improved estimates since measurement noise is reduced by trajectory smoothness

9. General model for tracking
The moving object of interest is characterized by an underlying state X
State X gives rise to measurements or observations Y
At each time t, the state changes to Xt and we get a new observation Yt X1 X2 Y1 Y2 Xt Yt …

10. Steps of tracking
Prediction: What is the next state of the object given past measurements?

11. Steps of tracking
Prediction: What is the next state of the object given past measurements?
Correction: Compute an updated estimate of the state from prediction and measurements

12. Steps of tracking
Prediction: What is the next state of the object given past measurements?
Correction: Compute an updated estimate of the state from prediction and measurements
Tracking can be seen as the process of propagating the posterior distribution of state given measurements across time

13. Simplifying assumptions
Only the immediate past matters
dynamics model

14. Simplifying assumptions
Only the immediate past matters
Measurements depend only on the current state
dynamics model
observation model

15. Simplifying assumptions
Only the immediate past matters
Measurements depend only on the current state
dynamics model
observation model … X1 X2 Xt Y1 Y2 Yt

16. Tracking as induction predict correct Base case:
Assume we have initial prior that predicts state in absence of any evidence: P(X0)
At the first frame, correct this given the value of Y0=y0
Given corrected estimate for frame t:
Predict for frame t+1
Correct for frame t+1
predict
correct

17. Tracking as induction Base case:
Assume we have initial prior that predicts state in absence of any evidence: P(X0)
At the first frame, correct this given the value of Y0=y0

18. Induction step: Prediction
Prediction involves representing given

19. Induction step: Prediction
Prediction involves representing given
Law of total probability

20. Induction step: Prediction
Prediction involves representing given
Conditioning on Xt–1

21. Induction step: Prediction
Prediction involves representing given
Independence assumption

22. Induction step: Correction
Correction involves computing given predicted value

23. Induction step: Correction
Correction involves computing given predicted value
Bayes rule

24. Induction step: Correction
Correction involves computing given predicted value
Independence assumption (observation yt depends only on state Xt)

25. Induction step: Correction
Correction involves computing given predicted value
Conditioning on Xt

26. Summary: Prediction and correction
dynamics model
corrected estimate from previous step

27. Summary: Prediction and correction
dynamics model
corrected estimate from previous step
observation model
predicted estimate

28. Linear Dynamic Models
Dynamics model: state undergoes linear tranformation plus Gaussian noise
Observation model: measurement is linearly transformed state plus Gaussian noise

29. Example: Constant velocity (1D)
State vector is position and velocity
Measurement is position only
(greek letters denote noise terms)

30. Example: Constant acceleration (1D)
State vector is position, velocity, and acceleration
Measurement is position only
(greek letters denote noise terms)

31. The Kalman filter
Method for tracking linear dynamical models in Gaussian noise
The predicted/corrected state distributions are Gaussian
You only need to maintain the mean and covariance
The calculations are easy (all the integrals can be done in closed form)

32. Propagation of Gaussian densities

33. The Kalman Filter: 1D state
Make measurement
Given corrected state from previous time step and all the measurements up to the current one, predict the distribution over the current step
Given prediction of state and current measurement, update prediction of state
Predict
Correct
Time advances (from t–1 to t)
Mean and std. dev. of predicted state:
Mean and std. dev. of corrected state:

34. 1D Kalman filter: Prediction
Linear dynamic model defines predicted state evolution, with noise
Want to estimate distribution for next predicted state

35. 1D Kalman filter: Prediction
Linear dynamic model defines predicted state evolution, with noise
Want to estimate distribution for next predicted state

36. 1D Kalman filter: Prediction
Linear dynamic model defines predicted state evolution, with noise
Want to estimate distribution for next predicted state
Update the mean:
Update the variance:

37. 1D Kalman filter: Correction
Mapping of state to measurements:
Predicted state:
Want to estimate corrected distribution

38. 1D Kalman filter: Correction
Mapping of state to measurements:
Predicted state:
Want to estimate corrected distribution

39. 1D Kalman filter: Correction
Mapping of state to measurements:
Predicted state:
Want to estimate corrected distribution
Update the mean:
Update the variance:

40. Prediction vs. correction
What if there is no prediction uncertainty
What if there is no measurement uncertainty
The measurement is ignored!
The prediction is ignored!

41. The Kalman filter: General case
PREDICT
CORRECT
show complete set of equations and repeat (on one slide) the cycle;
MAYBE shift equations up and show dimensions also….

42. Kalman filter pros and cons
Simple updates, compact and efficient
Cons
Unimodal distribution, only single hypothesis
Restricted class of motions defined by linear model

43. Propagation of general densities

44. Factored sampling Represent the state distribution non-parametrically
Prediction: Sample points from prior density for the state, P(X)
Correction: Weight the samples according to P(Y|X)
M. Isard and A. Blake, CONDENSATION -- conditional density propagation for visual tracking, IJCV 29(1):5-28, 1998

45. Particle filtering
We want to use sampling to propagate densities over time (i.e., across frames in a video sequence)
At each time step, represent posterior P(Xt|Yt) with weighted sample set
Previous time step’s sample set P(Xt-1|Yt-1) is passed to next time step as the effective prior
M. Isard and A. Blake, CONDENSATION -- conditional density propagation for visual tracking, IJCV 29(1):5-28, 1998

46. Particle filtering Start with weighted samples from previous time step
Sample and shift according to dynamics model
Spread due to randomness; this is predicted density P(Xt|Yt-1)
Weight the samples
according to observation density
Arrive at corrected density estimate P(Xt|Yt)
M. Isard and A. Blake, CONDENSATION -- conditional density propagation for visual tracking, IJCV 29(1):5-28, 1998

47. Particle filtering results

48. Tracking issues
Initialization
Manual
Background subtraction
Detection

49. Tracking issues Initialization
Obtaining observation and dynamics model
Generative observation model: “render” the state on top of the image and compare
Discriminative observation model: classifier or detector score
Dynamics model: learn (very difficult) or specify using domain knowledge

50. Tracking issues Initialization
Obtaining observation and dynamics model
Prediction vs. correction
If the dynamics model is too strong, will end up ignoring the data
If the observation model is too strong, tracking is reduced to repeated detection

51. Tracking issues Initialization
Obtaining observation and dynamics model
Prediction vs. correction
Data association
What if we don’t know which measurements to associate with which tracks?

52. Tracking issues Initialization
Obtaining observation and dynamics model
Prediction vs. correction
Data association
Drift
Errors caused by dynamical model, observation model, and data association tend to accumulate over time

53. Drift
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.

54. Data association
So far, we’ve assumed the entire measurement to be relevant to determining the state
In reality, there may be uninformative measurements (clutter) or measurements may belong to different tracked objects
Data association: task of determining which measurements go with which tracks

55. Data association
Simple strategy: only pay attention to the measurement that is “closest” to the prediction

56. Data association
Simple strategy: only pay attention to the measurement that is “closest” to the prediction
Doesn’t always work…

57. Data association
Simple strategy: only pay attention to the measurement that is “closest” to the prediction
More sophisticated strategy: keep track of multiple state/observation hypotheses
Can be done with particle filtering
This is a general problem in computer vision, there is no easy solution

58. Recall: Generative part-based models
h: assignment of features to parts
Model
Candidate parts

59. Recall: Generative part-based models
h: assignment of features to parts
h: assignment of features to parts
Model
Candidate parts

60. Tracking people by learning their appearance
Person model = appearance + structure (+ dynamics)
Structure and dynamics are generic, appearance is person-specific
Trying to acquire an appearance model “on the fly” can lead to drift
Instead, can use the whole sequence to initialize the appearance model and then keep it fixed while tracking
Given strong structure and appearance models, tracking can essentially be done by repeated detection (with some smoothing)
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.

61. Tracking people by learning their appearance
Tracker
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.

62. Representing people

63. Bottom-up initialization: Clustering
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.

64. Top-down initialization: Exploit “easy” poses
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.

65. Tracking by model detection
D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their Appearance. PAMI 2007.

66. Example results