1. Object Tracking, Trajectory Analysis and Event Detection in Intelligent Video Systems
Student: Hsu-Yung Cheng
Advisor: Jenq-Neng Hwang, Professor
Department of Electrical Engineering
University of Washington

2. Outlines Motivation Object Tracking Trajectory Analysis
Event Detection
Conclusions and Future Work

3. Motivation Advantage of Video-based systems Applications
Being able to capture a large variety of information
Relatively inexpensive
Easier to install, operate, and maintain
Applications
Security surveillance
Home care surveillance
Intelligent transportation systems
There is an urgent need for intelligent video systems to replace human operators to monitor the areas under surveillance.

4. System Modules for Intelligent Event Detection Systems

5. Challenges for Robust Tracking
Segmentation errors
Change of lighting conditions
Shadows
Occlusion
Inter-Object Partial/Complete Occlusion, Initial Occlusion, Background Occlusion

6. Inter-Object Occlusion

7. Initial Occlusion

8. Background Occlusion

9. Proposed Tracking Mechanism

10. Background Estimation and Updating
Based on Gaussian mixture models [Stauffer 1999]
Model the recent history of each pixel by a mixture of K Gaussian distributions.
Every pixel value is checked among the existing K Gaussian distributions for a match.
Update the weights for the K distributions and the parameters of the matched distribution
The kth Gaussian is ranked by ( )
The top-ranked Gaussians are selected as the background models.
Pixel values that belong to background models are accumulated and averaged as the background image.
The background image is updated for every certain interval of time.

11. Moving Object Segmentation
Based on background subtraction
Fourth order moment
[S. Colonnese et al. Proc. of SPIE 2003]
Thresholding

12. Kalman Filter
Kalman filters are modeled on a Markov chain built on linear operators perturbed by Gaussian noises.
At time k, each target has state
, where
and observation (measurement)
Kalman filters are based on linear dynamical systems discretised in the time domain. They are modelled on a Markov chain built on linear operators perturbed by Gaussian noise.
The state of the system is represented as a vector of real numbers.
At each discrete time increment, a linear operator is applied to the state to generate the new state, with some noise mixed in,
((and optionally some information from the controls on the system if they are known. ))
Then, another linear operator mixed with more noise generates the visible outputs from the hidden state.
The Kalman filter may be regarded as analogous to the hidden Markov model, with the key difference that the hidden state variables are continuous (as opposed to being discrete in the hidden Markov model).
Additionally, the hidden Markov model can represent an arbitrary distribution for the next value of the state variables, in contrast to the Gaussian noise model that is used for the Kalman filter.
There is a strong duality between the equations of the Kalman Filter and those of the hidden Markov model.
A review of this and other models is given in (Roweis and Ghahramani, 1999)
, where
Kalman, R. E. "A New Approach to Linear Filtering and Prediction Problems,“
Transactions of the ASME - Journal of Basic Engineering Vol. 82: pp , 1960.

13. Kalman Filter Phases Predict Updated State Estimate Predicted State
Observed
Measurements
Update

14. Updated State Estimate
Kalman Filter Phases
Update Phase
Predict Phase
Updated State Estimate
Predicted State
Updated Estimate Covariance
Kalman Gain
Predicted Estimate
Covariance
Innovation Covariance
(Pk|k): the error covariance matrix (a measure of the estimated accuracy of the state estimate).
Innovation (Measurement) Residual

15. Constructing Measurement Candidate List

16. Searching for measurement candidate representation points
Search for q1 and q2 in the two n x n windows centered around p1 and p2, respectively.
Compute the dissimilarities between the target object and the potential measurement candidates.

17. Data Association
To associate measurements with targets when performing updates
Nearest Neighbor Data Association
For all the measurement in the validation gate of a target, select the nearest measurement.
Probabilistic Data Association (PDA)
Joint Probabilistic Data Association (JPDA)
Nearest: metric can be euclidean distance, mahanobis distance, etc…

18. Probabilistic Data Association
Consider a single target independently
of others y2
denotes the event that the j th measurement belongs to that target. x
Combined (Weighted) Innovation y1 y3
Probabilistic Data Association
Consider a single target independently of others
Y . Bar-Shalom and E. Tse, “Tracking in a cluttered environment
with probabilistic data association,” Automatica, vol. 11, pp , Sept

19. Modified PDA for Video Object Tracking
To handle video objects (regions), incorporate the following factor when computing
Similarity measure: cross correlation function

20. Experimental Videos

21. Vehicle Tracking Results 1

22. Vehicle Tracking Results 2

23. Human Tracking Results

24. Object Tracking Statistics
Video
Sequence 1
Sequence 2
Sequence 3
Sequence 4
Ground Truth 71 64 92
130
Object Detected 72 61 93
128
Miss 3 5
False Alarm 1
Correctly Detected
125
Correctly Tracked 70 58
120
Detection Precision
0.986
1.000
0.989
0.977
Detection Recall
0.953
0.962
Tracking Success Rate
0.951
0.960
Occluded Object Tracking Success Rate: 0.855

25. Trajectory Analysis
Class 1
Class 3
Class 5
Class 2
Class 4
Class 6

26. Trajectory Smoothing Perform cubic spline interpolation
Sample the trajectory
Perform cubic spline interpolation
First of all, we sample the trajectory at a sampling rate determined by the trajectory length and the average speed. Our purpose is to smooth the trajectory but maintain the essential characteristics of the trajectory at the same time. Therefore, if the trajectory is long and the object moves slowly, then a longer sampling period is used. On the other hand, if the trajectory is short or the object moves at a higher speed, a shorter sampling period is adopted.

27. Angle Feature Extraction
Relative Angle
Absolute Angle

28. Hidden Markov Model N states Si , i=1,…, N Transition probability aij
Initial probability pi
Observation symbol probability bj(k)
A complete model l=(A,B,P)
A={aij}
B={bjk}
P={pi}
Sunny
Cloudy
Rainy
P(walk) = 0.5
P(bike) = 0.4
P(bus) = 0.1
P(walk) = 0.4
P(bike) = 0.3
P(bus) = 0.3
P(walk) = 0.2
P(bike) = 0.1
P(bus) = 0.7

29. Example of HMM
Observation sequence O = { walk, bike, bus, bus, bike, walk, … }

30. Three Problems in HMM
Given l, compute the probability that O is generated by this model
How likely did O happen at this place?
Given l, find the most likely sequence of hidden states that could have generated O
How did the weather change day-by-day?
Given a set of O, learn the most likely l
Train the parameters of the HMM
forward-backward algorithm
Viterbi algorithm
Baum-Welch algorithm

31. Left-to-right HMM for Trajectory Classification

32. K-means Clustering of Feature Points

33. Number of Training and Test sequences
Video for both training and testing
Trajectory Class
Training Trajectories
Testing Objects
Testing Trajectories
Class 1 12 64
307
Class 2 11 18 66
Class 3 13 27
Class 4 5 20
Class 5 8 26 32
Class 6 29 45
Video for testing only
When performing classification, the partial trajectories are tested after an object appears in the ROI for more than 90 frames. After that, the trajectory of the object is updated and tested every 5 frames. Therefore, for an object staying in the video scene for more than 90 frames, more than one trajectory will be created as the test trajectories

34. Trajectory Classification Statistics
Accuracy
Class 1
307
100%
Class 2 64 2
97.4%
Class 3 25
92.6%
Class 4 20
Class 5 1 31
96.8%
Class 6 43
95.5%

35. Anomalous Trajectories

36. Event Detection Type I Events Type II Events Type III Events
Simple rule-based decision logic
Entering a dangerous region
Stopping in the scene
Driving on the road shoulder
Type II Events
Based on trajectory classification results via HMM using angle features
Illegal U-turns or left turns
Anomalous trajectories
Type III Events
Based on trajectory classification results via HMM using speed features
Speed change

37. Conclusions and Future Works
Tracking
Kalman filtering for prediction
Modified PDA for data association
Basic Events
Simple rule-based decision logic
HMM
Higher Level Events
Combining basic events
More flexible models