1. Online Multi-Object Tracking via Structural Constraint Event Aggregation
Ju Hong Yoon
Chang-Ryeol Lee
Ming-Hsuan Yang
Kuk-Jin Yoon
KETI
CV Lab., GIST
UC Merced
In CVPR 2016

2. Outline Introduction Structural Constraint Event Aggregation
Structural constraint cost function
Event aggregation
Assignment event initialization and reduction
Two-Step Online MOT via SCEA
Experiments
Conclusion

3. Outline Introduction Structural Constraint Event Aggregation
Structural constraint cost function
Event aggregation
Assignment event initialization and reduction
Two-Step Online MOT via SCEA
Experiments
Conclusion

4. Introduction Data association Similar objects ?
Detections-to-detections
Multi-object tracking (MOT)
Detections-to-tracklets
Object appearances
Object appearances 被用來當作 Data association 的重要依據
Tracklets-to-tracklets
Similar objects ?

5. Motion model Introduction
Moving cameras not always smooth or predictable

6. Introduction A new data association method :
The structural motion constraints between objects
Location , Velocity
Event aggregation : Assignment ambiguities
reduce the assignment ambiguities caused by mis-detections

7. Introduction

8. Introduction Two-step online 2D MOT framework :
Structural constraint event aggregation
Infer and recover the missing objects
Using the structural constraints of objects between frames, we can re-track the missing ones from the tracked objects from the first step.

9. Outline Introduction Structural Constraint Event Aggregation
Structural constraint cost function
Event aggregation
Assignment event initialization and reduction
Two-Step Online MOT via SCEA
Experiments
Conclusion

10. Structural Constraint Event Aggregation
The state of an object 𝑖 at frame 𝑡 :
Structural motion constraint between two objects :
Position Velocity Size

11. Structural Constraint Event Aggregation

12. Outline Introduction Structural Constraint Event Aggregation
Structural constraint cost function
Event aggregation
Assignment event initialization and reduction
Two-Step Online MOT via SCEA
Experiments
Conclusion

13. Structural constraint cost function
The MOT task can be considered as a data association problem
𝑖 𝑘 1 2 3 . N 1 2 3 . M
finds the correct assignment event between objects and detections

14. Structural constraint cost function
The MOT task can be considered as a data association problem
If the detection 𝑘 is assigned to the object 𝑖,
Otherwise,
The best assignment event is then estimated by minimizing total assignment costs
finds the correct assignment event between objects and detections

15. Structural constraint cost function
A detection 𝑘 at frame 𝑡 :
不失一般性Without loss of generality, we remove the time index t
ai,0 stands for the case of mis-detected objects
每個k最多只會被分給一個i(不包含k=0)
每個i一定會對應到一個k(包含k=0)
mis-detected objects數量不會超過總數量

16. Structural constraint cost function
anchor assignment
structural constraint
aik=1
the structural constraint cost evades the error caused by the global camera motion

17. Structural constraint cost function
Size Appearance
p(d) denote the histogram of an object and a detection
b is the bin index and B is the number of bins

18. Structural constraint cost function
τ=4

20. Outline Introduction Structural Constraint Event Aggregation
Structural constraint cost function
Event aggregation
Assignment event initialization and reduction
Two-Step Online MOT via SCEA
Experiments
Conclusion

21. Event aggregation

22. Outline Introduction Structural Constraint Event Aggregation
Structural constraint cost function
Event aggregation
Assignment event initialization and reduction
Two-Step Online MOT via SCEA
Experiments
Conclusion
considering all of assignment events is not computationally efficient

23. Assignment event initialization and reduction
τ=0.7
If the above conditions are satisfied, ai,k = 1

24. Assignment event initialization and reduction
maximum number of objects in each partition = 5

26. Outline Introduction Structural Constraint Event Aggregation
Structural constraint cost function
Event aggregation
Assignment event initialization and reduction
Two-Step Online MOT via SCEA
Experiments
Conclusion
recovery mis-detected
the previous frame have been also updated with their corresponding detections

28. Two-Step Online MOT via SCEA

29. Two-Step Online MOT via SCEA
D : not-assigned detections and dummy detections d0

30. Two-Step Online MOT via SCEA

31. Two-Step Online MOT via SCEA
we select the object moving in the most similar direction and velocity

32. Two-Step Online MOT via SCEA
Hungarian algorithm

33. Two-Step Online MOT via SCEA
Update final tracking result with Kalman filter for smoothing :
location of a detection assigned to the object i

34. Two-Step Online MOT via SCEA
Structural constraint update :
we indirectly update the structural constraint variations by using the standard Kalman filter

35. Two-Step Online MOT via SCEA
Object management :
Add new objects (velocity = 0)
The distances and the appearance between a detection in the current frame and unassociated detections in the past a few frames are smaller than a certain threshold
Delete objects
If they are not associated with any detections for two frames
(e.g., 4)

36. Outline Introduction Structural Constraint Event Aggregation
Structural constraint cost function
Event aggregation
Assignment event initialization and reduction
Two-Step Online MOT via SCEA
Experiments
Conclusion
recovery mis-detected
the previous frame have been also updated with their corresponding detections

37. Experiments Data association evaluation
Efficiency of the event reduction
Comparisons with State-of-the-Art Methods

38. Data association evaluation
RMN Relative Motion Network [29]
LM Linear Motion (Baseline)
(without the structural constraints or event aggregation)
SCNN - Structural Constraint Nearest Neighbor
(without event aggregation)
[29] J. H. Yoon, M.-H. Yang, J. Lim, and K.-J. Yoon. Bayesian multi-object tracking using motion context from multiple objects. In WACV, 2015

39. Data association evaluation
ETH sequences (Bahnhof, Sunnyday, and Jelmoli sequences) [8]
include at most 10 false detections per each frame
RMN - well low level
[8] A. Ess, B. Leibe, K. Schindler, and L. V. Gool. A mobile vision system for robust multi-person tracking. In CVPR, 2008

40. Efficiency of the event reduction
with the gating technique

41. Comparisons with State-of-the-Art Methods
MDP [26]
TC ODAL [1]
RMOT [29]
NOMT-HM [5]
ODAMOT [11]
[1] S.-H. Bae and K.-J. Yoon. Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning. In CVPR, 2014
[5] W. Choi. Near-online multi-target tracking with aggregated local flow descriptor. In ICCV, 2015
[11] A. Gaidon and E. Vig. Online Domain Adaptation for Multi-Object Tracking. In BMVC, 2015
[26] Y. Xiang, A. Alahi, and S. Savarese. Learning to track:Online multi-object tracking by decision making. In ICCV,2015
[29] J. H. Yoon, M.-H. Yang, J. Lim, and K.-J. Yoon. Bayesian multi-object tracking using motion context from multiple objects.In WACV, 2015

42. Comparisons with State-of-the-Art Methods
Evaluation metrics :
MOTA - Multiple Object Tracking Accuracy
MOTP - Multiple Object Tracking Precision
MT - the number of mostly tracked
ML - the number of mostly lost
FG - the fragment
ID - the identity switch
Rec - the Recall
Prec - the Precision
sec/Hz - the runtime
AR - the average ranking
10個

43. Comparisons with State-of-the-Art Methods
Benchmark dataset :
KITTI dataset [12] : 29 sequences
Detections : DPM [10], regionlet [24]
MOT Challenge dataset [17] : 22 sequences
[10] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9):1627–1645, 2010
[24] X.Wang, M. Yang, S. Zhu, and Y. Lin. Regionlets for generic object detection. In ICCV, 2013
[12] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013
[17] L. Leal-Taix´e, A. Milan, I. Reid, S. Roth, and K. Schindler. Motchallenge 2015: Towards a benchmark for multi-target tracking. In arXiv: , 2015

44. OMDAMOT : the additional local detector to deal with missing objects caused by partial occlusions
NOMT-HM : the optical flow information to reduce ambiguities caused by similar appearance of objects
pedestrian
the motion cue (the optical flow) becomes less discriminative when motion of objects is small

45. Comparisons with State-of-the-Art Methods
TC ODAL : linear motion model to link the tracklets based on the Hungarian algorithm
MDP : learns the target state (Active, Tracked, Lost and Inactive) from a training dataset and its ground truth in an online manner
SCEA does not require any training datasets and it runs faster

46. Comparisons with State-of-the-Art Methods
MDP-KITTI : MDP on the KITTI dataset
MDP-MOTC : trained model provided with the original source code by the authors

47. Outline Introduction Structural Constraint Event Aggregation
Structural constraint cost function
Event aggregation
Assignment event initialization and reduction
Two-Step Online MOT via SCEA
Experiments
Conclusion
recovery mis-detected
the previous frame have been also updated with their corresponding detections

48. Conclusion Structural motion constraints - Large camera motion
Event aggregation - Assignment ambiguities
Two-step algorithm - Recover missing objects

49. Thanks for listening!