1. Performance Evaluation of Feature Detection for Local Optical Flow Tracking Tobias Senst, Brigitte Unger, Ivo Keller and Thomas Sikora

2. Overview
Performance Evaluation of Feature Detection for Local Optical Flow Tracking
Motivation
Evaluation Methodology
Evaluation Results
Conclusion

3. Motivation
The Kanade-Lucas-Thomas (KLT) feature tracker still remains as a widely accepted and utilized method for sparse motion estimation due to its high computational efficiency.
[Ali & Shah]
[Fradet et al.]
KLT – by 1991
Saad Ali and Mubarak Shah, A Lagrangian Particle Dynamics Approach for Crowd Flow Segmentation and Stability Analysis, CVPR 2007
D. C. Herath, Sarath Kodagoda and Gamini Dissanayake, Simultaneous Localisation and Mapping: A Stereo Vision Based Approach, IEEE Intelligent Robots Systems 2006
Hanna Kallén, Hakan Ardö and Olof Enqvist Tracking and Reconstruction of Vehicles for Accurate Position Estimation, WACV 2011
Fradet, Robert and Pérez Clustering point trajectories with various life-spans. CVMP 2009 (Visual Media Production)
[Herath et al.]
[Källén et al.]

4. Motivation
KLT feature tracker

5. Motivation KLT feature tracker Good Features To Track Feature Detector
[Shi & Tomasi,1994]

6. Local Optical Flow Tracking
Motivation
KLT feature tracker
Local Optical Flow Tracking
Good Features To Track
Pyramidal Lucas-Kanade Method
[Shi & Tomasi,1994]
[Bouguet, 2000]

7. Pyramidal Lucas-Kanade Method
Motivation
KLT feature tracker
Rejection
Good Features To Track
Pyramidal Lucas-Kanade Method
SSD
[Shi & Tomasi,1994]
[Bouguet, 2000]

8. Local Optical Flow Tracking
Motivation
Feature Detector
Local Optical Flow Tracking
Rejection
KLT feature tracker
Good Features To Track
Pyramidal Lucas-Kanade Method
SSD
[Shi & Tomasi,1994]
[Bouguet, 2000]
LMedS Local Optical Flow
[Kim et al., 2004]
Bi Direct : Bidirectional flow
Gain Adaptive PLK
[Zach et al., 2008]
Robust Local Optical Flow (mod L2 norm)
Bi-Direct.
[Senst et al., 2011]

9. Local Optical Flow Tracking
Motivation
Feature Detector
Good Features To Track
[Shi & Tomasi,1994]
Local Optical Flow Tracking
Good Feature To Track deals with:
Aperture Problem ?
Bi Direct : Bidirectional flow

10. Local Optical Flow Tracking
Motivation
Feature Detector
Good Features To Track
[Shi & Tomasi,1994]
Local Optical Flow Tracking
Good Feature To Track deals with:
Aperture Problem ?

11. Local Optical Flow Tracking
Motivation
Feature Detector
Good Features To Track
[Shi & Tomasi,1994]
Local Optical Flow Tracking
Good Feature To Track deals with:
Aperture Problem
does not deal with:
Generalized Aperture Problem
Local Optical Flow/ KLT – Tracker are based on the Motion Constancy Assumption W

12. Local Optical Flow Tracking
Motivation
Feature Detector
Good Features To Track
[Shi & Tomasi,1994]
Local Optical Flow Tracking
Good Feature To Track deals with:
Aperture Problem
does not deal with:
Generalized Aperture Problem
Pyramidal Implementation

13. Local Optical Flow Tracking
Motivation
What about other Feature Detectors?
Feature Detector
Local Optical Flow Tracking
Rejection
Good Features To Track
Robust Local Optical Flow (mod L2 norm)
Bi-Direct.
[Shi & Tomasi,1994]
[Senst et al., 2011]
FAST
[Rosten & Drummond,2006]
Find a new Feature detector
- In general we find the best combination by perform a Benchmark
SIFT
[Lowe,1999] …
MSER
[Matas, 2002]

14. Evaluation Methodology
How could we measure the performance of a Local Optical Flow based Feature Tracker?

15. Evaluation Methodology
How could we measure the performance of a Local Optical Flow based Feature Tracker?
Middlebury Optical Flow dataset is an established benchmark for dense motion data.
Local Optical Flow are not build for dense motion estimation (efficiency and accuracy) but for sparse
But we (Feature Tracking) want to pick some points and track them
These points should contain as much as motion information of the sequence
[Baker et al. 2007]

16. Evaluation Methodology
How could we measure the performance of a Local Optical Flow based Feature Tracker?
What are the favored behaviors of a Local Optical Flow based Feature Tracker?
Local Optical Flow Feature Tracker
A local optical flow tracker should have
Accuracy
Efficiency
Descriptive

17. Evaluation Methodology
How could we measure the performance of a Local Optical Flow based Feature Tracker?
Covering Measure:
Mean density of trajectories in a motion segment (r)
Covering the feature should be distributed over the image regarding the moving objects
In jedem segment eine hoh dichte an feature vektoren erwarten

18. Evaluation Methodology
How could we measure the performance of a Local Optical Flow based Feature Tracker?
Covering Measure:
Mean density of trajectories in a motion segment (r)
Accuracy Measure:
Median of endpoint error (MME)
Efficiency Measure:
Feature detection runtime (tD)
Overall runtime (t)
Percentage of rejected features trajectories (h)
Dataset:
Middlebury Flow Test Sequences, 2 Frames

19. Feature Detection Runtime
Evaluation Results
Feature Detection Runtime
Grove3 640x480
on a AMD Phenom II X GHZ
FAST 5ms, GFT 35, PGFT 74, SIFT 272, SURF 158, STAR 18, MSER 54

20. Good Features To Track (GFT)
Evaluation Results
Good Features To Track (GFT)
~200 Feature Points
~2500 Feature Points
Method
td(ms)
t(ms)
h(%)
MEE
r(%)
GFT 26 77
2.2
0.055
0.06
PGFT 55
106
1.8
0.059
0.07
FAST 1 50
0.5
0.051
0.08
SIFT
183
233
1.5
0.188
SURF
113
162
4.3
0.180
0.11
STAR 13 62
3.3
0.127
0.10
MSER 51
101
3.0
0.075
0.24
Method
td(ms)
t(ms)
h(%)
MEE
r(%)
GFT 27 89
3.2
0.057
0.99
PGFT 56
119
1.12
FAST 4 63
2.2
0.050
1.25
SIFT
414
476
1.9
0.78
SURF
237
299
3.3
0.066
1.06
STAR 18 74
2.0
0.94
MSER 93
148
2.3
0.051
0.54

21. Feature from Accelerated Segment Test (FAST)
Evaluation Results
Feature from Accelerated Segment Test (FAST)
~200 Feature Points
~2500 Feature Points
Method
td(ms)
t(ms)
h(%)
MEE
r(%)
GFT 26 77
2.2
0.055
0.06
PGFT 55
106
1.8
0.059
0.07
FAST 1 50
0.5
0.051
0.08
SIFT
183
233
1.5
0.188
SURF
113
162
4.3
0.180
0.11
STAR 13 62
3.3
0.127
0.10
MSER 51
101
3.0
0.075
0.24
Method
td(ms)
t(ms)
h(%)
MEE
r(%)
GFT 27 89
3.2
0.057
0.99
PGFT 56
119
1.12
FAST 4 63
2.2
0.050
1.25
SIFT
414
476
1.9
0.78
SURF
237
299
3.3
0.066
1.06
STAR 18 74
2.0
0.94
MSER 93
148
2.3
0.051
0.54

22. Speed Up Robust Features (SURF)
Evaluation Results
Speed Up Robust Features (SURF)
~200 Feature Points
~2500 Feature Points
Method
td(ms)
t(ms)
h(%)
MEE
r(%)
GFT 26 77
2.2
0.055
0.06
PGFT 55
106
1.8
0.059
0.07
FAST 1 50
0.5
0.051
0.08
SIFT
183
233
1.5
0.188
SURF
113
162
4.3
0.180
0.11
STAR 13 62
3.3
0.127
0.10
MSER 51
101
3.0
0.075
0.24
Method
td(ms)
t(ms)
h(%)
MEE
r(%)
GFT 27 89
3.2
0.057
0.99
PGFT 56
119
1.12
FAST 4 63
2.2
0.050
1.25
SIFT
414
476
1.9
0.78
SURF
237
299
3.3
0.066
1.06
STAR 18 74
2.0
0.94
MSER 93
148
2.3
0.051
0.54

23. Conclusion
Our goal was to provide a set of baseline results for tracking features with alternative detectors.
We provide an evaluation methodology based on accuracy, efficiency and descriptive measures.
We observed that the FAST is an efficient alternative to the GFT detector.
The SIFT shows limited improvements, but an improved GTF method could benefit from its scale space analysis.
The feature tracking evaluation would benefit from a dataset with more than two frames e.g. by measuring life-spans of trajectories.

24. Conclusion
Thank you!