1. KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track http://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf Optional: Lucas-Kanade 20 Years On http://www.ri.cmu.edu/projects/project_515.html http://www.ri.cmu.edu/projects/project_515.html

2. Feature matching vs. tracking Extract features independently and then match by comparing descriptors Extract features in first images and then try to find same feature back in next view What is a good feature? Image-to-image correspondences are key to passive triangulation-based 3D reconstruction

3. Feature point extraction Approximate SSD for small displacement Δ Find points for which the following is maximum maximize smallest eigenvalue of M

4. SIFT features Scale-space DoG maxima Verify minimum contrast and “cornerness” Orientation from dominant gradient Descriptor based on gradient distributions

5. Feature tracking Identify features and track them over video Small difference between frames potential large difference overall Standard approach: KLT (Kanade-Lukas-Tomasi)

6. Brightness constancy assumption Intermezzo: optical flow (small motion) 1D example possibility for iterative refinement

7. Brightness constancy assumption Intermezzo: optical flow (small motion) 2D example (2 unknowns) (1 constraint) ? isophote I(t)=I isophote I(t+1)=I the “aperture” problem

8. Intermezzo: optical flow How to deal with aperture problem? Assume neighbors have same displacement (3 constraints if color gradients are different)

9. Lucas-Kanade Assume neighbors have same displacement least-squares:

10. Compute translation assuming it is small Alternative derivation differentiate: Affine is also possible, but a bit harder (6x6 in stead of 2x2)

11. Revisiting the small motion assumption Is this motion small enough? Probably not—it’s much larger than one pixel (2 nd order terms dominate) How might we solve this problem? * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

12. Reduce the resolution! * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

13. image I t-1 image I Gaussian pyramid of image I t-1 Gaussian pyramid of image I image I image I t-1 u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels Coarse-to-fine optical flow estimation slides from Bradsky and Thrun

14. image I image J Gaussian pyramid of image I t-1 Gaussian pyramid of image I image I image I t-1 Coarse-to-fine optical flow estimation run iterative L-K warp & upsample...... slides from Bradsky and Thrun

15. Good feature to track Tracking Use same window in feature selection as for tracking itself maximize minimal eigenvalue of M Strategy: Look for strong well distributed features, typically few hundreds initialize and then track, renew feature when too many are lost

16. Example Simple displacement is sufficient between consecutive frames, but not to compare to reference template

17. Example

18. Synthetic example

19. Good features to keep tracking Perform affine alignment between first and last frame Stop tracking features with too large errors

20. Live demo OpenCV (try it out!) LKdemo

21. Triangulation C1C1 m1m1 L1L1 m2m2 L2L2 M C2C2 - calibration - correspondences

22. Triangulation Backprojection Triangulation Iterative least-squares Maximum Likelihood Triangulation

23. Backprojection Represent point as intersection of row and column Useful presentation for deriving and understanding multiple view geometry (notice 3D planes are linear in 2D point coordinates) Condition for solution?

24. Next class: epipolar geometry