KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track Optional: Lucas-Kanade 20 Years On

Feature matching vs. tracking Extract features independently and then match by comparing descriptors Extract features in first images and then try to find same feature back in next view What is a good feature? Image-to-image correspondences are key to passive triangulation-based 3D reconstruction

Feature point extraction Approximate SSD for small displacement Δ Find points for which the following is maximum maximize smallest eigenvalue of M

SIFT features Scale-space DoG maxima Verify minimum contrast and “cornerness” Orientation from dominant gradient Descriptor based on gradient distributions

Feature tracking Identify features and track them over video Small difference between frames potential large difference overall Standard approach: KLT (Kanade-Lukas-Tomasi)

Brightness constancy assumption Intermezzo: optical flow (small motion) 1D example possibility for iterative refinement

Brightness constancy assumption Intermezzo: optical flow (small motion) 2D example (2 unknowns) (1 constraint) ? isophote I(t)=I isophote I(t+1)=I the “aperture” problem

Intermezzo: optical flow How to deal with aperture problem? Assume neighbors have same displacement (3 constraints if color gradients are different)

Lucas-Kanade Assume neighbors have same displacement least-squares:

Compute translation assuming it is small Alternative derivation differentiate: Affine is also possible, but a bit harder (6x6 in stead of 2x2)

Revisiting the small motion assumption Is this motion small enough? Probably not—it’s much larger than one pixel (2 nd order terms dominate) How might we solve this problem? * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Reduce the resolution! * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

image I t-1 image I Gaussian pyramid of image I t-1 Gaussian pyramid of image I image I image I t-1 u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels Coarse-to-fine optical flow estimation slides from Bradsky and Thrun

image I image J Gaussian pyramid of image I t-1 Gaussian pyramid of image I image I image I t-1 Coarse-to-fine optical flow estimation run iterative L-K warp & upsample slides from Bradsky and Thrun

Good feature to track Tracking Use same window in feature selection as for tracking itself maximize minimal eigenvalue of M Strategy: Look for strong well distributed features, typically few hundreds initialize and then track, renew feature when too many are lost

Example Simple displacement is sufficient between consecutive frames, but not to compare to reference template

Example

Synthetic example

Good features to keep tracking Perform affine alignment between first and last frame Stop tracking features with too large errors

Live demo OpenCV (try it out!) LKdemo

Triangulation C1C1 m1m1 L1L1 m2m2 L2L2 M C2C2 - calibration - correspondences

Triangulation Backprojection Triangulation Iterative least-squares Maximum Likelihood Triangulation

Backprojection Represent point as intersection of row and column Useful presentation for deriving and understanding multiple view geometry (notice 3D planes are linear in 2D point coordinates) Condition for solution?

Next class: epipolar geometry