1. Distinctive image features from scale-invariant keypoints. David G. Lowe, Int. Journal of Computer Vision, 60, 2 (2004), pp. 91-110 Presented by: Shalomi Eldar Based (in part) on slides by Ofir Pele, Kirill Dyagilev and Ayelet Dominitz SIFT SIFT Scale Invariant Feature Transform Vision Topics Seminar 2009

2. 2 Image Matching Introduction Detection DescriptionApplications Motion Tracking Fundamental aspect of many problems: Object Recognition 3D structures Stereo Correspondence

3. 3 Features Detection Introduction Detection DescriptionApplications Give comprehensive description of image. Images from: M. Brown and D. G. Lowe. Recognising Panoramas. In Proceedings of the ) the International Conference on Computer Vision (ICCV2003 Enables matching! What are the desired features’ features?

4. 4 Features’ Features Introduction Detection DescriptionApplications Robustness => Invariance to changes in illumination, scale, rotation, affine, perspective. Locality => robustness to occlusion and clutter. Distinctiveness => easy to match to a large database of objects. Quantity => many features can be generated for even small objects. Efficiency => computationally “cheap”, real- time performance.

5. 5 SIFT Algorithm Introduction Detection DescriptionApplications SIFT - Scale Invariant Feature Transform Input: Image n x m. Output: Set of descriptors of image’s features.

6. 6 SIFT Algorithm 2. Keypoint localization. 1. Scale-space extrema detection. 3. Orientation assignment. 4. Keypoint descriptor. Introduction Detection DescriptionApplications Typical image of size 500x500 pixels produces about 2000 stable keypoints Near Real-time performance: Cascade approach – keep heavy operations only to keypoints that “survive”.

7. 7 SIFT Algorithm Introduction Detection DescriptionApplications 2. Keypoint localization. 1. Scale-space extrema detection. 3. Orientation assignment. 4. Keypoint descriptor. => Application - matching. We need only 3 keypoints matches for reliable identification!

8. 8 Today’s Talk Orientation Extracting Keypoints Distinctive DescriptionMatching Keypoints Extrema detection Correct localization Choosing robust keypoints only Local invariant orientation Building keypoint descriptor Nearest-neighbor algorithm Finding 3 matches Least-square affine approximation Detection Description Application 1 st part of the talk2 nd part of the talkLast part of the talk Z Introduction Detection DescriptionApplications

9. 9 SIFT Algorithm 2. Keypoint localization. 1. Scale-space extrema detection. 3. Orientation assignment. 4. Keypoint descriptor. Introduction Detection DescriptionApplications Collecting keypoint candidates…

10. 10 Introduction Detection DescriptionApplications Why Extrema? We want to find points that give us information about the objects in the image. We will represent the image in a way that gives us these edges as this representations extrema points. The information about the objects is in the object’s edges.

11. 11 Scale-space Representation Introduction Detection DescriptionApplications

12. 12 Low computation time - Only subtraction of smoothed images! Introduction Detection DescriptionApplications Scale-space Representation Difference-of-Gaussian (DoG):

13. 13 Here’s what we get: Introduction Detection DescriptionApplications DoG Pyramid Different frequencies features Scale invariance

14. 14 X is selected if it is larger or smaller than all 26 neighbors. Introduction Detection DescriptionApplications Extracting Keypoints Low cost - only several usually checked.

15. 15 Extrema detection product: Introduction Detection DescriptionApplications Extracting Keypoints Not all of them are good… 233x189 image => 832 DoG Keypoints. Each Keypoint is represented as.

16. 16 Problematic Keypoints Introduction Detection DescriptionApplications Inaccurate localization (due to scaling and sampling). Low contrast - sensitive to noise. Strong edge responses.

17. 17 SIFT Algorithm 2. Keypoint localization. 1. Scale-space extrema detection. 3. Orientation assignment. 4. Keypoint descriptor. Filtering Keypoints… Introduction Detection DescriptionApplications

18. 18 Inaccurate Keypoint Localization Introduction Detection DescriptionApplications x Sampling Detected Extrema True Extrema The Problem:

19. 19 Inaccurate Keypoint Localization Introduction Detection DescriptionApplications The Solution: Taylor expansion: Minimize to find accurate extrema: Brown & Lowe 2002 If offset from sampling point is larger than 0.5 - Keypoint should be in a different sampling point.

20. 20 Low Contrast Keypoints Introduction Detection DescriptionApplications Function value at the extremun - If (pixel values in range [0,1]) - keypoint is discarded. => down to 729 Keypoints after min. contrast threshold.

21. 21 Eliminating Edge Responses Introduction Detection DescriptionApplications The Problem: “Edge“ keypoints are poorly determined. Point can move along edge Point detection => unstable.

22. 22 Eliminating Edge Responses Introduction Detection DescriptionApplications The Solution: Check Keypoints “cornerness”. Point constrained High “cornerness”  No dominant principal curvature component.

23. 23 Finding “Cornerness” Introduction Detection DescriptionApplications Principal curvature are proportional to eigenvalues of Hessian matrix: Harris (1988) showed: Threshold: if r < 10 - ratio is too great, keypoint discarded.

24. 24 Stable Keypoints Introduction Detection DescriptionApplications => down to 536 Keypoints after “cornerness” threshold.

25. 25 Today’s Talk Orientation Extracting Keypoints Distinctive DescriptionMatching Keypoints Extrema detection Correct localization Choosing robust keypoints only Local invariant orientation Building keypoint descriptor Nearest-neighbor algorithm Finding 3 matches Least-square affine approximation Detection Description Application 1 st part of the talk2 nd part of the talkLast part of the talk  Introduction Detection DescriptionApplications DEMO

26. 26 SIFT Algorithm 2. Keypoint localization. 1. Scale-space extrema detection. 3. Orientation assignment. 4. Keypoint descriptor. Representation invariant to Rotation. Introduction Detection DescriptionApplications

27. 27 Gradients Introduction Detection DescriptionApplications For each sample point we compute gradient’s magnitude and orientation:

28. 28 Keypoints Orientation Introduction Detection DescriptionApplications Create gradient histogram (36 bins) weighted by magnitude and Gaussian window ( is 1.5 times that of the scale of a keypoint) Any histogram peak within 80% of highest peak is assigned to keypoint (multiple assignments possible).

29. 29 SIFT Algorithm 2. Keypoint localization. 1. Scale-space extrema detection. 3. Orientation assignment. 4. Keypoint descriptor. Distinctive (yet invariant) Keypoint Representation. Introduction Detection DescriptionApplications

30. 30 What do we have (and what don’t) For each Keypoint we have assigned location, scale and orientation. Provides invariance to these parameters. Introduction Detection DescriptionApplications Do: Sufficient distinctiveness. Invariance to other parameters, such as 3D viewpoint and change of illumination. Don’t:

31. 31 Keypoint Descriptor Introduction Detection DescriptionApplications Create 16 gradient histograms (8 bins) weighted by magnitude and Gaussian window ( is 0.5 times of the window) Keypoint Descriptor - 128 (4x4x8) element vector

32. 32 Change of Illumination Introduction Detection DescriptionApplications Change of brightness => doesn’t effect gradients (difference of pixels value). Change of contrast => doesn’t effect gradients (up to normalization). Saturation (non-linear change of illumination) => affects magnitudes much more than orientation. => Threshold gradient magnitudes to 0.2 and renormalize.

33. 33 Today’s Talk Orientation Extracting Keypoints Distinctive Description Matching Keypoints Extrema detection Correct localization Choosing robust keypoints only Local invariant orientation Building keypoint descriptor Nearest-neighbor algorithm Finding 3 matches Least-square affine approximation Detection Description Application 1 st part of the talk2 nd part of the talkLast part of the talk   Introduction Detection DescriptionApplications

34. 34 Object Recognition For training images: Extracting keypoints by SIFT. Creating descriptors database. Performing detailed geometric fit check for each cluster. Introduction Detection DescriptionApplications For query images: Extracting keypoints by SIFT. For each descriptor - finding nearest neighbor in DB. Finding cluster of at-least 3 keypoints.

35. 35 Keypoint Matching Best candidate mach: Introduction Detection DescriptionApplications Problem: There are keypoints that do not have correct matches (background clutter/were not detected in training images) Nearest neighbor.

36. 36 Keypoints Matching Solution : Introduction Detection DescriptionApplications closest neighbor found closest neighbor from different object Threshold : < 0.8 Eliminates 90% of false matches while discarding less than 5% of correct matches.

37. 37 No good algorithm for high dimensional spaces. Finding Nearest Neighbor Introduction Detection DescriptionApplications => Use Best-Bin-First (BBF) algorithm (Beis and Lowe, 1997). => Returns closest neighbor with high probability. Low cost - cutting off search after checking specific number of candidates. Good enough - we only need to show 0.8 ratio between first and second neighbor.

38. 38 Clustering We cluster 3 keypoints using Hough Transform. In order to identify an object with high probability - we need more than one match: Introduction Detection DescriptionApplications

39. 39 Geometric Verification Hough Transform found clusters of keypoints. Introduction Detection DescriptionApplications We would like to verify that these points do match geometrically to trained image. In order to do that we use least-squares approach to find affine transformation from training image to query image. Now we can be pretty sure we got the right object!

40. 40 Examples Introduction Detection DescriptionApplications Training images: Query image: Recognition (clutter, occlusion, illumination, etc.) :

41. 41 Examples Introduction Detection DescriptionApplications Training images: Query image: Recognition (different viewpoint, non-distinctive) : Total time to recognize all object in both examples is less than 0.3 sec. (on 2GHz Pentium 4 processor)

42. 42 Image Registration Introduction Detection DescriptionApplications [Brown & Lowe 2003]

43. 43 Examples Introduction Detection DescriptionApplications Real time Object-Recognition: Nose DEMO Motion Tracking: Recognition DEMO

44. 44 Nearest-neighbor algorithm Finding 3 matches Least-square affine approximation Extrema detection Correct localization Choosing robust keypoints only Summary    Introduction Detection DescriptionApplications Distinctive DescriptionMatching Keypoints Local invariant orientation Building keypoint descriptor Detection Description Application 1 st part of the talk2 nd part of the talkLast part of the talk Extracting Keypoints

45. 45 Thank you