Scale Invariant Feature Transform (SIFT) David G. Lowe University of British Columbia 1

References [1] D.G. Lowe, “Object recognition from local scale-invariant features,” in Proc. Seventh IEEE Int’l Conf. Computer Vision, vol. 2, pp. 1150 -1157, 1999. [2] D.G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” Int’l J. Computer Vision, vol. 2, no. 60, pp. 91-110, 2004. 2

Outline Introduction Algorithm Applications Conclusions 3

Outline Introduction Algorithm Applications Conclusions 4

An Example: Build a Panorama 5

How Do We Build Panorama? We need to match (align) images 6

Matching with Features Detect feature points in both images Find corresponding pairs Use these pairs to align images 7

An Example: Harris Detector It is non-invariant to image scale All points will be classified as edges Corner 8

Scale Invariant Detection Consider regions (e.g. circles) of different sizes around a point Regions of corresponding sizes will look the same in both images 9

Scale Invariant Detection The problem: how do we choose corresponding circles independently in each image? 10

Laplacian of Gaussian Operator Maxima of Laplacian gives best notion of scale: Characteristic scale of a local structure scale bad scale Good !

Introduction Generates image features, “keypoints” invariant to image scaling and rotation partially invariant to change in illumination and 3D camera viewpoint many can be extracted from typical images highly distinctive 12

Outline Introduction Algorithm Applications Conclusions 13

( ) Algorithm Scale-space extrema detection Keypoint localization detector Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor descriptor ( ) local descriptor 14

Outline Introduction Algorithm Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor Applications Conclusions 15

Detection of Scale-space Extrema Detect locations that are invariant to scale change of the image SIFT uses DoG filter for scale space because it is efficient and as stable as scale-normalized Laplacian of Gaussian 16

DoG Filtering Convolution with a variable-scale Gaussian Convolution with the DoG filter 17

Scale Space 18

Scale Space: an Example 19

Outline Introduction Algorithm Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor Applications Conclusions 20

Keypoint Localization X is selected if it is larger or smaller than all 26 neighbors 21

Accurate Keypoint Localization Reject the points that have low contrast (and are therefore sensitive to noise) or are poorly localized along and edge 233x189 original image 832 keypoints at extrema 729 keypoints after contrast filtering 536 keypoints after curvature filtering 22

Outline Introduction Algorithm Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor Applications Conclusions 23

Orientation Assignment By assigning a consistent orientation, the keypoint descriptor can be orientation invariant Choose a region around the keypoint Calculate the magnitude of the gradient Gaussian kernel Orientation histogram

Orientation Assignment: an Example (1) 25

Orientation Assignment: an Example (2) 26

Orientation Assignment: an Example (3) 27

Orientation Assignment: an Example (4) approximately 25 degrees 80% of peak value 28

Outline Introduction Algorithm Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor Applications Conclusions 29

Descriptor It computes a descriptor in order to get highly distinctive description The descriptors are as invariant as possible to remaining variations 30

Keypoint Descriptor (1) The best result: 8 orientations x 4x4 histogram array = 128 dimensions 31

Keypoint Descriptor (2) The 128D vectors are normalized to unit length in order to reduces effect of contrast change Because of using gradients, the discriptor is invariant to the changes in illumination 32

Outline Introduction Algorithm Applications Conclusion 33

Keypoint Matching The best candidate match for each keypoint is found by identifying its nearest neighbor in the database Using the ratio of distance (closest/next closest) chooses the correct matches reject the distance ratio that is greater than 0.8 34

Results

Object Recognition (1) It creates a Hough transform entry predicting the model location, orientation, and scale Solution for affine parameters: 36

Object Recognition (2) The LS solution can be determined by solving the corresponding normal equations 37

Recognition examples (1) 38

Recognition examples (2) 39

Outline Introduction Algorithm Applications Conclusions 40

Conclusions The SIFT keypoints are useful due to their distinctiveness This approach transforms an image into a large collection of local feature vectors It is invariant to image translation, scaling, and rotation, and partially invariant to illumination changes and affine or 3D projection 41