1. Harris Corner Detector & Scale Invariant Feature Transform (SIFT)

2. Harris Corner Detector

3. Harris Detector: Intuition
“flat” region: no change in all directions
“edge”: no change along the edge direction
“corner”: significant change in all directions

4. Moravec Corner Detector
Shift in any direction would result in a significant change at a corner.
Algorithm:
Shift in horizontal, vertical, and diagonal directions by one pixel.
Calculate the absolute value of the MSE for each shift.
Take the minimum as the cornerness response.

5. Harris Detector: Mathematics
Change of intensity for the shift [u,v]:
Intensity
Window function
Shifted intensity or
Window function w(x,y) =
Gaussian
1 in window, 0 outside

6. Harris Detector: Mathematics
Apply Taylor series expansion:

7. Harris Detector: Mathematics
For small shifts [u,v] we have the following approximation:
where M is a 22 matrix computed from image derivatives:

8. Harris Detector: Mathematics
Intensity change in shifting window: eigenvalue analysis
1, 2 – eigenvalues of M
direction of the fastest change
Ellipse E(u,v) = const
direction of the slowest change
(max)-1/2
(min)-1/2

9. Harris Detector: Mathematics2
Classification of image points using eigenvalues of M:
“Edge” 2 >> 1
“Corner” 1 and 2 are large, 1 ~ 2; E increases in all directions
1 and 2 are small; E is almost constant in all directions
“Edge” 1 >> 2
“Flat” region 1

10. Harris corner detector
Measure of corner response:
(k – empirical constant, k = )
No need to compute eigenvalues explicitly!

13. Eliminate small responses.

14. Find local maxima of the remaining.

16. Harris Detector: Scale
Rmin= 0
Rmin= 1500

17. Summary of the Harris detector

18. Harris Detector: Some Properties
Rotation invariance
Ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner response R is invariant to image rotation

19. Harris Detector: Some Properties
Partial invariance to affine intensity change
Only derivatives are used => invariance to intensity shift I  I + b
Intensity scale: I  a I R
x (image coordinate)
threshold

20. Harris Detector: Some Properties
But: non-invariant to image scale!
All points will be classified as edges
Corner !

21. Harris Detector: Some Properties
Quality of Harris detector for different scale changes
Repeatability rate:
# correspondences # possible correspondences
C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

22. Scale Invariant Detection
Consider regions (e.g. circles) of different sizes around a point
Regions of corresponding sizes will look the same in both images

23. Scale Invariant Detection
The problem: how do we choose corresponding circles independently in each image?

24. Scale Invariant Detection
Solution:
Design a function on the region (circle), which is “scale invariant” (the same for corresponding regions, even if they are at different scales)
Example: average intensity. For corresponding regions (even of different sizes) it will be the same.
For a point in one image, we can consider it as a function of region size (circle radius) f
region size
Image 1 f
region size
Image 2
scale = 1/2

25. Scale Invariant Detection
Common approach:
Take a local maximum of this function
Observation: region size, for which the maximum is achieved, should be invariant to image scale. f
region size
Image 1 f
region size
Image 2
scale = 1/2 s1 s2

26. Characteristic Scale
Ratio of scales corresponds to a scale factor between two images

27. Scale Invariant Detection
A “good” function for scale detection: has one stable sharp peak f
region size
bad f
region size
Good ! f
region size
bad
For usual images: a good function would be a one which responds to contrast (sharp local intensity change)

28. Scale Invariant Detection
Functions for determining scale
Kernels:
(Laplacian)
(Difference of Gaussians)
where Gaussian
L or DoG kernel is a matching filter. It finds blob-like structure. It turns out to be also successful in getting characteristic scale of other structures, such as corner regions.

29. Difference-of-Gaussians

30. Scale-Space Extrema Choose all extrema within 3x3x3 neighborhood.
X is selected if it is larger or smaller than all 26 neighbors

31. Scale Invariant Detectorsx y
 Harris 
 Laplacian 
Harris-Laplacian1 Find local maximum of:
Harris corner detector in space (image coordinates)
Laplacian in scale
SIFT (Lowe)2 Find local maximum of:
Difference of Gaussians in space and scale
scale x y
 DoG 
1 K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2004

32. Harris-Laplace Detector

33. Scale Invariant Detectors
Experimental evaluation of detectors w.r.t. scale change
Repeatability rate:
# correspondences # possible correspondences
K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001

34. Affine Invariant Detection
Above we considered: Similarity transform (rotation + uniform scale)
Now we go on to: Affine transform (rotation + non-uniform scale)

35. Affine Invariant Detection
Take a local intensity extremum as initial point
Go along every ray starting from this point and stop when extremum of function f is reached f
points along the ray
T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000.

36. Affine Invariant Detection
Such extrema occur at positions where intensity suddenly changes compared to the intensity changes up to that point.
In theory, leaving out the denominator would still give invariant positions. In practice, the local extrema would be shallow, and might result in inaccurate positions.
T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000.

37. Affine Invariant Detection
The regions found may not exactly correspond, so we approximate them with ellipses
Find the ellipse that best fits the region

38. Affine Invariant Detection
Covariance matrix of region points defines an ellipse:
( p = [x, y]T is relative to the center of mass)
Ellipses, computed for corresponding regions, also correspond!

39. Affine Invariant Detection
Algorithm summary (detection of affine invariant region):
Start from a local intensity extremum point
Go in every direction until the point of extremum of some function f
Curve connecting the points is the region boundary
Compute the covariance matrix
Replace the region with ellipse
T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000.

40. Affine Invariant Detection
Maximally Stable Extremal Regions
Threshold image intensities: I > I0
Extract connected components (“Extremal Regions”)
Find “Maximally Stable” regions
Approximate a region with an ellipse
J.Matas et.al. “Distinguished Regions for Wide-baseline Stereo”. Research Report of CMP, 2001.

41. Affine Invariant Detection : Summary
Under affine transformation, we do not know in advance shapes of the corresponding regions
Ellipse given by geometric covariance matrix of a region robustly approximates this region
For corresponding regions ellipses also correspond
Methods:
Search for extremum along rays [Tuytelaars, Van Gool]:
Maximally Stable Extremal Regions [Matas et.al.]

42. ? Point Descriptors We know how to detect points Next question:
How to match them? ?
Point descriptor should be:
Invariant
Distinctive

43. Descriptors Invariant to Rotation
Convert from Cartesian to Polar coordinates
Rotation becomes translation in polar coordinates
Take Fourier Transform
Magnitude of the Fourier transform is invariant to translation.

44. Descriptors Invariant to Rotation
Find local orientation
Dominant direction of gradient
Compute image regions relative to this orientation

45. Descriptors Invariant to Scale
Use the characteristic scale determined by detector to compute descriptor in a normalized frame

46. Affine Invariant Descriptors
Find affine normalized frame A A1 A2
rotation
Compute rotational invariant descriptor in this normalized frame
J.Matas et.al. “Rotational Invariants for Wide-baseline Stereo”. Research Report of CMP, 2003

47. Affine covariant regions

48. SIFT (Scale Invariant Feature Transform)

49. SIFT – Scale Invariant Feature Transform1
Empirically found2 to show very good performance, invariant to image rotation, scale, intensity change, and to moderate affine transformations
Scale = 2.5 Rotation = 450
1 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV K.Mikolajczyk, C.Schmid. “A Performance Evaluation of Local Descriptors”. CVPR 2003

50. SIFT – Scale Invariant Feature Transform
Descriptor overview:
Determine scale (by maximizing DoG in scale and in space), local orientation as the dominant gradient direction. Use this scale and orientation to make all further computations invariant to scale and rotation.
Compute gradient orientation histograms of several small windows (to produce 128 values for each point)
D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2004

51. Scale-space extrema detection
Need to find “characteristic scale” for each feature point.

52. Difference-of-Gaussians

53. Scale-Space Extrema Choose all extrema within 3x3x3 neighborhood.
X is selected if it is larger or smaller than all 26 neighbors

54. Keypoint Localization & Filtering
Now we have much less points than pixels.
However, still lots of points (~1000s)…
With only pixel-accuracy at best
And this includes many bad points
Brown & Lowe 2002

55. Keypoint Filtering - Low Contrast
Reject points with bad contrast:
DoG smaller than 0.03 (image values in [0,1])
Reject edges
Similar to the Harris detector; look at the autocorrelation matrix

56. Maxima in D

57. Remove low contrast and edges

58. Orientation assignment
By assigning a consistent orientation, the keypoint descriptor can be orientation invariant.
Let, for a keypoint, L is the image with the closest scale.
Compute gradient magnitude and orientation using finite differences:

59. Orientation assignment

60. Orientation assignment

61. Orientation assignment

62. Orientation assignment

63. Orientation Assignment
Any peak within 80% of the highest peak is used to create a keypoint with that orientation
~15% assigned multiplied orientations, but contribute significantly to the stability

64. SIFT descriptor

65. SIFT Descriptor Each point so far has x, y, σ, m, θ
Now we need a descriptor for the region
Could sample intensities around point, but…
Sensitive to lighting changes
Sensitive to slight errors in x, y, θ
Edelman et al. 1997

66. Image from: Jonas Hurrelmann
SIFT Descriptor
16x16 Gradient window is taken. Partitioned into 4x4 subwindows.
Histogram of 4x4 samples in 8 directions
Gaussian weighting around center( is 0.5 times that of the scale of a keypoint)
4x4x8 = 128 dimensional feature vector
Image from: Jonas Hurrelmann

67. SIFT Descriptor – Lighting changes
Gains do not affect gradients
Normalization to unit length removes contrast
Saturation affects magnitudes much more than orientation
Threshold gradient magnitudes with 0.2 and renormalize

68. Performance Very robust
80% Repeatability at:
10% image noise
45° viewing angle
1k-100k keypoints in database
Best descriptor in [Mikolajczyk & Schmid 2005]’s extensive survey

72. Recognition under occlusion