1. Feature Detection and Matching
Reading: Szeliski Chapter 4
Pages: 208~266

2. COMPARE with WHAT we learn before?
How to UNDERSTAND?
COMPARE with
WHAT we learn before?

3. Image Matching TexPoint fonts used in EMF.
by Diva Sian
by swashford
TexPoint fonts used in EMF.
Read the TexPoint manual before you delete this box.: AAAAAA

4. Harder Case
by Diva Sian
by scgbt

5. Even Harder Case
“How the Afghan Girl was Identified by Her Iris Patterns” Read the story

6. Harder still?
NASA Mars Rover images

7. Answer below (look for tiny colored squares…)
NASA Mars Rover images
with SIFT feature matches Figure by Noah Snavely

8. Features Readings Szeliski, Ch 4.1
All is Vanity, by C. Allan Gilbert,
Readings
Szeliski, Ch 4.1
(optional) K. Mikolajczyk, C. Schmid,  A performance evaluation of local descriptors. In PAMI 27(10):

9. Image Matching

10. Image Matching

11. Invariant Local Features
Find features that are invariant to transformations
geometric invariance: translation, rotation, scale
photometric invariance: brightness, exposure, …
Feature Descriptors

12. Advantages of Local Features
Locality
features are local, so robust to occlusion and clutter
Distinctiveness:
can differentiate a large database of objects
Quantity
hundreds or thousands in a single image
Efficiency
real-time performance achievable
Generality
exploit different types of features in different situations

13. More Motivation… Feature points are used for:
Image alignment (e.g., mosaics)
3D reconstruction
Motion tracking
Object recognition
Indexing and database retrieval
Robot navigation
… other

14. Features
Point/patch,
Edge/curve
Region

15. Want Uniqueness Look for image regions that are unusual
Lead to unambiguous matches in other images
How to define “unusual”?

17. Corner Detection
Basic idea: Find points where two edges meet—i.e., high gradient in two directions
“Cornerness” is undefined at a single pixel, because there’s only one gradient per point
Look at the gradient behavior over a small window
Categories image windows based on gradient statistics
Constant: Little or no brightness change
Edge: Strong brightness change in single direction
Flow: Parallel stripes
Corner/spot: Strong brightness changes in orthogonal directions

18. Matching Criterion (weighted) Summed Square Difference – SSD
Compare an image patch against itself

19. Local Measures of Uniqueness
Suppose we only consider a small window of pixels
What defines whether a feature is a good or bad candidate?
High level idea is that corners are good. You want to find windows that contain strong gradients AND gradients oriented in more than one direction
Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.

20. Feature Detection Local measure of feature uniqueness
How does the window change when you shift it?
Shifting the window in any direction causes a big change
“flat” region: no change in all directions
“edge”:
no change along the edge direction
“corner”: significant change in all directions
Slide adapted from Darya Frolova, Denis Simakov, Weizmann Institute.

21. Feature Detection: Mathematics
Consider shifting the window W by (u,v)
how do the pixels in W change?
compare each pixel before and after by summing up the squared differences (SSD)
this defines an SSD “error” of E(u,v): W

22. Harris Detector: Mathematics
Change of intensity for the shift [u,v]:
Auto-correlation function
Window function
Intensity
Shifted intensity or
Window function w(x,y) =
Gaussian
1 in window, 0 outside

23. Auto-Correlation Function

24. Auto-Correlation Function
Good unique minimum
1D aperture problem
No good peak

25. Small Motion Assumption
Taylor Series expansion of I:
If the motion (u,v) is small, then first order approx is good

26. Feature Detection: Mathematics
Image gradient
Harris detector with a [-2,-1,0,1,2] filter for Ix
Gaussian filter

27. Small Motion Assumption
Plugging this into the formula on

28. Feature Detection: MathematicsW

29. Feature Detection: Mathematics
This can be rewritten: x-
Auto-correlation matrix x+

30. Feature Detection: Mathematics

31. Feature Detection: Mathematics
For the example above
You can move the center of the blue window to anywhere on the blue unit circle
Which directions will result in the largest and smallest E values?
We can find these directions by looking at the eigenvectors of H

32. Quick eigenvalue/eigenvector review
The eigenvectors of a matrix A are the vectors x that satisfy:
The scalar  is the eigenvalue corresponding to x
The eigenvalues are found by solving:
In our case, A = H is a 2x2 matrix, so we have
The solution:
Once you know , you find x by solving

33. Feature Detection: Mathematics
This can be rewritten: x- x+
Eigenvalues and eigenvectors of H=A
Define shifts with the smallest and largest change (E value)
x+ = direction of largest increase in E.
+ = amount of increase in direction x+
x- = direction of smallest increase in E.
- = amount of increase in direction x+

34. Feature Detection: Mathematics
Intensity change in shifting window: eigenvalue analysis
1, 2 – eigenvalues of H
If we try every possible orientation n,
the max. change in intensity is 2
Ellipse E(u,v) = const
(max)-1/2
(min)-1/2

35. Feature Detection: Mathematics2
Classification of image points using eigenvalues of H:
“Edge” 2 >> 1
“Corner” 1 and 2 are large, 1 ~ 2; E increases in all directions
1 and 2 are small; E is almost constant in all directions
“Edge” 1 >> 2
“Flat” region 1

36. Feature Detection: Mathematics
How are +, x+, -, and x+ relevant for feature detection?
What’s our feature scoring function?

37. Feature Detection: Mathematics
How are +, x+, -, and x+ relevant for feature detection?
What’s our feature scoring function?
Want E(u,v) to be large for small shifts in all directions
the minimum of E(u,v) should be large, over all unit vectors [u v]
this minimum is given by the smaller eigenvalue (-) of H

38. Feature Detection Here’s what you do
Compute the gradient at each point in the image
Create the H matrix from the entries in the gradient
Compute the eigenvalues.
Find points with large response (- > threshold)
Choose those points where - is a local maximum as features

39. Feature Detection Here’s what you do
Compute the gradient at each point in the image
Create the A matrix from the entries in the gradient
Compute the eigenvalues.
Find points with large response (- > threshold)
Choose those points where - is a local maximum as features
[Shi and Tomasi 1994]

40. Harris Detector Measure of corner response: Harris and Stephens 1988
(k – empirical constant, k = )
The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22
Very similar to - but less expensive (no square root)

41. Harris Detector: Mathematics2
“Edge”
“Corner”
R depends only on eigenvalues of H
R is large for a corner
R is negative with large magnitude for an edge
|R| is small for a flat region
R < 0
R > 0
“Flat”
“Edge”
|R| small
R < 0 1

42. Harris Detector The Algorithm:
Find points with large corner response function R (R > threshold)
Take the points of local maxima of R

43. Harmonic Mean Smoother function in the region where
Brown, M., Szeliski, R., and Winder, S. (2005).
Smoother function in the region where

44. Isocontours of Response

45. Harris Detector: Workflow

46. Harris Detector: Workflow
Compute corner response R

47. Harris Detector: Workflow
Find points with large corner response: R>threshold

48. Harris Detector: Workflow
Take only the points of local maxima of R

49. Harris Detector: Workflow

50. Example: Gradient Covariances
Corners are where both eigenvalues are big
from Forsyth & Ponce
Detail of image with gradient covar-
iance ellipses for 3 x 3 windows
Full image

51. Example: Corner Detection (for camera calibration)
courtesy of B. Wilburn

52. Example: Corner Detection
courtesy of S. Smith
SUSAN corners

53. Harris Detector: Summary
Average intensity change in direction [u,v] can be expressed as a bilinear form:
Describe a point in terms of eigenvalues of H: measure of corner response
A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive

54. Outline of Feature Detection
Compute the horizontal and vertical derivatives of the image Ix and Iy by convolving the original image with derivatives of Gaussians
Compute the three images corresponding to the outer products of these gradients. (The matrix H is symmetric, so only three entries are needed.)
Convolve each of these images with a larger Gaussian.
Compute a scalar interest measure using one of the formulas discussed above.
Find local maxima above a certain threshold and report them as detected feature point locations.

55. Adaptive Non-Maximal Suppression (ANMS)
(a) Strongest (b) Strongest 500
Local maxima
Response value should be significantly (10%) larger than all of its neighbors within a radius (r)

56. Adaptive Non-Maximal Suppression (ANMS)
(a) Strongest (b) Strongest 500
(c) ANMS 250, r = (d) ANMS 500, r = 16

57. Invariance Suppose you rotate the image by some angle
Will you still pick up the same features?
What if you change the brightness?
Scale?
Rotation: yes (with some caveats—image resampling errors…)
Change brightness: probably, but will have to adjust thresholds
Scale: no, generally.

58. Invariance Repeatability of feature detector:
frequency with which keypoints are detected in one image are found within ε (ε=1.5) pixels of the corresponding location in a transformed image

59. Scale Invariance Detect features at a variety of scales
Multiple resolutions in a pyramid
Matching in all possible levels

60. Multi-Scale Oriented Patches

61. Scale invariant detection
Suppose you’re looking for corners
Show other scales and how the feature looks less corner like

62. Feature Detection Key idea: find scale that gives local maximum of f
f is a local maximum in both position and scale
Common definition of f: Laplacian
Difference between two Gaussian filtered images with different sigmas

63. Feature Detection Stable features in both location and scale
Laplacian of Gaussian (LoG)
Difference of Gaussian

64. Lindeberg et al., 1996 Lindeberg et al, 1996
Slide from Tinne Tuytelaars
Slide from Tinne Tuytelaars

72. Scale-Space Feature Detection
Empirical number of levels in each octave = 3

73. SIFT Feature Detection
Scale Invariant Feature Transform

74. Feature Detector
Sample image Harris response DoG response
Harris and DoG detectors tend to respond at complementary locations.

75. Rotation Invariance and orientation estimation
Dominant orientation
Average gradient within a region around the keypoint (larger window than detection window to make it more reliable)
Orientation histogram

76. Affine Invariance
For wide baseline stereo matching or object recognition

77. Affine Invariance
For a small enough patch, any continuous image wrapping can be well approximated by an affine deformation
Eigenvalue analysis for Hessian matrix or auto-correlation: using the principle axis and ratios to fit the affine coordinate frame
(Lindeberg and Garding 1997; Baumberg 2000; Mikolajczyk and Schmid 2004; Mikolajczyk, Tuytelaars, Schmid et al. 2005; Tuytelaars and Mikolajczyk 2007)

78. Affine Invariance Maximally Stable Extremal Region (MSER) Algorithm
Matas, Chum, Urban et al. (2004)
Algorithm
Binary regions by thresholding the image at all possible gray levels
Region area changes as the threshold changes
D_area/D_t is minimal  maximal stable

79. Keypoint Detection Very active area
(Xiao and Shah 2003; Koethe 2003; Carneiro and Jepson 2005; Kenney, Zuliani, and Manjunath 2005; Bay, Tuytelaars, and Van Gool 2006; Platel, Balmachnova, Florack et al. 2006; Rosten and Drummond 2006
Invariance to transformation such as scaling, rotations, noise, and blur
These experimental results, code, and pointers to the surveyed papers can be found on their Web site at
(Mikolajczyk, Tuytelaars, Schmid et al. 2005)

80. 4.1.2 Feature Descriptor
TBD

81. SIFT Feature Scale Invariant Feature Transform (SIFT)
Distinctive Image Features from Scale-Invariant Keypoints, David G. Lowe
Scale Invariant Feature Transform (SIFT)
Detect features that densely cover the image over the full range of scales and locations

82. Points and Patches
Two approaches to finding feature points and their correspondences
detect in one, then track
For video sequence application, tracking, stereo…
Detect independently, then match
For image stitching, recognition …

83. Keypoint Detection and Matching
Four steps:
Feature detection
Feature description
Feature matching
Feature tracking