1. Computer Vision Lecture 4 FEATURE DETECTION

2. Feature Detection Feature Description Feature Matching Today’s Topics 2

3. How to find camera parameters? Where is the camera, where is it directed at? What is the movement of the camera? Where are the objects located in 3D? What are the dimensions of objects in 3D? What is the 3D structure of a scene? How to process stereo video? How to detect and match image features? How to stitch images? Questions of Interest 3

4. Introduction to Feature Detection 4

5. Image Mozaics

6. Object Recognition

7. Image Matching

8. Image Matching using Feature Points NASA Mars Rover images

9. Feature Points are Used for … Image alignment (e.g., mosaics) 3D reconstruction Motion tracking Object recognition Indexing and database retrieval Robot navigation …

10. Advantages of Local Features Locality features are local, so robust to occlusion and clutter Distinctiveness can differentiate a large database of objects Quantity hundreds or thousands in a single image Efficiency real-time performance achievable Generality exploit different types of features in different situations

11. Point vs. Line Features

12. Processing Stages Feature detection/extraction (look for stable/accurate features) Feature description (aim for invariance to transformations) Feature matching (for photo collection) or tracking (for video)

13. Feature Detection 13

14. Invariant Local Features Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters Features Descriptors

15. Choosing interest points Where would you tell your friend to meet you?

16. Choosing interest points Where would you tell your friend to meet you?

17. Stable (Good) Features: High Accuracy in Localization

18. Stable (Good) Features “flat” region: no change in all directions “edge”: no change along the edge direction “corner”: significant change in all directions (uniqueness) Good feature Aperture problem

19. Autocorrelation: Indicator of “Good”ness

20. Autocorrelation Calculation Autocorrelation can be approximated by sum-squared-difference (SSD):

21. Corner Detection: Mathematics Change in appearance of window w(x,y) for the shift [u,v]: I(x, y) E(u, v) E(3,2) w(x, y)

22. Corner Detection: Mathematics I(x, y) E(u, v) E(0,0) w(x, y) Change in appearance of window w(x,y) for the shift [u,v]:

23. Corner Detection: Mathematics Intensity Shifted intensity Window function or Window function w(x,y) = Gaussian1 in window, 0 outside Source: R. Szeliski Change in appearance of window w(x,y) for the shift [u,v]:

24. Corner Detection: Mathematics We want to find out how this function behaves for small shifts Change in appearance of window w(x,y) for the shift [u,v]: E(u, v)

25. Corner Detection: Mathematics Local quadratic approximation of E(u,v) in the neighborhood of (0,0) is given by the second-order Taylor expansion: We want to find out how this function behaves for small shifts Change in appearance of window w(x,y) for the shift [u,v]:

26. Taylor Expansion

27. Corner Detection: Mathematics Second-order Taylor expansion of E(u,v) about (0,0):

28. Corner Detection: Mathematics Second-order Taylor expansion of E(u,v) about (0,0):

29. Corner Detection: Mathematics Second-order Taylor expansion of E(u,v) about (0,0):

30. The quadratic approximation simplifies to where M is a second moment matrix computed from image derivatives: M Corner Detection: Mathematics

31. Corners as distinctive interest points 2 x 2 matrix of image derivatives (averaged in neighborhood of a point). Notation:

32. The surface E(u,v) is locally approximated by a quadratic form. Let’s try to understand its shape. Interpreting the second moment matrix

33. First, consider the axis-aligned case (gradients are either horizontal or vertical) If either λ is close to 0, then this is not a corner, so look for locations where both are large. Interpreting the second moment matrix

34. Consider a horizontal “slice” of E(u, v): Interpreting the second moment matrix This is the equation of an ellipse.

35. Consider a horizontal “slice” of E(u, v): Interpreting the second moment matrix This is the equation of an ellipse. The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R direction of the slowest change direction of the fastest change ( max ) -1/2 ( min ) -1/2 Diagonalization of M:

36. Interpreting the eigenvalues 1 2 “Corner” 1 and 2 are large, 1 ~ 2 ; E increases in all directions 1 and 2 are small; E is almost constant in all directions “Edge” 1 >> 2 “Edge” 2 >> 1 “Flat” region Classification of image points using eigenvalues of M:

37. Corner response function “Corner” R > 0 “Edge” R < 0 “Flat” region |R| small α : constant (0.04 to 0.06)

38. Harris corner detector 1) Compute M matrix for each image window to get their cornerness scores. 2) Find points whose surrounding window gave large corner response (f> threshold) 3) Take the points of local maxima, i.e., perform non-maximum suppression C.Harris and M.Stephens. “A Combined Corner and Edge Detector.” Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988. “A Combined Corner and Edge Detector.”

39. Harris Detector [ Harris88 ] Second moment matrix 39 1. Image derivatives 2. Square of derivatives 3. Gaussian filter g(  I ) IxIx IyIy Ix2Ix2 Iy2Iy2 IxIyIxIy g(I x 2 )g(I y 2 ) g(I x I y ) 4. Cornerness function – both eigenvalues are strong har 5. Non-maxima suppression (optionally, blur first)

40. For Small Displacements … Taylor Series expansion of I(x,y): If the displacement (u,v) is small, then first order approximation can be used:

41. SSD for Small Displacements

42. Equivalently …

43. Auto-correlation Matrix M Let then and

44. Use Sobel Operator for Gradient Computation Gaussianderivative of Gaussian 01 -202 01 121 000 -2 Horizontal derivativeVertical derivative

45. Eigenvalues and Eigenvectors of the Auto-correlation Matrix where and are the two eigenvalues of. The eigenvector corresponding to gives the direction of largest increase in E, while the eigenvector corresponding to gives the direction of smallest increase in E. lower limit upper limit

46. How are +, e +, -, and e + relevant for feature detection? We want E(u,v) to be large for small shifts in all directions Thus, the minimum of E(u,v) should be large, this minimum is given by the smaller eigenvalue - of M

47. Feature Detection Algorithm Compute the gradient at each point in the image Create the M matrix for each point from the gradients in a window Compute the eigenvalues of each M Find points with large response ( - > threshold) Choose those points as features where - is a local maximum

48. The Harris Operator Called the “Harris Corner Detector” or “Harris Operator” Very similar to - but less expensive (no square root) Lots of other detectors, this is one of the most popular

49. The Harris Operator Harris operator

50. Harris Detector Example

51. f value (red = high, blue = low)

52. Threshold (f > value)

53. Find Local Maxima of f

54. Harris Features (in red)

55. Many Existing Detectors Available K. Grauman, B. Leibe Hessian & Harris [Beaudet ‘78], [Harris ‘88] Laplacian, DoG [Lindeberg ‘98], [Lowe 1999] Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01] Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04] EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02] Salient Regions [Kadir & Brady ‘01] Others…

56. Feature Description 56

57. Feature Descriptor How can we extract feature descriptors that are invariant to variations (transformations) and yet still discriminative enough to establish correct correspondences? ?

58. Transformation Invariance Find features that are invariant to transformations geometric invariance: translation, rotation, scale photometric invariance: brightness, exposure

59. Transformation Invariance We would like to find the same features regardless of the transformation between images. This is called transformational invariance. Most feature methods are designed to be invariant to Translation, 2D rotation, scale They can usually also handle Limited affine transformations (some are fully affine invariant) Limited 3D rotations (SIFT works up to about 60 degrees) Limited illumination/contrast changes

60. Types of invariance Illumination

61. Scale Types of invariance

62. Illumination Scale Rotation Types of invariance

63. Illumination Scale Rotation Affine Types of invariance

64. Illumination Scale Rotation Affine Full Perspective Types of invariance

65. How to Achieve Invariance 1. Make sure the detector is invariant Harris is invariant to translation and rotation Scale is trickier A common approach is to detect features at many scales using a Gaussian pyramid (e.g., MOPS) More sophisticated methods find “the best scale” to represent each feature (e.g., SIFT) 2. Design an invariant feature descriptor A descriptor captures the information in a region around the detected feature point The simplest descriptor: a square window of pixels What’s this invariant to? Let’s look at some better approaches…

66. Rotational Invariance for Feature Descriptors  Find dominant orientation of the image patch. This is given by e +, the eigenvector of M corresponding to +  Rotate the patch to horizontal direction  Intensity normalize the window by subtracting the mean, dividing by the standard deviation in the window CSE 576: Computer Visio 8 pixels 40 pixels Adapted from slide by Matthew Brown

67. Scale Invariance Detecting features at the finest stable scale possible may not always be appropriate e.g., when matching images with little high frequency detail (e.g., clouds), fine-scale features may not exist. One solution to the problem is to extract features at a variety of scales e.g., by performing the same operations at multiple resolutions in a pyramid and then matching features at the same level. This kind of approach is suitable when the images being matched do not undergo large scale changes e.g., when matching successive aerial images taken from an airplane or stitching panoramas taken with a fixed-focal-length camera.

68. Scale Invariance for Feature Descriptors

69. Scale Invariant Detection Key idea: find scale that gives local maximum of f f is a local maximum in both position and scale

70. Scale Invariant Feature Transform (SIFT) Features In order to detect the local maxima and minima of f, each sample point is compared to its eight neighbors in the current image and nine neighbors in the scale above and below It is selected only if it is larger than all of these neighbors or smaller than all of them. Instead of extracting features at many different scales and then matching all of them, it is more efficient to extract features that are stable in both location and scale.

71. Take 16x16 square window around detected feature Compute edge orientation (angle of the gradient - 90  ) for each pixel Throw out weak edges (threshold gradient magnitude) Create histogram of surviving edge orientations Scale Invariant Feature Transform Adapted from slide by David Lowe 0 22 angle histogram

72. SIFT Descriptor Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) Compute an orientation histogram for each cell 16 cells * 8 orientations = 128 dimensional descriptor Adapted from slide by David Lowe

73. Properties of SIFT Extraordinarily robust matching technique Can handle changes in viewpoint Up to about 60 degree out of plane rotation Can handle significant changes in illumination Sometimes even day vs. night (below) Fast and efficient—can run in real time Lots of code available http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT

74. Feature Matching 74

75. Feature Matching Given a feature in image I 1, how to find the best match in image I 2 ? 1. Define a distance function that compares two descriptors 2. Test all the features in I 2, find the one with minimum distance

76. Feature Distance SSD is commonly used to measure the difference between two features f 1 and f 2 Use ratio-of-distance = SSD(f1, f2) / SSD(f1, f2’) as a measure of reliability of the feature match: f 2 is best SSD match to f 1 in I 2 f 2 ’ is 2 nd best SSD match to f 1 in I 2 Ratio-of-distance is small for ambiguous matches f1f1 f2f2 f2'f2' I1I1 I2I2

77. Key trade-offs More Repeatable More Points B1B1 B2B2 B3B3 A1A1 A2A2 A3A3 Detection of interest points More Distinctive More Flexible Description of patches Robust to occlusion Works with less texture Minimize wrong matches Robust to expected variations Maximize correct matches Robust detection Precise localization

78. Evaluating Feature Matching Performance 78

79. Evaluating the Results How can we measure the performance of a feature matcher? 50 75 200 feature distance

80. True/false positives The distance threshold affects performance True positives = # of detected matches that are correct Suppose we want to maximize these—how to choose threshold? False positives = # of detected matches that are incorrect Suppose we want to minimize these—how to choose threshold? 50 75 200 feature distance false match true match

81. Terminology True positives = # of detected matching features False negatives= # of missed matching features False positives = # of detected not matching features True negatives = # of missed not matching features Matching features Detected to have a match

82. Recall, Precision, and F-Number Fraction of detected true positives among all positives fraction of relevant instances that are retrieved Fraction of true positives among all detected as positives fraction of retrieved instances that are relevant Approaches to 1 only when both recall and precision approach to 1

83. True and False Positive Rates Fraction of detected true positives among all positives Fraction of detected false positives among all negatives

84. Evaluating the Performance of a Feature Matcher 0.7 0 1 1 0.1 true positive rate false positive rate

85. Receiver Operating Characteristic (ROC) Curve 0.7 0 1 1 false positive rate true positive rate 0.1 Generated by counting # current/incorrect matches, for different thresholds Want to maximize area under the curve (AUC) Useful for comparing different feature matching methods For more info: http://en.wikipedia.org/wiki/Receiver_operating_characteristichttp://en.wikipedia.org/wiki/Receiver_operating_characteristic