1. Locating and Describing Interest Points Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem 03/02/10 Acknowledgment: Many keypoint slides from Grauman&Leibe 2008 AAAI Tutorial

2. What is “object recognition”?

3. 1. Identify a Specific Instance General objects – Challenges: rotation, scale, occlusion, localization – Approaches Geometric configurations of keypoints (Lowe 2004) – Works well for planar, textured objects

4. 1. Identify a Specific Instance Faces – Typical scenario: few examples per face, identify or verify test example – What’s hard: changes in expression, lighting, age, occlusion, viewpoint – Basic approaches (all nearest neighbor) 1.Project into a new subspace (or kernel space) (e.g., “Eigenfaces”=PCA) 2.Measure face features 3.Make 3d face model, compare shape+appearance (e.g., AAM)

5. 2. Detect Instance of a Category Much harder than specific instance recognition Challenges – Everything in instance recognition – Intraclass variation – Representation becomes crucial

6. 2. Detect Instance of a Category Template or sliding window Works well when – Object fits well into rectangular window – Interior features are discriminative Schneiderman Kanade 2000

7. 2. Detect Instance of a Category Parts-based Fischler and Elschlager 1973 Felzenszwalb et al. 2008

8. 3. Assign a label to a pixel or region Stuff – Materials, object regions, textures, etc. – Approaches Label patches + CRF Segmentation + Label Regions

9. Next two lectures Object instance recognition – Interest points (today) Detecting Representing Simple matching – Recognizing objects with interest points (Thurs) Geometric verification – RANSAC + Hough voting Efficient matching

10. General Process of Object Recognition Specify Object Model Generate Hypotheses Score Hypotheses Resolution

11. General Process of Object Recognition Example: Template Matching Specify Object Model Generate Hypotheses Score Hypotheses Resolution Intensity Template, at x-y Scanning window Normalized X-Corr Threshold + Non- max suppression

12. General Process of Object Recognition Example: Keypoint-based Instance Recognition Specify Object Model Generate Hypotheses Score Hypotheses Resolution B1B1 B2B2 B3B3 A1A1 A2A2 A3A3 Affine Parameters Choose hypothesis with max score above threshold # Inliers Affine-variant point locations

13. General Process of Object Recognition Example: Keypoint-based Instance Recognition Specify Object Model Generate Hypotheses Score Hypotheses Resolution B1B1 B2B2 B3B3 A1A1 A2A2 A3A3 Today’s Class

14. Overview of Keypoint Matching K. Grauman, B. Leibe B1B1 B2B2 B3B3 A1A1 A2A2 A3A3 1. Find a set of distinctive key- points 3. Extract and normalize the region content 2. Define a region around each keypoint 4. Compute a local descriptor from the normalized region 5. Match local descriptors

15. Main challenges Change in position and scale Change in viewpoint Occlusion Articulation

16. Goals for Keypoints Detect points that are repeatable and distinctive

17. Key trade-offs More Points More Repeatable B1B1 B2B2 B3B3 A1A1 A2A2 A3A3 Localization More Robust More Selective Description Robust to occlusion Works with less texture Robust detection Precise localization Deal with expected variations Maximize correct matches Minimize wrong matches

18. Keypoint localization More Points More Repeatable Key Trade-off Goals: –Repeatable detection –Precise localization –Interesting content

19. Keypoint Localization Goals: –Repeatable detection –Precise localization –Interesting content K. Grauman, B. Leibe

20. Keypoint Localization Goals: –Repeatable detection –Precise localization –Interesting content  Look for two-dimensional signal changes K. Grauman, B. Leibe

21. Choosing interest points If you wanted to meet a friend would you say a)“Let’s meet on campus.” b)“Let’s meet on Green street.” c)“Let’s meet at Green and Wright.” – Corner detection Or if you were in a secluded area: a)“Let’s meet in the Plains of Akbar.” b)“Let’s meet on the side of Mt. Doom.” c)“Let’s meet on top of Mt. Doom.” – Blob (valley/peak) detection

22. Choosing interest points Corners – “Let’s meet at Green and Wright.” Peaks/Valleys – “Let’s meet on top of Mt. Doom.”

23. Many Existing Detectors Available K. Grauman, B. Leibe Hessian & Harris [Beaudet ‘78], [Harris ‘88] Laplacian, DoG [Lindeberg ‘98], [Lowe 1999] Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01] Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04] EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02] Salient Regions [Kadir & Brady ‘01] Others…

24. Hessian Detector [ Beaudet78] Hessian determinant K. Grauman, B. Leibe I xx I yy I xy Intuition: Search for strong derivatives in two orthogonal directions

25. Hessian Detector [ Beaudet78] Hessian determinant K. Grauman, B. Leibe I xx I yy I xy In Matlab:

26. Hessian Detector – Responses [ Beaudet78] Effect: Responses mainly on corners and strongly textured areas.

27. Hessian Detector – Responses [ Beaudet78]

28. Harris Detector [ Harris88 ] Second moment matrix (autocorrelation matrix) K. Grauman, B. Leibe Intuition: Search for local neighborhoods where the image content has two main directions (eigenvectors).

29. Harris Detector [ Harris88 ] Second moment matrix (autocorrelation matrix) K. Grauman, B. Leibe 1.Image derivatives g x (  D ), g y (  D ), IxIx IyIy

30. Harris Detector [ Harris88 ] Second moment matrix (autocorrelation matrix) 30K. Grauman, B. Leibe 1.Image derivatives g x (  D ), g y (  D ), IxIx IyIy 30 2. Square of derivatives Ix2Ix2 Iy2Iy2 IxIyIxIy

31. Harris Detector [ Harris88 ] Second moment matrix (autocorrelation matrix) 1.Image derivatives g x (  D ), g y (  D ), 2. Square of derivatives IyIy 1. Image derivatives 2. Square of derivatives 3. Gaussian filter g(  I ) IxIx IyIy Ix2Ix2 Iy2Iy2 IxIyIxIy g(I x 2 )g(I y 2 )g(I x I y ) 31

32. Harris Detector [ Harris88 ] Second moment matrix (autocorrelation matrix) 32 1. Image derivatives 2. Square of derivatives 3. Gaussian filter g(  I ) IxIx IyIy Ix2Ix2 Iy2Iy2 IxIyIxIy g(I x 2 )g(I y 2 ) g(I x I y ) 4. Cornerness function – both eigenvalues are strong har 5. Non-maxima suppression g(I x I y )

33. Harris Detector: Mathematics 1.Want large eigenvalues, and small ratio 2.We know 3.Leads to (k :empirical constant, k = 0.04-0.06)

34. Harris Detector – Responses [ Harris88 ] Effect: A very precise corner detector.

35. Harris Detector – Responses [ Harris88 ]

36. So far: can localize in x-y, but not scale

37. Automatic Scale Selection K. Grauman, B. Leibe How to find corresponding patch sizes?

38. Automatic Scale Selection Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

39. Automatic Scale Selection Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

40. Automatic Scale Selection Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

41. Automatic Scale Selection Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

42. Automatic Scale Selection Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

43. Automatic Scale Selection Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

44. What Is A Useful Signature Function? Laplacian-of-Gaussian = “blob” detector K. Grauman, B. Leibe

45. Laplacian-of-Gaussian (LoG) Local maxima in scale space of Laplacian-of- Gaussian K. Grauman, B. Leibe       List of (x, y, s)

46. Results: Laplacian-of-Gaussian K. Grauman, B. Leibe

47. Difference-of-Gaussian (DoG) Difference of Gaussians as approximation of the Laplacian-of-Gaussian K. Grauman, B. Leibe - =

48. DoG – Efficient Computation Computation in Gaussian scale pyramid K. Grauman, B. Leibe  Original image Sampling with step    =2   

49. Results: Lowe’s DoG K. Grauman, B. Leibe

50. T. Tuytelaars, B. Leibe Orientation Normalization Compute orientation histogram Select dominant orientation Normalize: rotate to fixed orientation 0 2  [Lowe, SIFT, 1999]

51. Harris-Laplace [Mikolajczyk ‘01] 1.Initialization: Multiscale Harris corner detection     Computing Harris functionDetecting local maxima

52. Harris-Laplace [Mikolajczyk ‘01] 1.Initialization: Multiscale Harris corner detection 2.Scale selection based on Laplacian (same procedure with Hessian  Hessian-Laplace) K. Grauman, B. Leibe Harris points Harris-Laplace points

53. Maximally Stable Extremal Regions [Matas ‘02] Based on Watershed segmentation algorithm Select regions that stay stable over a large parameter range K. Grauman, B. Leibe

54. Example Results: MSER 54K. Grauman, B. Leibe

55. Comparison LoG Hessian MSER Harris

56. Available at a web site near you… For most local feature detectors, executables are available online: – http://robots.ox.ac.uk/~vgg/research/affine – http://www.cs.ubc.ca/~lowe/keypoints/ – http://www.vision.ee.ethz.ch/~surf K. Grauman, B. Leibe

57. Local Descriptors The ideal descriptor should be – Robust – Distinctive – Compact – Efficient Most available descriptors focus on edge/gradient information – Capture texture information – Color rarely used K. Grauman, B. Leibe

58. Local Descriptors: SIFT Descriptor [Lowe, ICCV 1999] Histogram of oriented gradients Captures important texture information Robust to small translations / affine deformations K. Grauman, B. Leibe

59. Details of Lowe’s SIFT algorithm Run DoG detector – Find maxima in location/scale space – Remove edge points Find all major orientations – Bin orientations into 36 bin histogram Weight by gradient magnitude Weight by distance to center (Gaussian-weighted mean) – Return orientations within 0.8 of peak Use parabola for better orientation fit For each (x,y,scale,orientation), create descriptor: – Sample 16x16 gradient mag. and rel. orientation – Bin 4x4 samples into 4x4 histograms – Threshold values to max of 0.2, divide by L2 norm – Final descriptor: 4x4x8 normalized histograms Lowe IJCV 2004

60. Matching SIFT Descriptors Nearest neighbor (Euclidean distance) Threshold ratio of nearest to 2 nd nearest descriptor Lowe IJCV 2004

61. SIFT Repeatability Lowe IJCV 2004

62. SIFT Repeatability

63. Lowe IJCV 2004

64. SIFT Repeatability Lowe IJCV 2004

65. Local Descriptors: SURF K. Grauman, B. Leibe Fast approximation of SIFT idea  Efficient computation by 2D box filters & integral images  6 times faster than SIFT  Equivalent quality for object identification [Bay, ECCV’06], [Cornelis, CVGPU’08] GPU implementation available  Feature extraction @ 200Hz (detector + descriptor, 640×480 img)  http://www.vision.ee.ethz.ch/~surf

66. Local Descriptors: Shape Context Count the number of points inside each bin, e.g.: Count = 4 Count = 10... Log-polar binning: more precision for nearby points, more flexibility for farther points. Belongie & Malik, ICCV 2001 K. Grauman, B. Leibe

67. Local Descriptors: Geometric Blur Example descriptor ~ Compute edges at four orientations Extract a patch in each channel Apply spatially varying blur and sub-sample (Idealized signal) Berg & Malik, CVPR 2001 K. Grauman, B. Leibe

68. Choosing a detector What do you want it for? – Precise localization in x-y: Harris – Good localization in scale: Difference of Gaussian – Flexible region shape: MSER Best choice often application dependent – Harris-/Hessian-Laplace/DoG work well for many natural categories – MSER works well for buildings and printed things Why choose? – Get more points with more detectors There have been extensive evaluations/comparisons – [Mikolajczyk et al., IJCV’05, PAMI’05] – All detectors/descriptors shown here work well

69. Comparison of Keypoint Detectors Tuytelaars Mikolajczyk 2008

70. Choosing a descriptor Again, need not stick to one For object instance recognition or stitching, SIFT or variant is a good choice

71. Things to remember Keypoint detection: repeatable and distinctive – Corners, blobs, stable regions – Harris, DoG Descriptors: robust and selective – spatial histograms of orientation – SIFT

72. Next time Recognizing objects using keypoints