1. Features, Feature descriptors, Matching Jana Kosecka George Mason University

2. MSRI Workshop, January 2005 2 Computer Vision Visual Sensing Images I(x,y) – brightness patterns - image appearance depends on structure of the scene - material and reflectance properties of the objects - position and strength of light sources

3. MSRI Workshop, January 2005 3 photometric properties of the environment geometric properties of the environment What gives rise to images

4. MSRI Workshop, January 2005 4 Basic ingredients Radiance – amount of energy emitted along certain direction Iradiance – amount of energy received along certain direction BRDF – bidirectional reflectance distribution Lambertian surfaces – the appearance depends only on radiance, not on the viewing direction Image intensity for a Lambertian surface

5. MSRI Workshop, January 2005 5 Challenges

6. MSRI Workshop, January 2005 6 Image Primitives and Matching Given an image point in left image, what is the (corresponding) point in the right image, which is the projection of the same 3-D point

7. MSRI Workshop, January 2005 7 Image Primitives and Correspondence Difficulties – ambiguities, large changes of appearance, due to change of viewpoint, non-uniquess

8. MSRI Workshop, January 2005 8 Correspondence Lambertian assumption Rigid body motion Matching - Correspondence radiance

9. MSRI Workshop, January 2005 9 Translational model Affine model Transformation of the intensity values taking into account occlusions and noise Local Deformation Models

10. MSRI Workshop, January 2005 10 Matching and Correspondence Motivated by problems Reconstruction of 3D scene from multiple views Object recognition using (constellation of) features models Varieties Small base-line matching Wide base-line matching – large view point changes For now assuming Lambertian assumption – appearance of a local surface patch is independent of the viewpoint

11. MSRI Workshop, January 2005 11 Translational model RHS approximation by the first two terms of Taylor series Small baseline Brightness constancy constraint Feature Tracking and Optical Flow

12. MSRI Workshop, January 2005 12 Integrate around over image patch Solve Feature Tracking and Optical flow

13. MSRI Workshop, January 2005 13 rank(G) = 0 blank wall problem rank(G) = 1 aperture problem rank(G) = 2 enough texture – good feature candidates Conceptually: In reality: choice of threshold is involved Optical Flow, Feature Tracking

14. MSRI Workshop, January 2005 14 Affine feature tracking Intensity offset Contrast change

15. MSRI Workshop, January 2005 15 Qualitative properties of the motion fields Previous method - assumption locally constant flow Alternative regularization techniques (locally smooth flow fields, integration along contours) Optical Flow

16. MSRI Workshop, January 2005 16 Compute eigenvalues of G If smalest eigenvalue  of G is bigger than  - mark pixel as candidate feature point Alternatively feature quality function (Harris Corner Detector) Point Feature Extraction

17. MSRI Workshop, January 2005 17 Harris Corner Detector - Example

18. MSRI Workshop, January 2005 18 Feature Selection Compute Image Gradient Compute Feature Quality measure for each pixel Search for local maxima Feature Quality Function Local maxima of feature quality function

19. MSRI Workshop, January 2005 19 Feature Tracking Translational motion model Closed form solution 1. Build an image pyramid 2. Start from coarsest level 3. Estimate the displacement at the coarsest level 4. Iterate until finest level

20. MSRI Workshop, January 2005 20 Coarse to fine feature tracking 1. compute 2. warp the window in the second image by 3. update the displacement 4. go to finer level 5. At the finest level repeat for several iterations 0 2 1

21. MSRI Workshop, January 2005 21 Tracked Features

22. MSRI Workshop, January 2005 22 Wide baseline matching Point features detected by Harris Corner detector

23. MSRI Workshop, January 2005 23 Sum of squared differences Normalize cross-correlation Sum of absolute differences Region based Similarity Metric

24. MSRI Workshop, January 2005 24 NCC score for two widely separated views NCC score

25. MSRI Workshop, January 2005 25 Advanced matching techniques ( ) 1. Selected salient image locations - points, pieces of countours 2. Associate Local photometric descriptors 3. Invariance to image transformations + illumination changes NCC - is not invariant with respect to image transformation

26. MSRI Workshop, January 2005 26 Summary of the approach Very good results in the presence of occlusion and clutter local information discriminant greyvalue information robust estimation of the global relation between images for limited view point changes Solution for more general view point changes wide baseline matching (different viewpoint, scale and rotation) local invariant descriptors based on greyvalue information

27. MSRI Workshop, January 2005 27 Local descriptors Greyvalue derivatives Invariance to image rotation : differential invariants [Koenderink87]

28. MSRI Workshop, January 2005 28 Feature Detection and Matching Detection of interest points/regions Harris detector (extension to scale and affine invariance) Computation of descriptors for each point (e.g. diff. invariants, steerable filters, SIFT descriptor) Similarity of descriptors (Euclidean distance, Mahalanobis Distance)

29. MSRI Workshop, January 2005 29 Keypoint Detector and SIFT Descriptor Each image is characterized by a set of scale- invariant keypoints and their associated descriptors [D. Lowe,2000] Keypoints - extrema in DOG pyramid Descriptor – 8 bin orientation histograms computed over 4 x 4 grid overlayed over pixel neighbourhood and stacked together to form a 128 dim feature vector

30. MSRI Workshop, January 2005 30 SIFT Keypoints

31. MSRI Workshop, January 2005 31 Overview Scale invariance is not sufficient for large baseline changes State of the art on affine invariant points/regions Affine invariant interest points Application to recognition

32. MSRI Workshop, January 2005 32 Scale invariant interest points Invariant points + associated regions [Mikolajczyk & Schmid’01] multi-scale Harris points selection of points at the characteristic scale with Laplacian Courtesy of Schimd’01

33. MSRI Workshop, January 2005 33 Viewpoint changes Locally approximated by an affine transformation detected scale invariant regionprojected region Courtesy of Schimd’01

34. MSRI Workshop, January 2005 34 Affine invariant Harris points Localization & scale influence affine neighhorbood => affine invariant Harris points (Mikolajczyk & Schmid’02) Iterative estimation of these parameters 1. localization – local maximum of the Harris measure 2. scale – automatic scale selection with the Laplacian 3. affine neighborhood – normalization with second moment matrix Repeat estimation until convergence Initialization with multi-scale interest points

35. MSRI Workshop, January 2005 35 Alternative features/descriptors Affine invariant regions (Tuytelaars et al.’00) ellipses fitted to intensity maxima parallelogram formed by interest points and edges Maximally stable regions (Matas et al. BMVC’02) regions stable across large range of thresholds, connected components of thresholded image descriptors – rotationaly and affine invariant and color moments

36. MSRI Workshop, January 2005 36 Feature Matches 33 correct matches Courtesy of Schimd’01

37. MSRI Workshop, January 2005 37 Pieces of Countour/Line descriprors Select salient pieces using scale invariant detection techniques Characterize either the intensity profile along contour/or local neighbourhood with sideness information – form the descriptor Type of suitable salient regions depends of the class of objects Computational model of visual attention can guide the process of selecting salient regions

38. MSRI Workshop, January 2005 38 Additional changes of the appearance