1. Features CSE 455, Winter 2010 February 1, 2010

2. Neel Joshi, CSE 455, Winter 2010 2 Review From Last Time

3. Neel Joshi, CSE 455, Winter 2010 Measuring height 1 2 3 4 5 5.4 2.8 3.3 Camera height

4. Neel Joshi, CSE 455, Winter 2010 4 The cross ratio  A Projective Invariant  Something that does not change under projective transformations (including perspective projection) P1P1 P2P2 P3P3 P4P4 The cross-ratio of 4 collinear points Can permute the point ordering 4! = 24 different orders (but only 6 distinct values) This is the fundamental invariant of projective geometry

5. Neel Joshi, CSE 455, Winter 2010 5 Monocular Depth Cues  Stationary Cues:  Perspective  Relative size  Familiar size  Aerial perspective  Occlusion  Peripheral vision  Texture gradient

6. Neel Joshi, CSE 455, Winter 2010 6 Camera calibration  Goal: estimate the camera parameters  Version 1: solve for projection matrix Version 2: solve for camera parameters separately –intrinsics (focal length, principle point, pixel size) –extrinsics (rotation angles, translation) –radial distortion

7. Neel Joshi, CSE 455, Winter 2010 7 Calibration using a reference object  Place a known object in the scene  identify correspondence between image and scene  compute mapping from scene to image Issues must know geometry very accurately must know 3D->2D correspondence

8. Neel Joshi, CSE 455, Winter 2010 Alternative: multi-plane calibration Images courtesy Jean-Yves Bouguet, Intel Corp. Advantage Only requires a plane Don’t have to know positions/orientations Good code available online! –Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/ http://www.intel.com/research/mrl/research/opencv/ –Matlab version by Jean-Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html http://www.vision.caltech.edu/bouguetj/calib_doc/index.html –Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/ http://research.microsoft.com/~zhang/Calib/

9. Neel Joshi, CSE 455, Winter 2010 9 Today  Features  What are Features?  Edges  Corners  Generally “interesting” parts of an image All is Vanity, by C. Allan Gilbert, 1873-1929 Readings M. Brown et al. Multi-Image Matching using Multi-Scale Oriented Patches, CVPR 2005Multi-Image Matching using Multi-Scale Oriented Patches

10. Neel Joshi, CSE 455, Winter 2010 10 Edge detection  Convert a 2D image into a set of curves  Extracts salient features of the scene  More compact than pixels

11. Neel Joshi, CSE 455, Winter 2010 11 Image gradient  The gradient of an image:  The gradient direction is given by: how does this relate to the direction of the edge? The edge strength is given by the gradient magnitude The gradient points in the direction of most rapid increase in intensity

12. Neel Joshi, CSE 455, Winter 2010 12 The Canny edge detector  original image (Lena)

13. Neel Joshi, CSE 455, Winter 2010 13 The Canny edge detector norm of the gradient

14. Neel Joshi, CSE 455, Winter 2010 14 The Canny edge detector thresholding

15. Neel Joshi, CSE 455, Winter 2010 15 The Canny edge detector thinning (non-maximum suppression)

16. Neel Joshi, CSE 455, Winter 2010 16 Image matching by Diva SianDiva Sian by swashfordswashford TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAA

17. Neel Joshi, CSE 455, Winter 2010 17 Harder case by Diva SianDiva Sianby scgbtscgbt

18. Neel Joshi, CSE 455, Winter 2010 18 Even harder case “How the Afghan Girl was Identified by Her Iris Patterns” Read the storystory

19. Neel Joshi, CSE 455, Winter 2010 Harder still? NASA Mars Rover images

20. Neel Joshi, CSE 455, Winter 2010 NASA Mars Rover images with SIFT feature matches Figure by Noah Snavely Answer below (look for tiny colored squares…)

21. Neel Joshi, CSE 455, Winter 2010 Can we do image matching with Edges?

22. Neel Joshi, CSE 455, Winter 2010 22 Image Matching

23. Neel Joshi, CSE 455, Winter 2010 23 Image Matching

24. Neel Joshi, CSE 455, Winter 2010 Invariant local features  Find features that are invariant to transformations  geometric invariance: translation, rotation, scale  photometric invariance: brightness, exposure, … Feature Descriptors

25. Neel Joshi, CSE 455, Winter 2010 25 Advantages of local features  Locality  features are local, so robust to occlusion and clutter  Distinctiveness:  can differentiate a large database of objects  Quantity  hundreds or thousands in a single image  Efficiency  real-time performance achievable  Generality  exploit different types of features in different situations

26. Neel Joshi, CSE 455, Winter 2010 26 More motivation…  Feature points are used for:  Image alignment (e.g., mosaics)  3D reconstruction  Motion tracking  Object recognition  Indexing and database retrieval  Robot navigation  … other

27. Neel Joshi, CSE 455, Winter 2010 27 What makes a good feature? Snoop demo

28. Neel Joshi, CSE 455, Winter 2010 28 Want uniqueness  Look for image regions that are unusual  Lead to unambiguous matches in other images  How to define “unusual”?

29. Neel Joshi, CSE 455, Winter 2010 29 Local measures of uniqueness  Suppose we only consider a small window of pixels  What defines whether a feature is a good or bad candidate? Slide adapted form Darya Frolova, Denis Simakov, Weizmann Institute.

30. Neel Joshi, CSE 455, Winter 2010 30 Feature detection “flat” region: no change in all directions “edge”: no change along the edge direction “corner”: significant change in all directions  Local measure of feature uniqueness  How does the window change when you shift it?  Shifting the window in any direction causes a big change Slide adapted form Darya Frolova, Denis Simakov, Weizmann Institute.

31. Neel Joshi, CSE 455, Winter 2010 31 Consider shifting the window W by (u,v) how do the pixels in W change? compare each pixel before and after by summing up the squared differences (SSD) this defines an SSD “error” of E(u,v): Feature detection: the math W

32. Neel Joshi, CSE 455, Winter 2010 32 Taylor Series expansion of I: If the motion (u,v) is small, then first order approx is good Plugging this into the formula on the previous slide… Small motion assumption

33. Neel Joshi, CSE 455, Winter 2010 33 Consider shifting the window W by (u,v) how do the pixels in W change? compare each pixel before and after by summing up the squared differences this defines an “error” of E(u,v): Feature detection: the math W

34. Neel Joshi, CSE 455, Winter 2010 34 Feature detection: the math This can be rewritten: For the example above You can move the center of the green window to anywhere on the blue unit circle Which directions will result in the largest and smallest E values? We can find these directions by looking at the eigenvectors of H

35. Neel Joshi, CSE 455, Winter 2010 35 Quick eigenvalue/eigenvector review  The eigenvectors of a matrix A are the vectors x that satisfy:  The scalar is the eigenvalue corresponding to x  The eigenvalues are found by solving:  In our case, A = H is a 2x2 matrix, so we have  The solution:  Once you know, you find x by solving

36. Neel Joshi, CSE 455, Winter 2010 36 Feature detection: the math This can be rewritten: Eigenvalues and eigenvectors of H Define shifts with the smallest and largest change (E value) x + = direction of largest increase in E. + = amount of increase in direction x + x - = direction of smallest increase in E. - = amount of increase in direction x + x - x +

37. Neel Joshi, CSE 455, Winter 2010 37 Feature detection: the math How are +, x +, -, and x + relevant for feature detection? What’s our feature scoring function?

38. Neel Joshi, CSE 455, Winter 2010 38 Feature detection: the math How are +, x +, -, and x + relevant for feature detection? What’s our feature scoring function? Want E(u,v) to be large for small shifts in all directions the minimum of E(u,v) should be large, over all unit vectors [u v] this minimum is given by the smaller eigenvalue ( - ) of H

39. Neel Joshi, CSE 455, Winter 2010 39 Feature detection summary Here’s what you do Compute the gradient at each point in the image Create the H matrix from the entries in the gradient Compute the eigenvalues. Find points with large response ( - > threshold) Choose those points where - is a local maximum as features

40. Neel Joshi, CSE 455, Winter 2010 40 Feature detection summary Here’s what you do Compute the gradient at each point in the image Create the H matrix from the entries in the gradient Compute the eigenvalues. Find points with large response ( - > threshold) Choose those points where - is a local maximum as features

41. Neel Joshi, CSE 455, Winter 2010 41 The Harris operator - is a variant of the “Harris operator” for feature detection The trace is the sum of the diagonals, i.e., trace(H) = h 11 + h 22 Very similar to - but less expensive (no square root) Called the “Harris Corner Detector” or “Harris Operator” Lots of other detectors, this is one of the most popular

42. Neel Joshi, CSE 455, Winter 2010 42 The Harris operator Harris operator

43. Neel Joshi, CSE 455, Winter 2010 43 Harris detector example

44. Neel Joshi, CSE 455, Winter 2010 44 f value (red high, blue low)

45. Neel Joshi, CSE 455, Winter 2010 45 Threshold (f > value)

46. Neel Joshi, CSE 455, Winter 2010 46 Find local maxima of f

47. Neel Joshi, CSE 455, Winter 2010 47 Harris features (in red)

48. Neel Joshi, CSE 455, Winter 2010 48 Infinite Images

49. Neel Joshi, CSE 455, Winter 2010 49 Image Analogies