1. Computer Vision Template matching and object recognition Marc Pollefeys COMP 256 Some slides and illustrations from D. Forsyth, T. Tuytelaars, …

2. Computer Vision Aug 26/28-Introduction Sep 2/4CamerasRadiometry Sep 9/11Sources & ShadowsColor Sep 16/18Linear filters & edges(hurricane Isabel) Sep 23/25Pyramids & TextureMulti-View Geometry Sep30/Oct2StereoProject proposals Oct 7/9Tracking (Welch)Optical flow Oct 14/16-- Oct 21/23Silhouettes/carving(Fall break) Oct 28/30-Structure from motion Nov 4/6Project updateProj. SfM Nov 11/13Camera calibrationSegmentation Nov 18/20FittingProb. segm.&fit. Nov 25/27Matching templates(Thanksgiving) Dec 2/4Matching relationsRange data Dec 9Final project Tentative class schedule

3. Computer Vision Discussion assignment 2 Simple stereo using SSD on 11x11 windows and a disparity range of [0,150] without histogram equalization with histogram equalization

4. Computer Vision Assignment 3 Use Hough, RANSAC and EM to estimate noisy line embedded in noise (details on the web by tonight)

5. Computer Vision Reminder EM (Expectation Maximization) Alternate Expectation (determine feature appartenance) Maximization (determine ML model parameters) optimization (weighted with  i ) counting

6. Computer Vision Last class: Recognition by matching templates Classifiers PCA LDA decision boundaries, not prob.dens. dimensionality reduction maximize discrimination

7. Computer Vision Last class: Recognition by matching templates Neural Networks Support Vector Machines Universal approximation property Optimal separating hyperplane (OSH) support vectors Convex problem! also for non-linear boundaries

8. Computer Vision Matching by relations Idea: –find bits, then say object is present if bits are ok Advantage: –objects with complex configuration spaces don’t make good templates internal degrees of freedom aspect changes (possibly) shading variations in texture etc.

9. Computer Vision Simplest Define a set of local feature templates –could find these with filters, etc. –corner detector+filters Think of objects as patterns Each template votes for all patterns that contain it Pattern with the most votes wins

10. Computer Vision Figure from “Local grayvalue invariants for image retrieval,” by C. Schmid and R. Mohr, IEEE Trans. Pattern Analysis and Machine Intelligence, 1997 copyright 1997, IEEE

11. Computer Vision Probabilistic interpretation Write Assume Likelihood of image given pattern

12. Computer Vision Possible alternative strategies Notice: –different patterns may yield different templates with different probabilities –different templates may be found in noise with different probabilities

13. Computer Vision Employ spatial relations Figure from “Local grayvalue invariants for image retrieval,” by C. Schmid and R. Mohr, IEEE Trans. Pattern Analysis and Machine Intelligence, 1997 copyright 1997, IEEE

14. Computer Vision Figure from “Local grayvalue invariants for image retrieval,” by C. Schmid and R. Mohr, IEEE Trans. Pattern Analysis and Machine Intelligence, 1997 copyright 1997, IEEE

15. Computer Vision Example Training examples Test image

16. Computer Vision

17. Computer Vision

18. Computer Vision Finding faces using relations Strategy: –Face is eyes, nose, mouth, etc. with appropriate relations between them –build a specialised detector for each of these (template matching) and look for groups with the right internal structure –Once we’ve found enough of a face, there is little uncertainty about where the other bits could be

19. Computer Vision Finding faces using relations Strategy: compare Notice that once some facial features have been found, the position of the rest is quite strongly constrained. Figure from, “Finding faces in cluttered scenes using random labelled graph matching,” by Leung, T. ;Burl, M and Perona, P., Proc. Int. Conf. on Computer Vision, 1995 copyright 1995, IEEE

20. Computer Vision Detection This means we compare

21. Computer Vision Issues Plugging in values for position of nose, eyes, etc. –search for next one given what we’ve found when to stop searching –when nothing that is added to the group could change the decision –i.e. it’s not a face, whatever features are added or –it’s a face, and anything you can’t find is occluded what to do next –look for another eye? or a nose? –probably look for the easiest to find What if there’s no nose response –marginalize

22. Computer Vision Figure from, “Finding faces in cluttered scenes using random labelled graph matching,” by Leung, T. ;Burl, M and Perona, P., Proc. Int. Conf. on Computer Vision, 1995 copyright 1995, IEEE

23. Computer Vision Pruning Prune using a classifier –crude criterion: if this small assembly doesn’t work, there is no need to build on it. Example: finding people without clothes on –find skin –find extended skin regions –construct groups that pass local classifiers (i.e. lower arm, upper arm) –give these to broader scale classifiers (e.g. girdle)

24. Computer Vision Pruning Prune using a classifier –better criterion: if there is nothing that can be added to this assembly to make it acceptable, stop –equivalent to projecting classifier boundaries.

25. Computer Vision Horses

26. Computer Vision Hidden Markov Models Elements of sign language understanding –the speaker makes a sequence of signs –Some signs are more common than others –the next sign depends (roughly, and probabilistically) only on the current sign –there are measurements, which may be inaccurate; different signs tend to generate different probability densities on measurement values Many problems share these properties –tracking is like this, for example

27. Computer Vision Hidden Markov Models Now in each state we could emit a measurement, with probability depending on the state and the measurement We observe these measurements

28. Computer Vision HMM’s - dynamics

29. Computer Vision HMM’s - the Joint and Inference

30. Computer Vision Trellises Each column corresponds to a measurement in the sequence Trellis makes the collection of legal paths obvious Now we would like to get the path with the largest negative log- posterior Trellis makes this easy, as follows.

31. Computer Vision

32. Computer Vision Fitting an HMM I have: –sequence of measurements –collection of states –topology I want –state transition probabilities –measurement emission probabilities Straightforward application of EM –discrete vars give state for each measurement –M step is just averaging, etc.

33. Computer Vision HMM’s for sign language understanding-1 Build an HMM for each word

34. Computer Vision HMM’s for sign language understanding-2 Build an HMM for each word Then build a language model

35. Computer Vision Figure from “Real time American sign language recognition using desk and wearable computer based video,” T. Starner, et al. Proc. Int. Symp. on Computer Vision, 1995, copyright 1995, IEEE User gesturing For both isolated word recognition tasks and for recognition using a language model that has five word sentences (words always appearing in the order pronoun verb noun adjective pronoun ), Starner and Pentland’s displays a word accuracy of the order of 90%. Values are slightly larger or smaller, depending on the features and the task, etc.

36. Computer Vision HMM’s can be spatial rather than temporal; for example, we have a simple model where the position of the arm depends on the position of the torso, and the position of the leg depends on the position of the torso. We can build a trellis, where each node represents correspondence between an image token and a body part, and do DP on this trellis.

37. Computer Vision

38. Computer Vision Figure from “Efficient Matching of Pictorial Structures,” P. Felzenszwalb and D.P. Huttenlocher, Proc. Computer Vision and Pattern Recognition2000, copyright 2000, IEEE

39. Computer Vision Recognition using local affine and photometric invariant features Hybrid approach that aims to deal with large variations in –Viewpoint Tuytelaars and Van Gool, BMVC2000

40. Computer Vision Recognition using local affine and photometric invariant features Hybrid approach that aims to deal with large variations in –Viewpoint –Illumination

41. Computer Vision Recognition using local affine and photometric invariant features Hybrid approach that aims to deal with large variations in –Viewpoint –Illumination –Background

42. Computer Vision Recognition using local affine and photometric invariant features Hybrid approach that aims to deal with large variations in –Viewpoint –Illumination –Background –and Occlusions

43. Computer Vision Recognition using local affine and photometric invariant features Hybrid approach that aims to deal with large variations in –Viewpoint –Illumination –Background –and Occlusions  Use local invariant features Invariant features = features that are preserved under a specific group of transformations Robust to occlusions and changes in background Robust to changes in viewpoint and illumination

44. Computer Vision Affine geometric deformations Linear photometric changes Transformations for planar objects

45. Computer Vision Local invariant features ‘Affine invariant neighborhood’

46. Computer Vision Local invariant features

47. Computer Vision Local invariant features Geometry-based region extraction –Curved edges –Straight edges Intensity-based region extraction

48. Computer Vision Geometry-based method (curved edges)

49. Computer Vision Geometry-based method (curved edges) 1.Harris corner detection

50. Computer Vision Geometry-based method (curved edges) 2.Canny edge detection

51. Computer Vision Geometry-based method (curved edges) 3.Evaluation relative affine invariant parameter along two edges

52. Computer Vision Geometry-based method (curved edges) 4.Construct 1-dimensional family of parallelogram shaped regions

53. Computer Vision Geometry-based method (curved edges) f 5.Select parallelograms based on local extrema of invariant function

54. Computer Vision Geometry-based method (curved edges) 5.Select parallelograms based on local extrema of invariant function

55. Computer Vision Geometry-based method (straight edges) Relative affine invariant parameters are identically zero!

56. Computer Vision Geometry-based method (straight edges) 1.Harris corner detection

57. Computer Vision Geometry-based method (straight edges) 2.Canny edge detection

58. Computer Vision Geometry-based method (straight edges) 3.Fit lines to edges

59. Computer Vision Geometry-based method (straight edges) 4.Select parallelograms based on local extrema of invariant functions

60. Computer Vision Geometry-based method (straight edges) 4.Select parallelograms based on local extrema of invariant functions

61. Computer Vision Intensity based method 1.Search intensity extrema 2.Observe intensity profile along rays 3.Search maximum of invariant function f(t) along each ray 4.Connect local maxima 5.Fit ellipse 6.Double ellipse size

62. Computer Vision Intensity based method

63. Computer Vision Comparison Intensity-based method More robust Geometry-based method Less computations More environments

64. Computer Vision Robustness “Correct” detection of single environment cannot be guaranteed –Non-planar region –Noise, quantization errors –Non-linear photometric distortion –Perspective-distortion –… All regions of an object / image should be considered simultaneously

65. Computer Vision 1.Extract affine invariant regions 2.Describe region with feature vector of moment invariants e.g. Search for corresponding regions

66. Computer Vision 1.Extract affine invariant regions 2.Describe region with feature vector of moment invariants 3.Search for corresponding regions based on Mahalanobis distance 4.Check cross-correlation (after normalization) 5.Check consistency of correspondences Search for corresponding regions

67. Computer Vision Semi-local constraints = check consistency of correspondences Epipolar constraint ( RANSAC ) based on 7 points Geometric constraints Photometric constraints based on a combination of only 2 regions

68. Computer Vision Experimental validation degrees symmetric correct Number of matches

69. Computer Vision Experimental validation symmetric correct scale Number of matches error

70. Computer Vision Experimental validation symmetric correct Number of matches illumination reference

71. Computer Vision Object recognition and localization ‘Appearance’-based approach = objects are modeled by a set of reference images Voting principle based on number of similar regions More invariance = requires less reference images

72. Computer Vision Object recognition and localization

73. Computer Vision Object recognition and localization

74. Computer Vision Wide-baseline stereo

75. Computer Vision Wide-baseline stereo

76. Computer Vision Wide-baseline stereo

77. Computer Vision = Searching of ‘similar’ images in a database based on image content Local features Similarity = images contain the same object or the same scene Voting principle –Based on the number of similar regions Content-based image retrieval from database

78. Computer Vision Database ( > 450 images) Search image Content-based image retrieval from database

79. Computer Vision Content-based image retrieval from database

80. Computer Vision Content-based image retrieval from database

81. Computer Vision Application: virtual museum guide

82. Computer Vision Next class: Range data Reading: Chapter 21