1. Shape-Representation and Shape Similarity Dr. Rolf Lakaemper Part 1: Shapes

2. May I introduce myself… Rolf Lakaemper PhD (Doctorate Degree) 2000 Hamburg University, Germany Currently Assist. Professor at Department of Computer and Information Sciences, Temple University, Philadelphia, USA Main Research Area: Computer Vision

3. Research Goal Teaching robots to recognize the world they see …using SHAPE

4. Motivation WHY SHAPE ?

5. Motivation These objects are recognized by…

6. Motivation These objects are recognized by… TextureColorContextShape XX XX X X X XX

7. Why Shape ? Several applications in computer vision use shape processing: Object recognition Image retrieval Processing of pictorial information Video compression (eg. MPEG-7) … This presentation focuses on object recognition and image retrieval.

8. Motivation Typical Application: Multimedia: Image Database Query by Shape / Texture / … (Color / Keyword)

9. ISS Database Example : ISS-Database http://knight.cis.temple.edu/~shape

10. The Interface (JAVA – Applet)

11. The Sketchpad: Query by Shape

12. The First Guess: Different Shape - Classes

13. Selected shape defines query by shape – class

14. Result

15. ISS Database ISS: Query by Shape / Texture Sketch of Shape Query: by Shape only Result: Satisfying ?

16. ISS Database SHAPE recognition seems to be possible and leads to satisfying results !

17. ISS Database We’ll talk about the ISS Database a bit later, so stay alert !

18. Overview Part 1: General thoughts about shape recognition Feature based approaches Part 2: Part based, direct approaches The ISS database Applications

19. Data Retrieval The most obvious sensor to gain the data for shape recognition is a camera. But shape is not only perceived by visual means: tactical sensors can also provide shape information that are processed in a similar way. robots’ range sensor provide shape information, too. Hence shape is a general, widely applicable object descriptor!

20. Shape Typical problems: How to describe shape ? What is the matching transformation? No one-to-one correspondence Occlusion Noise

21. Shape Partial match: only part of query appears in part of database shape

22. What is Shape ? … let’s start with some properties easy to agree on: Shape describes a spatial region Shape is a (the ?) specific part of spatial cognition Typically addresses 2D space

23. What is Shape ? Shape or Not ? Continuous transformation from shape to two shapes: Is there a point when it stops being a single shape?

24. What is Shape ? But there’s no doubt that a single, connected region is a shape. Right ?

25. What is Shape ? A single, connected region. But a shape ? A question of scale !

26. What is Shape ? There’s no easy, single definition of shape In difference to geometry, arbitrary shape is not covered by an axiomatic system Different applications in object recognition focus on different shape related features Special shapes can be handled Typically, applications in object recognition employ a similarity measure to determine a plausibility that two shapes correspond to each other

27. Similarity So the new question is: What is Shape Similarity ? or How to Define a Similarity Measure

28. Similarity Again: it’s not so simple (sorry). There’s nothing like THE similarity measure

29. Similarity Measure Requirements to a similarity measure Should not incorporate context knowledge (no AI), thus computes generic shape similarity

30. Similarity Measure Requirements to a similarity measure Must be able to deal with noise Must be invariant with respect to basic transformations Next: Strategy Scaling (or resolution) Rotation Rigid / non-rigid deformation

31. Similarity Measure Requirements to a similarity measure Must be able to deal with noise Must be invariant with respect to basic transformations Must be in accord with human perception

32. Similarity Measure Desired Properties of a Similarity Function C (Basri et al. 1998) C should be a metric C should be continuous C should be invariant (to…)

33. Properties Metric Properties S set of patterns Metric: d: S  S  R satisfying 1. Self-identity :  x  S, d(x,x)=0 2. Positivity :  x  y  S, d(x,y)>0 3. Symmetry :  x, y  S, d(x,y)= d(y,x) 4. Triangle inequality :  x, y, z  S, d(x,z)  d(x,y)+d(y,z) Semi-metric: 1, 2, 3 Pseudo-metric: 1, 3, 4 S with fixed metric d is called metric space

34. Properties 1.Self-identity :  x  S, d(x,x)=0 2.Positivity :  x  y  S, d(x,y)>0 …surely makes sense

35. Properties

37. In general: a similarity measure in accordance with human perception is NOT a metric. This leads to deep problems in further processing, e.g. clustering, since most of these algorithms need metric spaces !

38. Similarity Measures: Overview Similarity Measure depends on Shape Representation Boundary Area (discrete: = point set) Structural (e.g. Skeleton) Comparison Model feature vector direct

39. Similarity Measures directfeature based Boundary Spring model, Cum. Angular Function, Chaincode, Arc Decomposition (ASR- Algorithm) Central Dist. Fourier Distance histogram … Area (point set) Hausdorff … Moments Zernike Moments … Structure Skeleton … ---

40. Feature Based Coding This category defines all approaches that determine a feature-vector for a given shape. Two operations need to be defined: a mapping of shape into the feature space and a similarity of feature vectors. RepresentationFeature ExtractionVector Comparison

41. Feature Based Coding Again: TWO operations need to be defined… We hence have TWO TIMES an information reduction of the basic representation, which by itself is already a mapping of the ‘reality’. RepresentationFeature ExtractionVector Comparison

42. Example Vector of Elementary Descriptors Shape A,B given as Area (continous) or Point Sets (discrete)

43. Vector Comparison

44. Similarity (scalar value)

45. Vector Comparison All Feature Vector approaches have similar properties: Provide a compact representation this is especially interesting for database indexing ! Works for any shape Requires complete shapes (global comparison) Sensible to noise (except Zernike moments which are computationally demanding) Map dissimilar shapes to similar feature vectors (!) They can be used as a prefilter for database applications ! Make the choice of a similarity function difficult

46. Direct Comparison End of Feature Based Coding ! Next: Direct Comparison

47. Part II: Behind The Scenes of the ISS - Database: Modern Techniques of Shape Recognition and Database Retrieval

48. Overview Topics: The Shape Recognition Algorithm Implemented in ISS Possible Applications in Different Areas of Computer Vision

49. Results first… Image Database providing query by Keyword Texture Shape Shape is given by user-sketch, a mouse- drawn outline

50. ISS - GUI

51. The Sketchpad: Query by Shape

52. The First Guess: Different Shape - Classes

53. Selected shape defines query by shape – class

54. Result

55. Key Steps Retrieval by Vantage Objects Retrieval by Direct Shape Comparison

56. Wide range of applications...... recognition of complex and arbitrary patterns... invariance to basic transformations... results which are in accord with human perception... parameter-free operation Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception Industrial requirements...... robustness... low processing time... applicable to three main tasks of recognition

57. Wide range of applications...... recognition of complex and arbitrary patterns... invariance to basic transformations... results which are in accord with human perception... parameter-free operation Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception Industrial requirements...... robustness... low processing time Next: Strategy Scaling (or resolution) Rotation Rigid / non-rigid deformation... applicable to three main tasks of recognition

58. Wide range of applications...... recognition of complex and arbitrary patterns... results which are in accord with human perception... applicable to three main tasks of recognition... parameter-free operation Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception... robustness Industrial requirements...... robustness... low processing time... invariance to basic transformations... low processing time Simple Recognition (yes / no) Common Rating (best of...) Analytical Rating (best of, but...)

59. The 2 nd Step First: Shape Comparison Developed by Dr. Latecki / Dr. Lakaemper in cooperation with Siemens AG, Munich, for industrial applications in...... robotics... multimedia (MPEG – 7) ISS implements the ASR (Advanced Shape Recognition) Algorithm

60. MPEG 7 MPEG-7: ASR outperformes classical approaches ! Similarity test (70 basic shapes, 20 different deformations): Wavelet Contour Heinrich Hertz Institute Berlin67.67 % Multilayer EigenvectorHyundai70.33 % Curvature Scale SpaceMitsubishi ITE-VIL75.44 % ASRHamburg Univ./Siemens AG76.45 % DAG Ordered TreesMitsubishi/Princeton University60.00 % Zernicke MomentsHanyang University70.22 % (Capitulation :-)IBM--.-- %

61. The shape similarity algorithm behind the ISS- database is a direct, part based similarity measure.

62. Motivation WHY PARTS ?

63. Motivation

64. Global similarity measures fail at: Occlusion Global Deformation Partial Match (actually everything that occurs under ‘real’ conditions)

65. Requirements for a Part Based Shape Representation Principal approach: Hoffman/Richards (’85): ‘Part decomposition should precede part description’ => No primitives, but general principles

66. Parts No primitives, but general principals “When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle)

67. Parts How should parts be defined ? Some approaches: Decomposition of interior Skeletons Maximally convex parts Best combination of primitives Boundary Based High Curvature Points Constant Curvature Segments

68. Visual Parts Motivated by psychological experiments (Hoffmann/Richards): split bounding-curve into convex / concave arcs

69. ASR: Strategy Source:2D - Image Arc – Matching Contour – Segmentation Contour Extraction Object - Segmentation Evolution

70. Curve Evolution Target: reduce data by elimination of irrelevant features, preserve relevant features... noise reduction... shape simplification:

71. Curve Evolution: Tangent Space Transformation from image-space to tangent-space bild s.22

72. Tangent Space: Properties In tangent space...... the height of a step shows the turn-angle... monotonic increasing intervals represent convex arcs... height-shifting corresponds to rotation... the resulting curve can be interpreted as 1 – dimensional signal => idea: filter signal in tangent space (demo: 'fishapplet')

73. Curve Evolution: Step Compensation New nonlinear filter: merging of 2 steps with area – difference F given by:  pq p + q F F  F  q p

74. Curve Evolution: Step Compensation Interpretation in image – space:... Polygon – linearization... removal of visual irrelevant vertices p q removed vertex

75. Curve Evolution: Step Compensation Interpretation in image – space:... Polygon – linearization... removal of visual irrelevant vertices next: Iterative SC

76. Curve Evolution: Iterative Step Compensation Keep it simple: repeated step compensation ! Remark: there are of course some traps...

77. The evolution...... reduces the shape-complexity... is robust to noise... is invariant to translation, scaling and rotation... preserves the position of important vertices... extracts line segments... is in accord with visual perception... offers noise-reduction and shape abstraction... is parameter free Curve Evolution: Properties... is translatable to higher dimensions

78. Curve Evolution: Properties Robustness (demo: noiseApplet)(demo: noiseApplet)

79. Curve Evolution: Properties Preservation of position, no blurring !

80. Strong relation to digital lines and segments Curve Evolution: Properties

81. Noise reduction as well as shape abstraction Curve Evolution: Properties

82. Parameter free Curve Evolution: Properties

83. Extendable to higher dimensions Curve Evolution: Properties

84. Extendable to higher dimensions Curve Evolution: Properties

85. Extendable to higher dimensions Curve Evolution: Properties

86. Extendable to higher dimensions Curve Evolution: Properties

87. Shape Comparison: Measure Tangent space offers an intuitive measure:

88. Shape Comparison: Measure Drawback: not adaptive to unequally distributed noise Solution: partition bounding curve

89. Shape Comparison: Contour Segmentation Solution: partition bounding curve

90. Shape Comparison: Contour Segmentation Motivated by psychological experiments (Hoffmann/Richards): split bounding-curve into convex / concave arcs

91. Shape Comparison: Correspondence Optimal arc-correspondence: find one to many (many to one) correspondence, that minimizes the arc-measure !

92. Graph of Correspondence a0 a1 a2 a3 b0 b1 b2 b3 a0 b0 a1 a2 a3 b1 b2 b3 Graph:... edge represents correspondence... node represents matched arcs arc correspondence

93. Shape Comparison: Correspondence Example: a0 a1 a2 a3 b0 b1 b2 b3 a0 b0 a1 a2 a3 b1 b2 b3

94. Shape Comparison: Correspondence Result: Optimal correspondence is given by cheapest way

95. Correspondence: Results

96. (Movie Deer.avi)

97. Correspondence: Results Correspondence and arc-measure allow...... the identification of visual parts as well as... the identification of the entire object... a robust recognition of defective parts... a shape matching which is in accord with human perception

98. ASR: Applications in Computer Vision Robotics: Shape Screening (Movie: Robot2.avi) Straightforward Training Phase Recognition of Rough Differences Recognition of Differences in Detail Recognition of Parts

99. ASR: Applications in Computer Vision Application 2: View Invariant Human Activity Recognition (Dr. Cen Rao and Mubarak Shah, School of Electrical Engineering and Computer Science, University of Central Florida)

100. Application: Human Activity Recognition Human Action Defined by Trajectory Action Recognition by Comparison of Trajectories (Movie: Trajectories) Rao / Shah: Extraction of ‘Dynamic Instants’ by Analysis of Spatiotemporal Curvature Comparison of ‘Dynamic Instants’ (Sets of unconnected points !) ASR: Simplification of Trajectories by Curve Evolution Comparison of Trajectories

101. Application: Human Activity Recognition Trajectory Simplification

102. Activity Recognition: Typical Set of Trajectories

103. Trajectories in Tangent Space

104. Trajectory Comparison by ASR: Results

105. Recognition of 3D Objects by Projection Background: MPEG 7 uses fixed view angles Improvement: Automatic Detection of Key Views

106. Automatic Detection of Key Views (Pairwise) Comparison of Adjacent Views Detects Appearance of Hidden Parts

107. Automatic Detection of Key Views Expected Result (work in progress):

108. Conclusion: Research in Shape Similarity has a lot of challenges, some solutions, and for sure is fun ! That’s it, Thanks !