1. Shape-Representation and Shape Similarity CIS 601 by Rolf Lakaemper modified by Longin Jan Latecki

2. Motivation WHY SHAPE ?

3. Motivation We’ve seen this already in the introduction of this course: These objects are recognized by…

4. Motivation These objects are recognized by… TextureColorContextShape XX XX X X X XX

5. Motivation Shape is not the only, but a very powerful descriptor of image content

6. Why Shape ? Several applications in computer vision use shape processing: Object recognition Image retrieval Processing of pictorial information Video compression (eg. MPEG-7) …

7. ISS Database Example 2: ISS-Database http://knight.cis.temple.edu/~shape

8. The Interface (JAVA – Applet)

9. ISS Database ISS: Query by Shape Sketch of Shape Query: by Shape only Result: Satisfying ?

10. ISS Database The ISS-Database will be topic of this tutorial

11. Overview Why shape ? What is shape ? Shape similarity (Metrices) Classes of similarity measures (Feature Based Coding) Examples for global similarity

12. Why Shape ? Shape is probably the most important property that is perceived about objects. It allows to predict more facts about an object than other features, e.g. color (Palmer 1999) Thus, recognizing shape is crucial for object recognition. In some applications it may be the only feature present, e.g. logo recognition

13. Why Shape ? Shape is not only perceived by visual means: tactical sensors can also provide shape information that are processed in a similar way. robots’ range sensor provide shape information, too.

14. Shape Typical problems: How to describe shape ? What is the matching transformation? No one-to-one correspondence Occlusion Noise

15. Shape Partial match: only a part of the query appears in a part of the database shape

16. What is Shape ? Plato, "Meno", 380 BC: "figure is the only existing thing that is found always following color“ "figure is limit of solid"

17. What is Shape ? … let’s start with some properties easier to agree on: Shape describes a spatial region Shape is a (the ?) specific part of spatial cognition Typically addresses 2D space

18. What is Shape ? 3D => 2D projection

19. What is Shape ? the original 3D (?) object

20. What is Shape ? Moving on from the naive understanding, some questions arise: Is there a maximum size for a shape to be a shape? Can a shape have holes? Does shape always describe a connected region? How to deal with/represent partial shapes (occlusion / partial match) ?

21. What is Shape ? Shape or Not ? Continuous transformation from shape to two shapes: Is there a point when it stops being a single shape?

22. What is Shape ? But there’s no doubt that a single, connected region is a shape. Right ?

23. What is Shape ? A single, connected region. But a shape ? A question of scale !

24. What is Shape ? There’s no easy, single definition of shape In difference to geometry, arbitrary shape is not covered by an axiomatic system Different applications in object recognition focus on different shape related features Special shapes can be handled Typically, applications in object recognition employ a similarity measure to determine a plausibility that two shapes correspond to each other

25. Similarity So the new question is: What is Shape Similarity ? or How to Define a Similarity Measure

26. Similarity Again: it’s not so simple (sorry). There’s nothing like THE similarity measure

27. Similarity which similarity measure, depends on which required properties, depends on which particular matching problem, depends on which application

28. Similarity... robustness... invariance to basic transformations Simple Recognition (yes / no) Common Rating (best of...) Analytical Rating (best of, but...) …which application

29. Similarity …which problem computation problem: d(A,B) decision problem: d(A,B) <e ? decision problem: is there g: d(g(A),B) <e ? optimization problem: find g: min d(g(A),B)

30. Similarity …which properties: We concentrate here on the computational problem d(A,B)

31. Similarity Measure Requirements to a similarity measure Should not incorporate context knowledge (no AI), thus computes generic shape similarity

32. Similarity Measure Requirements to a similarity measure Must be able to deal with noise Must be invariant with respect to basic transformations Next: Strategy Scaling (or resolution) Rotation Rigid / non-rigid deformation

33. Similarity Measure Requirements to a similarity measure Must be able to deal with noise Must be invariant with respect to basic transformations Must be in accord with human perception

34. Similarity Measure Some other aspects worth consideration: Similarity of structure Similarity of area Can all these aspects be expressed by a single number?

35. Similarity Measure Desired Properties of a Similarity Function C (Basri et al. 1998) C should be a metric C should be continuous C should be invariant (to…)

36. Properties Metric Properties S set of patterns Metric: d: S  S  R satisfying 1. Self-identity :  x  S, d(x,x)=0 2. Positivity :  x  y  S, d(x,y)>0 3. Symmetry :  x, y  S, d(x,y)= d(y,x) 4. Triangle inequality :  x, y, z  S, d(x,z)  d(x,y)+d(y,z) Semi-metric: 1, 2, 3 Pseudo-metric: 1, 3, 4 S with fixed metric d is called metric space

37. Properties 1.Self-identity :  x  S, d(x,x)=0 2.Positivity :  x  y  S, d(x,y)>0 …surely makes sense

38. Properties

40. In general: a similarity measure in accordance with human perception is NOT a metric. This leads to deep problems in further processing, e.g. clustering, since most of these algorithms need metric spaces !

41. Properties

43. Some more properties: One major difference should cause a greater dissimilarity than some minor ones. S must not diverge for curves that are not smooth (e.g. polygons).

44. Similarity Measures Classes of Similarity Measures: Similarity Measure depends on Shape Representation Boundary Area (discrete: = point set) Structural (e.g. Skeleton) Comparison Model feature vector direct

45. Similarity Measures directfeature based Boundary Spring model, Cum. Angular Function, Chaincode, Arc Decomposition (ASR- Algorithm) Central Dist. Fourier Distance histogram … Area (point set) Hausdorff … Moments Zernike Moments … Structure Skeleton … ---

46. Feature Based Coding Feature Based Coding (again…) This category defines all approaches that determine a feature-vector for a given shape. Two operations need to be defined: a mapping of shape into the feature space and a similarity of feature vectors. RepresentationFeature ExtractionVector Comparison

47. Another feature you should have heard of: (Discrete) Moments Shape A,B given as Area (continuous) or Point Sets (discrete)

48. Moments Discrete Point Sets

49. Moments

52. Discrete Moments Exercise: Please compute all 7 moments for the following shapes, compare the vectors using different comparison techniques

53. Discrete Moments Result: each shape is transformed to a 7- dimensional vector. To compare the shapes, compare the vectors (how ?).

54. 3D Distance Histogram Another Example 3D Distance Histogram Shape A,B given as 3D point set

55. 3D Distance Histogram

56. Feature Based Coding Again: Two operations need to be defined: a mapping of shape into the feature space and a similarity of feature vectors. We hence have TWO TIMES an information reduction of the basic representation, which by itself is already a mapping of the ‘reality’. RepresentationFeature ExtractionVector Comparison

57. Direct Comparison End of Feature Based Coding ! Next: Direct Comparison

58. Vector ComparisonDirect Comparison Example 1 Hausdorff Distance Shape A,B given as point sets A={a1,a2,…} B={b1,b2,…}

59. Vector ComparisonFeature Based Coding

60. Vector ComparisonHausdorff Distance

61. Vector ComparisonHausdorff Distance Hausdorff: Unstable with respect to noise (This is easy to fix ! How ?) Problem: Invariance ! Nevertheless: Hausdorff is the motor behind many applications in specific fields (e.g. character recognition)

62. Vector ComparisonBoundary Representation Example 2 Chain code Comparison Shape A,B given as chain codes

63. Vector ComparisonBoundary Representation

64. Getting Boundaries " As output of image segmentation, we obtain objects that can be viewed as bitmaps. " Let f be a bitmap, i.e., a binary image with 0s representing the background. We can obtain the boundary of the object represented with f using Matlab function: B = boundaries(f); the obtained boundary is 8-connected.

65. I = imread('pout.tif'); figure, imshow(I); figure, imhist(I); BW = im2bw(I, 0.45); % makes a binary image % all pixels above 0.4*255 are 1 and % the rest is 0 which is black figure; imshow(BW) ; B = boundaries(BW); figure; imshow(B);

66. Homework 10 " For certain images, objects of interest can be segmented using simple tools. Your task: compute the rabbit's boundary. Link to the image.image

67. The original image. All pixels having a greater red than green ratio After two stages of morphological processing, we are ready to get the contour.

68. Vector ComparisonBoundary Representation A binary image can be converted into a ‘chain code’ representing the boundary. The boundary is traversed and a string representing the curvature is constructed. 0 123 4 567 C 5,6,6,3,3,4,3,2,3,4,5,3,…

69. Chain Code

72. Vector ComparisonBoundary Representation Resulting strings are then compared using classical string-matching techniques. Not very robust.

73. Vector ComparisonBoundary Representation Digital curves suffer from effects caused by digitalization, e.g. rotation:

74. Shape Signatures [st, angle, x0,y0] = signature(B, x0, y0); figure; plot(angle, st); Resulting strings are then compared using string-matching techniques.

76. Vector ComparisonStructural Representation Structural approaches capture the structure of a shape, typically by representing shape as a graph. Typical example: skeletons

77. Vector ComparisonStructural Representation Skeletons Shape A,B primarily given as area or boundary, structure is derived from representation

78. Vector ComparisonStructural Representation The computation can be described as a medial axis transform, a kind of discrete generalized voronoi.

79. Vector ComparisonStructural Representation The graph is constructed mirroring the adjacency of the skeleton’s parts. Edges are labeled according to the qualitative classes. Matching two shapes requires matching two usually different graphs against each other.

80. Vector ComparisonStructural Representation Problems of skeletons: - Pruning

81. Vector ComparisonStructural Representation -Robustness

82. Vector ComparisonShape similarity All similarity measures shown can not deal with occlusions or partial matching (except skeletons ?) ! They are useful (and used) for specific applications, but are not sufficient to deal with arbitrary shapes Solution: Part – based similarity !

83. Shape-Representation and Shape Similarity PART 2: PART BASED SIMILARITY

84. Motivation WHY PARTS ?

85. Motivation

88. Global similarity measures fail at: Occlusion Global Deformation Partial Match (actually everything that occurs under ‘real’ conditions)

89. Parts Requirements for a Part Based Shape Representation (Siddiqi / Kimia ’96: ‘Parts of Visual Form: Computational Aspects’)

90. Parts How should parts be defined / computed ? Some approaches: Decomposition of interior Skeletons Maximally convex parts Best combination of primitives Boundary Based High Curvature Points Constant Curvature Segments

91. Parts Principal approach: Hoffman/Richards (’85): ‘Part decomposition should precede part description’ => No primitives, but general principles

92. Parts No primitives, but general principals “When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle)

93. Parts “When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle) Divide a plane curve into parts at negative minima of curvature

94. Parts Different notions of parts: Parts: object is composed of rigid parts Protrusions: object arises from object by deformation due to a (growth) process (morphology) Bends: Parts are result of bending the base object

95. Parts The Shape Triangle

96. Parts This lecture focuses on parts, i.e. on partitioning a shape

97. Framework A Framework for a Partitioning Scheme Scheme must be invariant to 2 classes of changes: Global changes : translations, rotations & scaling of 2D shape, viewpoint,… Local changes: occlusions, movement of parts (rigid/non-rigid deformation)

98. Framework A general decomposition of a shape should be based on the interaction between two parts rather than on their shapes. -> Partitioning by Part Lines

99. Framework Definition 1: A part line is a curve whose end points rest on the boundary of the shape, which is entirely embedded in it, and which divides it into two connected components.

100. Definition 2: A partitioning scheme is a mapping of a connected region in the image to a finite set of connected regions separated by part-lines. Framework

101. Definition 3: A partitioning scheme is invariant if the part lines of a shape that is transformed by a combination of translations, rotations and scalings are transformed in exactly the same manner. Framework

102. Definition 4: A partitioning scheme is robust if for any two shapes A and B, which are exactly the same in some neighborhood N, the part lines contained in N for A and B are exactly equivalent. Framework

103. Definition 5: A partitioning scheme is stable if slight deformations of the boundary of a shape cause only slight changes in its part lines Framework

104. Definition 6: A partitioning scheme is scale- tuned if when moving from coarse to fine scale, part lines are only added, not removed, leading to a hierarchy of parts. Framework

105. A general purpose partitioning scheme that is consistent with these requirements is the partitioning by limbs and necks Framework

106. Definition : A limb is a part-line going through a pair of negative curvature minima with co-circular boundary tangents on (at least) one side of the part-line Limbs and Necks

107. Motivation: co-circularity Limbs and Necks The decomposition of the right figure is no longer intuitive: absence of ‘good continuation’

108. Smooth continuation: an example for form from function Shape of object is given by natural function Different parts having different functions show sharp changes in the 3d surface of the connection Projection to 2d yields high curvature points Limbs and Necks

109. Examples of limb based parts Limbs and Necks

110. Definition : A neck is a part-line which is also a local minimum of the diameter of an inscribed circle Limbs and Necks

111. Motivation for necks: Form From Function Natural requirements (e.g. space for articulation and economy of mass at the connection) lead to a narrowing of the joint between two parts Limbs and Necks

112. The Limb and Neck partitioning scheme is consistent with the previously defined requirements Invariance Robustness Stability Scale tuning Limbs and Necks

113. Examples: Limbs and Necks

114. The scheme presented does NOT include a similarity measure ! Limbs and Necks

115. Part Respecting Similarity Measures Algorithms

116. Curvature Scale Space (Mokhtarian/Abbasi/Kittler) A similarity measure implicitely respecting parts CSS

117. Creation of reflection-point based feature-vector which implicitly contains part – information

118. CSS Properties: Boundary Based Continuous Model (!) Computes Feature Vector compact representation of shape Performs well !

119. CSS Some results (Database: 450 marine animals)

120. CSS The main problem: CSS is continuous, the computer vision world is discrete. How to measure curvature in discrete boundaries ?

121. Dominant Points Local curvature = average curvature in ‘region of support’ To define regions of support, ‘dominant points’ are needed !

122. Dominant Points (“Things should be expressed as simple as possible, but not simpler”, A. Einstein) Idea: given a discrete boundary S compute polygonal boundary S’ with minimum number of vertices which is visually similar to S.

123. Dominant Points Example Algorithms ( 3 of billions…) Ramer Line Fitting Discrete Curve Evolution

124. DCE Discrete Curve Evolution (Latecki / Lakaemper ’99) Idea: Detect subset of visually significant points

125. Discrete Curve Evolution (DCE) We achieve a comparable level of detail with DCE. Before a similarity measure is applied, the shape of objects is simplified by DCE in order to reduce influence of noise, simplify the shape by removing irrelevant shape features without changing relevant shape features.

126. Curve Evolution Target: reduce data by elimination of irrelevant features, preserve relevant features... noise reduction... shape simplification:

127. Discrete Curve Evolution (DCE) u v w u v w It yields a sequence: P=P 0,..., P m P i+1 is obtained from P i by deleting the vertices of P i that have minimal relevance measure K(v, P i ) = |d(u,v)+d(v,w)-d(u,w)| >

128. Discrete Curve EvolutionDiscrete Curve Evolution: Preservation of position, no blurring

129. Discrete Curve EvolutionDiscrete Curve Evolution: robustness with respect to noise

130. Discrete Curve Evolution: extraction of linear segments

131. Discrete Curve Evolution: mathematical properties Convexity Theorem (trivial) Discrete curve evolution (when applied to a polygon) converges to a convex polygon. Continuity Theorem (nontrivial) Discrete curve evolution is continuous. L. J. Latecki, R.-R. Ghadially, R. Lakämper, and U. Eckhardt: Continuity of the discrete curve evolution. Journal of Electronic Imaging 9, pp. 317-326, 2000. Polygon Recovery (nontrivial) DCE allows to recover polygons from their digital images. L.J. Latecki and A. Rosenfeld: Recovering a Polygon form Noisy Data. Computer Vision and Image Understanding (CVIU) 86, 1-20, 2002.

132. Comparable level of detail for DCE (=stop condition) is based on a threshold on the relevance measure

133. Comparable level of detail for DCE is based on a threshold on the relevance measure

134. Scale Space Approaches to Curve Evolution 1.reaction-diffusion PDEs 2.polygonal analogs of the PDE-evolution (Bruckstein et al. 1995) 3.approximation (e.g., Bengtsson and Eklundh 1991) Main differences: [to 1, 2:] Each vertex of the polygon is moved at a single evolution step, whereas in our approach the remaining vertices do not change their positions. [to 1, 3:] Our approach is parameter-free (we only need a stop condition)

136. The evolution...... reduces the shape-complexity... is robust to noise... is invariant to translation, scaling and rotation... preserves the position of important vertices... extracts line segments... is in accord with visual perception... offers noise-reduction and shape abstraction... is parameter free Curve Evolution: Properties... is translatable to higher dimensions

137. Extendable to higher dimensions Curve Evolution: Properties

138. Extendable to higher dimensions Curve Evolution: Properties

139. Extendable to higher dimensions Curve Evolution: Properties

140. Extendable to higher dimensions Curve Evolution: Properties

141. Scale Space Ordered set of representations on different information levels

142. The polygonal representation achieved by the DCE has a huge advantage : It allows easy boundary partitioning using convex / concave parts (remember the limbs !) Polygonal Representation

143. Some results of part line decomposition: DCE

144. The ASR (Advanced Shape Recognition) Algorithm uses the boundary parts achieved by the polygonal representation for a part based similarity measure ! (Note: this is NOT the area partitioning shown in the previous slide) ASR

145. Behind The Scenes of the ISS - Database: Modern Techniques of Shape Recognition and Database Retrieval How does it work ?

146. The 2 nd Step First: Shape Comparison Developed by Hamburg University in cooperation with Siemens AG, Munich, for industrial applications in...... robotics... multimedia (MPEG – 7) ISS implements the ASR (Advanced Shape Recognition) Algorithm

147. Reticent Proudness… MPEG-7: ASR outperforms classical approaches ! Similarity test (70 basic shapes, 20 different deformations): Wavelet Contour Heinrich Hertz Institute Berlin67.67 % Multilayer EigenvectorHyundai70.33 % Curvature Scale SpaceMitsubishi ITE-VIL75.44 % ASRHamburg Univ./Siemens AG76.45 % DAG Ordered TreesMitsubishi/Princeton University60.00 % Zernicke MomentsHanyang University70.22 % (Capitulation :-)IBM--.-- %

148. ASR: Strategy Source:2D - Image Arc – Matching Contour – Segmentation Contour Extraction Object - Segmentation Evolution

149. ASR: Strategy Arc – Matching DCE Contour – Segmentation

150. Contour Segmentation Correspondence ? Similarity of parts ?

151. Part Similarity Similarity of parts ? = Boundary Similarity Measure = Similarity of polygons

152. The ASR is used in the ISS Database ASR / ISS