1. Shape-Representation and Shape Similarity PART 2 Dr. Rolf Lakaemper

2. Motivation WHY PARTS ?

3. Motivation

6. Global similarity measures fail at: Occlusion Global Deformation Partial Match (actually everything that occurs under ‘real’ conditions)

7. Parts Requirements for a Part Based Shape Representation (Siddiqi / Kimia ’96: ‘Parts of Visual Form: Computational Aspects’)

8. Parts How should parts be defined / computed ? Some approaches: Decomposition of interior Skeletons Maximally convex parts Best combination of primitives Boundary Based High Curvature Points Constant Curvature Segments

9. Parts Principal approach: Hoffman/Richards (’85): ‘Part decomposition should precede part description’ => No primitives, but general principles

10. Parts No primitives, but general principals “When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle)

11. Parts “When two arbitrarily shaped surfaces are made to interpenetrate they always meet in a contour of concave discontinuity of their tangent planes” (transversality principle) Divide a plane curve into parts at negative minima of curvature

12. Parts Different notions of parts: Parts: object is composed of rigid parts Protrusions: object arises from object by deformation due to a (growth) process (morphology) Bends: Parts are result of bending the base object

13. Parts The Shape Triangle

14. Parts This lecture focuses on parts, i.e. on partitioning a shape

15. Framework A Framework for a Partitioning Scheme Scheme must be invariant to 2 classes of changes: Global changes : translations, rotations & scaling of 2D shape, viewpoint,… Local changes: occlusions, movement of parts (rigid/non-rigid deformation)

16. Framework A general decomposition of a shape should be based on the interaction between two parts rather than on their shapes. -> Partitioning by Part Lines

17. Framework Definition 1: A part line is a curve whose end points rest on the boundary of the shape, which is entirely embedded in it, and which divides it into two connected components.

18. Definition 2: A partitioning scheme is a mapping of a connected region in the image to a finite set of connected regions separated by part-lines. Framework

19. Definition 3: A partitioning scheme is invariant if the part lines of a shape that is transformed by a combination of translations, rotations and scalings are transformed in exactly the same manner. Framework

20. Definition 4: A partitioning scheme is robust if for any two shapes A and B, which are exactly the same in some neighborhood N, the part lines contained in N for A and B are exactly equivalent. Framework

21. Definition 5: A partitioning scheme is stable if slight deformations of the boundary of a shape cause only slight changes in its part lines Framework

22. Definition 6: A partitioning scheme is scale- tuned if when moving from coarse to fine scale, part lines are only added, not removed, leading to a hierarchy of parts. Framework

23. A general purpose partitioning scheme that is consistent with these requirements is the partitioning by limbs and necks Framework

24. Definition : A limb is a part-line going through a pair of negative curvature minima with co-circular boundary tangents on (at least) one side of the part-line Limbs and Necks

25. Motivation: co-circularity Limbs and Necks The decomposition of the right figure is no longer intuitive: absence of ‘good continuation’

26. Smooth continuation: an example for form from function Shape of object is given by natural function Different parts having different functions show sharp changes in the 3d surface of the connection Projection to 2d yields high curvature points Limbs and Necks

27. Examples of limb based parts Limbs and Necks

28. Definition : A neck is a part-line which is also a local minimum of the diameter of an inscribed circle Limbs and Necks

29. Motivation for necks: form from function Natural requirements (e.g. space for articulation and economy of mass at the connection) lead to a narrowing of the joint between two parts Limbs and Necks

30. The Limb and Neck partitioning scheme is consistent with the previously defined requirements Invariance Robustness Stability Scale tuning Limbs and Necks

31. Examples: Limbs and Necks

32. The scheme presented does NOT include a similarity measure ! Limbs and Necks

33. Part Respecting Similarity Measures Algorithms

34. Curvature Scale Space (Mokhtarian/Abbasi/Kittler) A similarity measure implicitely respecting parts CSS

35. Creation of reflection-point based feature-vector which implicitely contains part – information

36. CSS Properties: Boundary Based Continous Model (!) Computes Feature Vector compact representation of shape Performs well !

37. CSS The idea: Smooth (continous) boundary curve using convolution with an increasing gaussian kernel Use the runlength position of curvature zero- crossings on the boundary as index set for each kernel size, thereby creating the ‘Curvature Scale Space’ The maxima of the CSS are used for shape representation Similarity of shape is defined by difference between the maxima of the CSS representation

38. CSS Smooth (continous) boundary curve using convolution with an increasing gaussian kernel Boundary curve S: S={(x(u),y(u) | u  [0,1]} Each coordinate of S is convolved with a 1D Gaussion kernel of width d The resulting curve S(d) is smoother than S

39. CSS Inflection points (curvature zero crossings)

40. Use the runlength position of curvature zero- crossings on the boundary as index set for each kernel size, thereby creating the ‘Curvature Scale Space’ CSS

41. The maxima of the CSS are used for shape representation

42. CSS Similarity of shape is defined by difference between the maxima of the CSS representation

43. CSS Similarity of shape is defined by difference between the maxima of the CSS representation

44. CSS Some results (Database: 450 marine animals)

45. CSS Some results (Database: 450 marine animals)

46. CSS Problems of CSS: Convex shapes don’t have inflection points Different shapes can have identical CSS !

47. CSS The main problem: CSS is continous, the computer vision world is discrete. How to measure curvature in discrete boundaries ?

48. Dominant Points Local curvature = average curvature in ‘region of support’ To define regions of support, ‘dominant points’ are needed !

49. Dominant Points (“Things should be expressed as simple as possible, but not simpler”, A. Einstein) Idea: given a discrete boundary S compute polygonal boundary S’ with minimum number of vertices which is visually similar to S.

50. Dominant Points Example Algorithms ( 3 of billions…) Ramer Line Fitting Discrete Curve Evolution

51. DCE Discrete Curve Evolution (Latecki / Lakaemper ’99) Idea: Detect subset of visually significant points

52. Curve Evolution Target: reduce data by elimination of irrelevant features, preserve relevant features... noise reduction... shape simplification:

53. Curve Evolution: Tangent Space Transformation from image-space to tangent-space

54. Tangent Space: Properties In tangent space...... the height of a step shows the turn-angle... monotonic increasing intervals represent convex arcs... height-shifting corresponds to rotation... the resulting curve can be interpreted as 1 – dimensional signal => idea: filter signal in tangent space (demo: 'fishapplet')

55. Curve Evolution: Step Compensation (Nonlinear) filter: merging of 2 steps with area – difference F given by:  pq p + q F F  F  q p

56. Curve Evolution: Step Compensation Interpretation in image – space:... Polygon – linearization... removal of visual irrelevant vertices p q removed vertex

57. Curve Evolution: Step Compensation Interpretation in image – space:... Polygon – linearization... removal of visual irrelevant vertices next: Iterative SC

58. Curve Evolution: Iterative Step Compensation Keep it simple: repeated step compensation ! Remark: there are of course some traps... (demo: EvoApplet)

59. Curve Evolution: Iterative Step Compensation Remark: there are of course some traps: " Self intersection / Topology preservation " Stop parameter " Edge movement

60. The evolution...... reduces the shape-complexity... is robust to noise... is invariant to translation, scaling and rotation... preserves the position of important vertices... extracts line segments... is in accord with visual perception... offers noise-reduction and shape abstraction... is parameter free Curve Evolution: Properties... is translatable to higher dimensions

61. Curve Evolution: Properties Robustness (demo: noiseApplet)(demo: noiseApplet)

62. Curve Evolution: Properties Preservation of position, no blurring !

63. Strong relation to digital lines and segments Curve Evolution: Properties

64. Noise reduction as well as shape abstraction Curve Evolution: Properties

65. Parameter free (?) Curve Evolution: Properties

66. Extendable to higher dimensions Curve Evolution: Properties

67. Extendable to higher dimensions Curve Evolution: Properties

68. Extendable to higher dimensions Curve Evolution: Properties

69. Extendable to higher dimensions Curve Evolution: Properties

70. Result: " The DCE creates a polygonal shape representation in different levels of granularity: Scale Space " Curvature can be defined as the turning angle at the vertices " Regions of support are defined by vertices " Easy traceable Scale Space is created, since no points are relocated Curve Evolution: Properties

71. Scale Space Ordered set of representations on different information levels

72. The polygonal representation achieved by the DCE has a huge advantage : It allows easy boundary partitioning using convex / concave parts (remember the limbs !) (MATLAB Demo MatchingDemo) Polygonal Representation

73. Some results of part line decomposition: DCE

74. The ASR (Advanced Shape Recognition) Algorithm uses the boundary parts achieved by the polygonal representation for a part based similarity measure ! (Note: this is NOT the area partitioning shown in the previous slide) ASR

75. The ASR is used in the ISS Database ASR / ISS

76. The Interface (JAVA – Applet)

77. The Sketchpad: Query by Shape

78. The First Guess: Different Shape - Classes

79. Selected shape defines query by shape – class

80. Result

81. Specification of different shape in shape – class

82. Result

83. Let's go for another shape...

84. ...first guess...

85. ...and final result

86. Query by Shape, Texture and Keyword

87. Result

88. Behind The Scenes of the ISS - Database: Modern Techniques of Shape Recognition and Database Retrieval How does it work ?

89. The 2 nd Step First: Shape Comparison Developed by Hamburg University in cooperation with Siemens AG, Munich, for industrial applications in...... robotics... multimedia (MPEG – 7) ISS implements the ASR (Advanced Shape Recognition) Algorithm

90. Reticent Proudness… MPEG-7: ASR outperformes classical approaches ! Similarity test (70 basic shapes, 20 different deformations): Wavelet Contour Heinrich Hertz Institute Berlin67.67 % Multilayer EigenvectorHyundai70.33 % Curvature Scale SpaceMitsubishi ITE-VIL75.44 % ASRHamburg Univ./Siemens AG76.45 % DAG Ordered TreesMitsubishi/Princeton University60.00 % Zernicke MomentsHanyang University70.22 % (Capitulation :-)IBM--.-- %

91. Wide range of applications...... recognition of complex and arbitrary patterns... invariance to basic transformations... results which are in accord with human perception... parameter-free operation Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception Industrial requirements...... robustness... low processing time... applicable to three main tasks of recognition

92. Wide range of applications...... recognition of complex and arbitrary patterns... invariance to basic transformations... results which are in accord with human perception... parameter-free operation Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception Industrial requirements...... robustness... low processing time Next: Strategy Scaling (or resolution) Rotation Rigid / non-rigid deformation... applicable to three main tasks of recognition

93. Wide range of applications...... recognition of complex and arbitrary patterns... results which are in accord with human perception... applicable to three main tasks of recognition... parameter-free operation Requirements Robust automatic recognition of arbitrary shaped objects which is in accord with human visual perception... robustness Industrial requirements...... robustness... low processing time... invariance to basic transformations... low processing time Simple Recognition (yes / no) Common Rating (best of...) Analytical Rating (best of, but...)

94. Different Approaches... Correlation Pattern Matching... Geometrical description...... Hough – Transformation Feature – Vectors...... (Zernicke - ) Moments Based on Visual Parts...... Mokhtarian... ASR

95. Curvature Scale Space (Mokhtarian, Mitsubishi) Creation of reflection-point based feature-vector which implicitely contains part – information

96. ASR: Strategy Source:2D - Image Arc – Matching Contour – Segmentation Contour Extraction Object - Segmentation Evolution

97. ASR: Strategy Arc – Matching DCE Contour – Segmentation

98. Contour Segmentation Correspondence ? Similarity of parts ?

99. Part Similarity Similarity of parts ? = Boundary Similarity Measure = Similarity of polygons

100. Tangent Space Transformation from image-space to tangent-space

101. Shape Comparison: Measure Tangent space offers an intuitive measure:

102. Shape Comparison: Measure Drawback: not adaptive to unequally distributed noise if used globally ! …but works for single parts

103. Shape Comparison: Contour Segmentation

104. Shape Comparison: Correspondence Optimal arc-correspondence: find one to many (many to one) correspondence, that minimizes the arc-measure ! next: Corr. -example

105. Graph of Correspondence a0 a1 a2 a3 b0 b1 b2 b3 a0 b0 a1 a2 a3 b1 b2 b3 Graph:... edge represents correspondence... node represents matched arcs arc correspondence

106. Shape Comparison: Correspondence Example: a0 a1 a2 a3 b0 b1 b2 b3 a0 b0 a1 a2 a3 b1 b2 b3

107. Shape Comparison: Correspondence Result: Optimal correspondence is given by cheapest way next: Corr. - Results

108. Correspondence: Results

109. (MATLAB Demo)

110. Correspondence: Results Correspondence and arc-measure allow...... the identification of visual parts as well as... the identification of the entire object... a robust recognition of defective parts... a shape matching which is in accord with human perception

111. ASR Results " Correspondence and arc-measure meet the requirements stated by Kimia et al. " Discrete " Easy computable

112. ASR: Applications in Computer Vision Robotics: Shape Screening (Movie: Robot2.avi) Straightforward Training Phase Recognition of Rough Differences Recognition of Differences in Detail Recognition of Parts

113. ASR: Applications in Computer Vision Application 2: View Invariant Human Activity Recognition (Dr. Cen Rao and Mubarak Shah, School of Electrical Engineering and Computer Science, University of Central Florida)

114. Application: Human Activity Recognition Human Action Defined by Trajectory Action Recognition by Comparison of Trajectories (Movie: Trajectories) Rao / Shah: Extraction of ‘Dynamic Instants’ by Analysis of Spatiotemporal Curvature Comparison of ‘Dynamic Instants’ (Sets of unconnected points !) ASR: Simplification of Trajectories by Curve Evolution Comparison of Trajectories

115. Application: Human Activity Recognition Trajectory Simplification

116. Activity Recognition: Typical Set of Trajectories

117. Trajectories in Tangent Space

118. Trajectory Comparison by ASR: Results

119. Recognition of 3D Objects by Projection Background: MPEG 7 uses fixed view angles Improvement: Automatic Detection of Key Views

120. Automatic Detection of Key Views (Pairwise) Comparison of Adjacent Views Detects Appearance of Hidden Parts

121. Automatic Detection of Key Views Result (work in progress):

122. The Main Application: Back to ISS Task: Create Image Database Problem: Response Time Comparison of 2 Shapes: 23ms on Pentium1Ghz ISS contains 15,000 images: Response Time about 6 min. Clustering not possible: ASR failed on measuring dissimilarities !

123. Solution: Full search on entire database using a simpler comparison Vantage Objects (Vleugels / Veltkamp, 2000) provide a simple comparison of n- dimensional vectors (n typically < 100) Vantage Objects

124. The Idea: Compare the query-shape q to a predefined subset S of the shapes in the database D The result is an n-dimensional Vantage Vector V, n = | S | Vantage Objects q s1s1 s2s2 s3s3 snsn … v1v1 v2v2 v3v3 vnvn

125. - - Each shape can be represented by a single Vantage Vector - -The computation of the Vantage Vector calls the ASR – comparison only n times - -ISS uses 54 Vantage Objects, reducing the comparison time (needed to create the Vantage Vector) to < 1.5s - -How to compare the query object to the database ? Vantage Objects

126. - - Create the Vantage Vector v i for every shape d i in the database D - - Create the Vantage Vector v q for the query-shape q - - compute the (euclidean) distance between v q and v i - - best response is minimum distance - Note: computing the Vantage Vectors for the database objects is an offline process ! Vantage Objects

127. - How to define the set S of Vantage Objects ? Vantage Objects

128. - Algorithm 1 (Vleugels / Veltkamp 2000): - Predefine the number n of Vantage Objects - S 0 = { } - Iteratively add shapes d i  D\S i-1 to S i-1 such that - S i = S i-1  d i - and -  k=1..i-1  e (d i, s k ) maximal. (  e = eucl. dist.) Stop if i = n. Vantage Objects

129. - Result: - Did not work for ISS. Vantage Objects

130. - Algorithm 2 (Latecki / Henning / Lakaemper): Def.: A(s 1,s 2 ): ASR distance of shapes s 1,s 2 q: query shape ‘Vantage Query’ : determining the result r by minimizing e(v q, v i ) v i = Vantage Vector to s i ‘ASR Query’: determining the result r by minimizing A(q,d i ) Vantage Query has certain loss of retrieval quality compared to ASR query. - Define a loss function l to model the extent of retrieval performance Vantage Objects

131. Given a Database D and a set V of Vantage Vectors, the loss of retrieval performance for a single query by shape q is given by: l V,D (q) = A(q,r), Where r denotes the resulting shape of the vantage query to D using q. Property: l V,D (q) is minimal if r is the result of the ASR-Query. Vantage Objects

132. Now define retrieval error function L(S) of set S={s 1,…, s n }  D of Vantage Vectors of Database D: L(S) = 1/n  l S,D\{si} (s i ) Task: Find subset S  D such that L(S) is minimal. Vantage Objects

133. Algorithm: V0={ } iteratively determine s j in D\S j-1 such that S j =S j-1  s j and L(V j ) minimal. Stop if improvement is low Vantage Objects

134. Result: Worked fine for ISS, though handpicked objects stil performed better. Vantage Objects Handpicked Algorithm 2 Number of Vantage Objects L(S)

135. …some of the Vantage Objects used in ISS: Vantage Objects