1. Shape Descriptors I Thomas Funkhouser CS597D, Fall 2003 Princeton University Thomas Funkhouser CS597D, Fall 2003 Princeton University

2. 3D Representations What properties are required for analysis and retrieval? Intuitive specificationYesNoNoNo Guaranteed continuityYesNoNoNo Guaranteed validityYesNoNoNo Efficient boolean operationsYesNoNoNo Efficient renderingYesYesNoNo AccurateYesYes?? Concise???Yes StructureYesYesYesYes Editing DisplayAnalysis Retrieval Property

3. Shape Analysis Problems Examples: Feature detection Segmentation Labeling Registration Matching  Retrieval Recognition Classification Clustering “How can we find 3D models best matching a query?” 1) 2) 3) 4) Query Ranked Matches

4. Shape Definition from Merriam-Webster’s Dictionary: a : the visible makeup characteristic of a particular item or kind of item b : spatial form or contour

5. Shape Shape is independent of similarity transformation (rotation, scale, translation, mirror) =

6. Shape Similarity Need a shape distance function d(A,B) that: matches our intuitive notion of shape similarity can be computed robustly and efficiently Perhaps, shape distance function should be a metric: Non-negative:d(A,B)  0 for all A and B Identity:d(A,B) = 0 if and only if A=B Symmetry: d(A,B) = d(B,A) for all A and B Triangle inequality: d(A,B) + d(B,C)  d(A,C)

7. Example Distance Functions L p norm: Hausdorff distance: Others (Fréchet, etc.)

8. Shape Matching Compute shape distance function for pair of 3D models Can matching two objects Can find most similar object among a small set Are these the same chair?

9. Shape Retrieval Find 3D models with shape most similar to query Searching large database must take less than O(n) Is this blue chair in the database?

10. Shape Retrieval Build searchable shape index Shape Retrieval Similar Objects Shape Index Shape Descriptor Shape Analysis Shape Analysis Database of 3D Models Geometric Query

11. Shape Retrieval Find 3D models with shape similar to query 3D Query 3D Database Best Matches

12. Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating 3D Query Shape Descriptor 3D Database Best Matches

13. Challenge Need shape descriptor that is:  Concise to store Quick to compute Efficient to match Discriminating 3D Database 3D Query Shape Descriptor Best Matches

14. Challenge Need shape descriptor that is: Concise to store  Quick to compute Efficient to match Discriminating 3D Database 3D Query Shape Descriptor Best Matches

15. Challenge Need shape descriptor that is: Concise to store Quick to compute  Efficient to match Discriminating 3D Database 3D Query Shape Descriptor Best Matches

16. Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match  Discriminating 3D Database 3D Query Shape Descriptor Best Matches

17. Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating  Invariant to transformations Insensitive to noise Insensitive to topology Robust to degeneracies Different Transformations (translation, scale, rotation, mirror)

18. Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating Invariant to transformations  Insensitive to noise Insensitive to topology Robust to degeneracies Scanned Surface Image courtesy of Ramamoorthi et al.

19. Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating Invariant to transformations Insensitive to noise  Insensitive to topology Robust to degeneracies Images courtesy of Viewpoint & Stanford Different Tessellations Different Genus

20. Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating Invariant to transformations Insensitive to noise Insensitive to topology  Robust to degeneracies Images courtesy of Utah & De Espona No Bottom! &*Q?@#A%!

21. Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors

22. Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors Images courtesy of Amenta & Osada

23. Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors Image courtesy of De Espona ?

24. Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors ?

25. Statistical Shape Descriptors Alignment-dependent Voxels Wavelets Moments Extended Gaussian Image Spherical Extent Function Spherical Attribute Image Alignment-independent Shape histograms Harmonic descriptor Shape distributions

26. Feature Vectors Map shape onto point in multi-dimensional space Similarity measure is distance in feature space Feature 2 Feature 1 File cabinets Tables Desks Image courtesy of Mao Chen

27. Feature Vectors Cluster, classify, recognize, and retrieve similar feature vectors using standard methods Feature 2 Feature 1 File cabinets Tables Desks Image courtesy of Mao Chen What feature vectors?

28. Voxels Use voxel values as feature vector (shape descriptor) Feature space has N 3 dimensions (one dimension for each voxel) d(A,B) = ||A-B|| N Example: d =, N AB A-B

29. Voxels Can store distance transform (DT) in voxels ||A- DT (B)|| 1 represents sum of distances from every point on surface of A to closest point on surface of B Distance Transform Surface Image courtesy of Misha Kazhdan

30. Voxels Can store distance transform (DT) in voxels ||A- DT (B)|| 1 represents sum of distances from every point on surface of A to closest point on surface of B Distance Transform Surface Image courtesy of Misha Kazhdan

31. Voxels Can build hierarchical search structure e.g., interior nodes store MIV and MSV Image courtesy of Daniel Keim, SIGMOD 1999

32. Voxel Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

33. Evaluation Metric Precision-recall curves Precision = retrieved_in_class / total_retrieved Recall = retrieved_in_class / total_in_class 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1

34. Evaluation Metric Precision-recall curves Precision = 0 / 0 Recall = 0 / 5 456 7 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 123 98 Ranked Matches Query

35. Evaluation Metric Precision-recall curves Precision = 1 / 1 Recall = 1 / 5 456 7 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 123 98 Ranked Matches Query

36. Evaluation Metric Precision-recall curves Precision = 2 / 3 Recall = 2 / 5 456 7 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 123 98 Ranked Matches Query

37. Evaluation Metric Precision-recall curves Precision = 3 / 5 Recall = 3 / 5 456 7 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 123 98 Ranked Matches Query

38. Evaluation Metric Precision-recall curves Precision = 4 / 7 Recall = 4 / 5 456 7 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 123 98 Ranked Matches Query

39. Evaluation Metric Precision-recall curves Precision = 5 / 9 Recall = 5 / 5 456 7 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 123 98 Ranked Matches Query

40. Voxel Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

41. Voxel Retrieval Results 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 Voxels Random

42. Voxels Properties Discriminating Insensitive to noise Insensitive to topology Robust to degeneracies Quick to compute Efficient to match? XConcise to store XInvariant to transforms

43. Wavelets Define shape with wavelet coefficients 16,000 coefficients400 coefficients100 coefficients20 coefficients Image courtesy of Jacobs, Finkelstein, & Salesin

44. Wavelets Descriptor 1: Given an NxNxN grid, generate an NxNxN array of the wavelet coefficients for the standard Haar basis functions Jacobs, Finkelstein, & Salesin SIGGRAPH 95

45. Wavelets Descriptor 1: Given an NxNxN grid, generate an NxNxN array of the wavelet coefficients for the standard Haar basis functions Descriptor 2: Truncate: Find the m largest coefficients and set all others equal to zero Quantize: Set the non-zero coefficients to +1 or –1 depending on their sign Jacobs, Finkelstein, & Salesin SIGGRAPH 95

46. Jackie Chan Example Original Image (256x256)

47. Truncated And Quantized to 5000

48. Truncated And Quantized to 1000

49. Truncated And Quantized to 500

50. Truncated 100

51. Truncated 50

52. Truncated 10

53. Torus Example

54. Torus Truncated to 5000

55. Torus Truncated to 1000

56. Torus Truncated to 500

57. Torus Truncated to 100

58. Torus Truncated to 50

59. Wavelets Distance Function 1: The query metric is defined by: where A[i,j,k] and B[i,j,k] are the truncated and quantized coefficients and w i,j,k are weights, fine tuned to the database. Jacobs, Finkelstein, & Salesin SIGGRAPH 95

60. Wavelets Distance Function 2: The query metric can be approximated by: to enable efficient indexing and search. Jacobs, Finkelstein, & Salesin SIGGRAPH 95

61. Wavelets Properties Insensitive to noise Insensitive to topology Robust to degeneracies Quick to compute Efficient to match Concise to store Discriminating? XInvariant to transforms Jacobs, Finkelstein, & Salesin SIGGRAPH 95

62. Moments Define shape by moments of inertia:

63. Moments Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

64. Moments Retrieval Results 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 Voxels Moments [Elad et al.] Random

65. Moments Retrieval Results 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 Voxels Moments [Elad et al.] Random

66. Moments Properties Insensitive to topology Robust to degeneracies Quick to compute Efficient to match Concise to store XInsensitive to noise XInvariant to transforms XDiscriminating

67. Extended Gaussian Image Define shape with histogram of normal directions Invertible for convex objects Spherical function 3D Model EGI

68. EGI Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

69. EGI Retrieval Results 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision 1 Voxels Moments [Elad et al.] EGI [Horn 84] Random

70. Extended Gaussian Images Properties Insensitive to topology Quick to compute Efficient to match Concise to store XInsensitve to noise XRobust to degeneracies XInvariant to transforms XDiscriminating

71. Other Rotation-Dependent Descriptors Spherical Extent Functions (Vranic & Saupe, 2000) Shape Histograms (sectors) (Ankherst, 1999)

72. Shape Descriptors II Thomas Funkhouser CS597D, Fall 2003 Princeton University Thomas Funkhouser CS597D, Fall 2003 Princeton University

73. Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors

74. Statistical Shape Descriptors Alignment-dependent Voxels Wavelets Moments Extended Gaussian Image Spherical Extent Function Spherical Attribute Image Alignment-independent Shape histograms Harmonic descriptor Shape distributions

75. Statistical Shape Descriptors Alignment-dependent Voxels Wavelets Moments Extended Gaussian Image Spherical Extent Function Spherical Attribute Image Alignment-independent Shape histograms Harmonic descriptor Shape distributions

76. Alignment Translation (Center of Mass) Scale (Radial Deviation)

77. Alignment Rotation (PCA) Principal axes are eigenvectors associated with largest eigenvalues of 2nd order moments covariance matrix PCA Computation Principal Axis Alignment

78. Rotation (PCA) Principal axes are eigenvectors associated with largest eigenvalues of 2nd order moments covariance matrix Not very robust!

79. Alignment Mirror PCA does not give directions for principal axes Need heuristics to determine positive axes!

80. Alignment-Independent Descriptors Observation: it is difficult to normalize for differences in rotation and mirroring Motivation: build a shape descriptor that is invariant to rotations and mirrors and as discriminating as possible Three mugs aligned automatically with PCA

81. Shape Histograms Shape descriptor stores histogram of how much surface resides at different radii from center of mass Image courtesy of Ankerst et al, 1999 Shape Histograms (shells) (Ankherst, 1999) Radius

82. Shape Histograms Shape descriptor stores histogram of how much surface resides at different radii from center of mass Image courtesy of Misha Kazhdan Shape Descriptor 3D Model Spherical Decomposition 0.7 0.3 0.1

83. Shape Histogram Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

84. Shape Histogram Retrieval Results Precision-recall curves (mean for all queries) 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision Shape Histogram [Ankerst et al.] EGI [Horn] Moments [Elad et al.] Random 1

85. Shape Histograms Properties Insensitive to noise Insensitive to topology Robust to degeneracies Quick to compute Efficient to match Concise to store Invariant to rotations Discriminating?

86. Harmonic Shape Descriptor Key idea: Decompose each sphere into irreducible set of rotation independent components Store “how much” of the model resides in each component 3D Model Shape Descriptor Harmonic Decompositions

87. Step 1: Normalization Normalize for translation and scale 3D Model

88. Step 2: Voxelization Rasterize polygon surfaces into 3D voxel grid 3D Voxel Grid

89. Step 3: Spherical Decomposition Intersect with concentric spheres Spherical Functions

90. Step 4: Frequency Decomposition Represent each spherical function as a sum of harmonic frequencies (orders) Spherical Functions

91. Represent each spherical function as a sum of harmonic frequencies (orders) Step 4: Frequency Decomposition Spherical Function Spherical Functions

92. Represent each spherical function as a sum of harmonic frequencies (orders) Step 4: Frequency Decomposition +++= … Spherical Function Harmonic Decomposition

93. Represent each spherical function as a sum of harmonic frequencies (orders) Step 4: Frequency Decomposition = +++ +++ Constant 1 st Order2 nd Order = … … Spherical Function

94. Represent each spherical function as a sum of harmonic frequencies (orders) Step 4: Frequency Decomposition = +++ +++ Frequency Decomposition = … … Spherical Function Amplitudes are invariant to rotation

95. Step 5: Amplitude Computation Store “how much” (L 2 -norm) of the shape resides in each harmonic frequency of each sphere Frequency Radius Harmonic Shape Descriptor

96. Matching Harmonic Descriptors Define similarity as L 2 -distance between descriptors Enables nearest neighbor indexing and fast search Provides lower bound for L 2 -distance between models, = - - - - Sim

97. Harmonic Shape Descriptor Properties  Concise to store? Quick to compute? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Invariant to transforms? Efficient to match? Discriminating? Frequency Radius 2048 bytes per model (16 frequencies x 32 radii x 4 bytes)

98. Harmonic Shape Descriptor Properties Concise to store  Quick to compute? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Invariant to transforms? Efficient to match? Discriminating? 1.6 seconds (on average) Polygons Voxels Spherical Decomposition Frequency Decomposition Harmonic Shape Descriptor frequency radius

99. Harmonic Shape Descriptor Properties Concise to store  Quick to compute? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Invariant to transforms? Efficient to match? Discriminating? 1.6 seconds (on average) Polygons Voxels Spherical Decomposition Frequency Decomposition Harmonic Shape Descriptor frequency radius

100. Harmonic Shape Descriptor Properties Concise to store Quick to compute  Insensitive to noise  Insensitive to topology  Robust to degeneracies Invariant to transforms? Efficient to match? Discriminating? Rasterize polygon surfaces (no solid reconstruction)

101. Harmonic Shape Descriptor Properties Concise to store Quick to compute Insensitive to noise Insensitive to topology Robust to degeneracies  Invariant to transforms Efficient to match? Discriminating? Rotation Mirror Translation (w/ normalization) Scale (w/ normalization) {

102. Harmonic Shape Descriptor Properties Concise to store Quick to compute Insensitive to noise Insensitive to topology Robust to degeneracies Invariant to transforms  Efficient to match? Discriminating? Indexed Not Indexed 0.23 seconds to search 17,500 models

103. Harmonic Shape Descriptor Properties Concise to store Quick to compute Insensitive to noise Insensitive to topology Robust to degeneracies Invariant to transforms Efficient to match?  Discriminating?

104. Harmonic Matching Results Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

105. Harmonic Retrieval Results Precision-recall curves (mean for all queries) 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision Harmonic Shape Descriptor Shape Histogram [Ankerst et al.] EGI [Horn] Moments [Elad et al.] Random 1

106. Statistical Shape Descriptors Alignment-dependent Voxels Wavelets Moments Extended Gaussian Image Spherical Extent Function Spherical Attribute Image Alignment-independent Shape histograms Harmonic descriptor  Shape distributions

107. Shape Distributions Motivation: general approach to finding a common parameterization for matching 3D SurfaceAudio 2D Contour3D Volume

108. Shape Distributions Key idea: map 3D surfaces to common parameterization by randomly sampling shape function 3D ModelsD2 Shape Distributions Randomly sample shape function Similarity Measure Distance Probability

109. Which Shape Function? Implementation: simple shape functions based on angles, distances, areas, and volumes A3 (angle) D1 (distance) [Ankerst 99] D2 (distance) D3 (area)   D4 (volume)

110. D2 Shape Distribution Properties Concise to store? Quick to compute? Invariant to transforms? Efficient to match? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Discriminating?

111. D2 Shape Distribution Properties  Concise to store?  Quick to compute? Invariant to transforms? Efficient to match? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Discriminating? 512 bytes (64 values) 0.5 seconds (10 6 samples) Distance Probability Skateboard

112. D2 Shape Distribution Properties Concise to store Quick to compute  Invariant to transforms? Efficient to match? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Discriminating? Translation Rotation Mirror { Normalized Means Scale (w/ normalization) Skateboard Porsche

113. Distance Probability Skateboard D2 Shape Distribution Properties Concise to store Quick to compute Invariant to transforms  Efficient to match? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Discriminating? Porsche

114. D2 Shape Distribution Properties Concise to store Quick to compute Invariant to transforms Efficient to match  Insensitive to noise?  Insensitive to topology?  Robust to degeneracies? Discriminating? 1% Noise

115. D2 Shape Distribution Properties Concise to store Quick to compute Invariant to transforms Efficient to match Insensitive to noise Insensitive to topology Robust to degeneracies  Discriminating?

116. D2 Shape Distribution Results Question How discriminating are D2 shape distributions? Test database 133 polygonal models 25 classes 4 Mugs 6 Cars 3 Boats

117. D2 Shape Distribution Results D2 distributions are different across classes D2 shape distributions for 15 classes of objects

118. D2 Shape Distribution Results D2 distributions for 5 tanks (gray) and 6 cars (black) Distance Probability

119. D2 Shape Distribution Results Similarity Matrix Darkness represents similarity Blocks Tanks, cars Airplanes Humans Helicopters

120. D2 Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

121. D2 Retrieval Results Precision-recall curves (mean for all queries) 00.20.40.60.8 0 0.2 0.4 0.6 0.8 1 Recall Precision Harmonic Shape Descriptor D2 Shape Distribution [Osada et al.] Shape Histogram [Ankerst et al.] EGI [Horn] Moments [Elad et al.] Random 1

122. Shape Distributions Next steps: Better shape functions Better comparsion methods Analysis apps

123. D2 Shape Distribution Results D2 shape distributions for 15 classes of objects Line Segment Recognizing gross shapes with D2 distributions

124. D2 Shape Distribution Results Recognizing gross shapes with D2 distributions D2 shape distributions for 15 classes of objects Circle

125. D2 Shape Distribution Results Recognizing gross shapes with D2 distributions D2 shape distributions for 15 classes of objects Cylinder

126. D2 Shape Distribution Results Recognizing gross shapes with D2 distributions D2 shape distributions for 15 classes of objects Sphere

127. D2 Shape Distribution Results Recognizing gross shapes with D2 distributions D2 shape distributions for 15 classes of objects Two Spheres

128. Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors

129. Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,...  Point descriptors Next Time!