Shape Descriptors I Thomas Funkhouser CS597D, Fall 2003 Princeton University Thomas Funkhouser CS597D, Fall 2003 Princeton University

3D Representations What properties are required for analysis and retrieval? Intuitive specificationYesNoNoNo Guaranteed continuityYesNoNoNo Guaranteed validityYesNoNoNo Efficient boolean operationsYesNoNoNo Efficient renderingYesYesNoNo AccurateYesYes?? Concise???Yes StructureYesYesYesYes Editing DisplayAnalysis Retrieval Property

Shape Analysis Problems Examples: Feature detection Segmentation Labeling Registration Matching  Retrieval Recognition Classification Clustering “How can we find 3D models best matching a query?” 1) 2) 3) 4) Query Ranked Matches

Shape Definition from Merriam-Webster’s Dictionary: a : the visible makeup characteristic of a particular item or kind of item b : spatial form or contour

Shape Shape is independent of similarity transformation (rotation, scale, translation, mirror) =

Shape Similarity Need a shape distance function d(A,B) that: matches our intuitive notion of shape similarity can be computed robustly and efficiently Perhaps, shape distance function should be a metric: Non-negative:d(A,B)  0 for all A and B Identity:d(A,B) = 0 if and only if A=B Symmetry: d(A,B) = d(B,A) for all A and B Triangle inequality: d(A,B) + d(B,C)  d(A,C)

Example Distance Functions L p norm: Hausdorff distance: Others (Fréchet, etc.)

Shape Matching Compute shape distance function for pair of 3D models Can matching two objects Can find most similar object among a small set Are these the same chair?

Shape Retrieval Find 3D models with shape most similar to query Searching large database must take less than O(n) Is this blue chair in the database?

Shape Retrieval Build searchable shape index Shape Retrieval Similar Objects Shape Index Shape Descriptor Shape Analysis Shape Analysis Database of 3D Models Geometric Query

Shape Retrieval Find 3D models with shape similar to query 3D Query 3D Database Best Matches

Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating 3D Query Shape Descriptor 3D Database Best Matches

Challenge Need shape descriptor that is:  Concise to store Quick to compute Efficient to match Discriminating 3D Database 3D Query Shape Descriptor Best Matches

Challenge Need shape descriptor that is: Concise to store  Quick to compute Efficient to match Discriminating 3D Database 3D Query Shape Descriptor Best Matches

Challenge Need shape descriptor that is: Concise to store Quick to compute  Efficient to match Discriminating 3D Database 3D Query Shape Descriptor Best Matches

Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match  Discriminating 3D Database 3D Query Shape Descriptor Best Matches

Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating  Invariant to transformations Insensitive to noise Insensitive to topology Robust to degeneracies Different Transformations (translation, scale, rotation, mirror)

Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating Invariant to transformations  Insensitive to noise Insensitive to topology Robust to degeneracies Scanned Surface Image courtesy of Ramamoorthi et al.

Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating Invariant to transformations Insensitive to noise  Insensitive to topology Robust to degeneracies Images courtesy of Viewpoint & Stanford Different Tessellations Different Genus

Challenge Need shape descriptor that is: Concise to store Quick to compute Efficient to match Discriminating Invariant to transformations Insensitive to noise Insensitive to topology  Robust to degeneracies Images courtesy of Utah & De Espona No Bottom!

Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors

Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors Images courtesy of Amenta & Osada

Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors Image courtesy of De Espona ?

Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors ?

Statistical Shape Descriptors Alignment-dependent Voxels Wavelets Moments Extended Gaussian Image Spherical Extent Function Spherical Attribute Image Alignment-independent Shape histograms Harmonic descriptor Shape distributions

Feature Vectors Map shape onto point in multi-dimensional space Similarity measure is distance in feature space Feature 2 Feature 1 File cabinets Tables Desks Image courtesy of Mao Chen

Feature Vectors Cluster, classify, recognize, and retrieve similar feature vectors using standard methods Feature 2 Feature 1 File cabinets Tables Desks Image courtesy of Mao Chen What feature vectors?

Voxels Use voxel values as feature vector (shape descriptor) Feature space has N 3 dimensions (one dimension for each voxel) d(A,B) = ||A-B|| N Example: d =, N AB A-B

Voxels Can store distance transform (DT) in voxels ||A- DT (B)|| 1 represents sum of distances from every point on surface of A to closest point on surface of B Distance Transform Surface Image courtesy of Misha Kazhdan

Voxels Can store distance transform (DT) in voxels ||A- DT (B)|| 1 represents sum of distances from every point on surface of A to closest point on surface of B Distance Transform Surface Image courtesy of Misha Kazhdan

Voxels Can build hierarchical search structure e.g., interior nodes store MIV and MSV Image courtesy of Daniel Keim, SIGMOD 1999

Voxel Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

Evaluation Metric Precision-recall curves Precision = retrieved_in_class / total_retrieved Recall = retrieved_in_class / total_in_class Recall Precision 1

Evaluation Metric Precision-recall curves Precision = 0 / 0 Recall = 0 / Recall Precision Ranked Matches Query

Evaluation Metric Precision-recall curves Precision = 1 / 1 Recall = 1 / Recall Precision Ranked Matches Query

Evaluation Metric Precision-recall curves Precision = 2 / 3 Recall = 2 / Recall Precision Ranked Matches Query

Evaluation Metric Precision-recall curves Precision = 3 / 5 Recall = 3 / Recall Precision Ranked Matches Query

Evaluation Metric Precision-recall curves Precision = 4 / 7 Recall = 4 / Recall Precision Ranked Matches Query

Evaluation Metric Precision-recall curves Precision = 5 / 9 Recall = 5 / Recall Precision Ranked Matches Query

Voxel Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

Voxel Retrieval Results Recall Precision 1 Voxels Random

Voxels Properties Discriminating Insensitive to noise Insensitive to topology Robust to degeneracies Quick to compute Efficient to match? XConcise to store XInvariant to transforms

Wavelets Define shape with wavelet coefficients 16,000 coefficients400 coefficients100 coefficients20 coefficients Image courtesy of Jacobs, Finkelstein, & Salesin

Wavelets Descriptor 1: Given an NxNxN grid, generate an NxNxN array of the wavelet coefficients for the standard Haar basis functions Jacobs, Finkelstein, & Salesin SIGGRAPH 95

Wavelets Descriptor 1: Given an NxNxN grid, generate an NxNxN array of the wavelet coefficients for the standard Haar basis functions Descriptor 2: Truncate: Find the m largest coefficients and set all others equal to zero Quantize: Set the non-zero coefficients to +1 or –1 depending on their sign Jacobs, Finkelstein, & Salesin SIGGRAPH 95

Jackie Chan Example Original Image (256x256)

Truncated And Quantized to 5000

Truncated And Quantized to 1000

Truncated And Quantized to 500

Truncated 100

Truncated 50

Truncated 10

Torus Example

Torus Truncated to 5000

Torus Truncated to 1000

Torus Truncated to 500

Torus Truncated to 100

Torus Truncated to 50

Wavelets Distance Function 1: The query metric is defined by: where A[i,j,k] and B[i,j,k] are the truncated and quantized coefficients and w i,j,k are weights, fine tuned to the database. Jacobs, Finkelstein, & Salesin SIGGRAPH 95

Wavelets Distance Function 2: The query metric can be approximated by: to enable efficient indexing and search. Jacobs, Finkelstein, & Salesin SIGGRAPH 95

Wavelets Properties Insensitive to noise Insensitive to topology Robust to degeneracies Quick to compute Efficient to match Concise to store Discriminating? XInvariant to transforms Jacobs, Finkelstein, & Salesin SIGGRAPH 95

Moments Define shape by moments of inertia:

Moments Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

Moments Retrieval Results Recall Precision 1 Voxels Moments [Elad et al.] Random

Moments Retrieval Results Recall Precision 1 Voxels Moments [Elad et al.] Random

Moments Properties Insensitive to topology Robust to degeneracies Quick to compute Efficient to match Concise to store XInsensitive to noise XInvariant to transforms XDiscriminating

Extended Gaussian Image Define shape with histogram of normal directions Invertible for convex objects Spherical function 3D Model EGI

EGI Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

EGI Retrieval Results Recall Precision 1 Voxels Moments [Elad et al.] EGI [Horn 84] Random

Extended Gaussian Images Properties Insensitive to topology Quick to compute Efficient to match Concise to store XInsensitve to noise XRobust to degeneracies XInvariant to transforms XDiscriminating

Other Rotation-Dependent Descriptors Spherical Extent Functions (Vranic & Saupe, 2000) Shape Histograms (sectors) (Ankherst, 1999)

Shape Descriptors II Thomas Funkhouser CS597D, Fall 2003 Princeton University Thomas Funkhouser CS597D, Fall 2003 Princeton University

Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors

Statistical Shape Descriptors Alignment-dependent Voxels Wavelets Moments Extended Gaussian Image Spherical Extent Function Spherical Attribute Image Alignment-independent Shape histograms Harmonic descriptor Shape distributions

Statistical Shape Descriptors Alignment-dependent Voxels Wavelets Moments Extended Gaussian Image Spherical Extent Function Spherical Attribute Image Alignment-independent Shape histograms Harmonic descriptor Shape distributions

Alignment Translation (Center of Mass) Scale (Radial Deviation)

Alignment Rotation (PCA) Principal axes are eigenvectors associated with largest eigenvalues of 2nd order moments covariance matrix PCA Computation Principal Axis Alignment

Rotation (PCA) Principal axes are eigenvectors associated with largest eigenvalues of 2nd order moments covariance matrix Not very robust!

Alignment Mirror PCA does not give directions for principal axes Need heuristics to determine positive axes!

Alignment-Independent Descriptors Observation: it is difficult to normalize for differences in rotation and mirroring Motivation: build a shape descriptor that is invariant to rotations and mirrors and as discriminating as possible Three mugs aligned automatically with PCA

Shape Histograms Shape descriptor stores histogram of how much surface resides at different radii from center of mass Image courtesy of Ankerst et al, 1999 Shape Histograms (shells) (Ankherst, 1999) Radius

Shape Histograms Shape descriptor stores histogram of how much surface resides at different radii from center of mass Image courtesy of Misha Kazhdan Shape Descriptor 3D Model Spherical Decomposition

Shape Histogram Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

Shape Histogram Retrieval Results Precision-recall curves (mean for all queries) Recall Precision Shape Histogram [Ankerst et al.] EGI [Horn] Moments [Elad et al.] Random 1

Shape Histograms Properties Insensitive to noise Insensitive to topology Robust to degeneracies Quick to compute Efficient to match Concise to store Invariant to rotations Discriminating?

Harmonic Shape Descriptor Key idea: Decompose each sphere into irreducible set of rotation independent components Store “how much” of the model resides in each component 3D Model Shape Descriptor Harmonic Decompositions

Step 1: Normalization Normalize for translation and scale 3D Model

Step 2: Voxelization Rasterize polygon surfaces into 3D voxel grid 3D Voxel Grid

Step 3: Spherical Decomposition Intersect with concentric spheres Spherical Functions

Step 4: Frequency Decomposition Represent each spherical function as a sum of harmonic frequencies (orders) Spherical Functions

Represent each spherical function as a sum of harmonic frequencies (orders) Step 4: Frequency Decomposition Spherical Function Spherical Functions

Represent each spherical function as a sum of harmonic frequencies (orders) Step 4: Frequency Decomposition +++= … Spherical Function Harmonic Decomposition

Represent each spherical function as a sum of harmonic frequencies (orders) Step 4: Frequency Decomposition = Constant 1 st Order2 nd Order = … … Spherical Function

Represent each spherical function as a sum of harmonic frequencies (orders) Step 4: Frequency Decomposition = Frequency Decomposition = … … Spherical Function Amplitudes are invariant to rotation

Step 5: Amplitude Computation Store “how much” (L 2 -norm) of the shape resides in each harmonic frequency of each sphere Frequency Radius Harmonic Shape Descriptor

Matching Harmonic Descriptors Define similarity as L 2 -distance between descriptors Enables nearest neighbor indexing and fast search Provides lower bound for L 2 -distance between models, = Sim

Harmonic Shape Descriptor Properties  Concise to store? Quick to compute? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Invariant to transforms? Efficient to match? Discriminating? Frequency Radius 2048 bytes per model (16 frequencies x 32 radii x 4 bytes)

Harmonic Shape Descriptor Properties Concise to store  Quick to compute? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Invariant to transforms? Efficient to match? Discriminating? 1.6 seconds (on average) Polygons Voxels Spherical Decomposition Frequency Decomposition Harmonic Shape Descriptor frequency radius

Harmonic Shape Descriptor Properties Concise to store  Quick to compute? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Invariant to transforms? Efficient to match? Discriminating? 1.6 seconds (on average) Polygons Voxels Spherical Decomposition Frequency Decomposition Harmonic Shape Descriptor frequency radius

Harmonic Shape Descriptor Properties Concise to store Quick to compute  Insensitive to noise  Insensitive to topology  Robust to degeneracies Invariant to transforms? Efficient to match? Discriminating? Rasterize polygon surfaces (no solid reconstruction)

Harmonic Shape Descriptor Properties Concise to store Quick to compute Insensitive to noise Insensitive to topology Robust to degeneracies  Invariant to transforms Efficient to match? Discriminating? Rotation Mirror Translation (w/ normalization) Scale (w/ normalization) {

Harmonic Shape Descriptor Properties Concise to store Quick to compute Insensitive to noise Insensitive to topology Robust to degeneracies Invariant to transforms  Efficient to match? Discriminating? Indexed Not Indexed 0.23 seconds to search 17,500 models

Harmonic Shape Descriptor Properties Concise to store Quick to compute Insensitive to noise Insensitive to topology Robust to degeneracies Invariant to transforms Efficient to match?  Discriminating?

Harmonic Matching Results Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

Harmonic Retrieval Results Precision-recall curves (mean for all queries) Recall Precision Harmonic Shape Descriptor Shape Histogram [Ankerst et al.] EGI [Horn] Moments [Elad et al.] Random 1

Statistical Shape Descriptors Alignment-dependent Voxels Wavelets Moments Extended Gaussian Image Spherical Extent Function Spherical Attribute Image Alignment-independent Shape histograms Harmonic descriptor  Shape distributions

Shape Distributions Motivation: general approach to finding a common parameterization for matching 3D SurfaceAudio 2D Contour3D Volume

Shape Distributions Key idea: map 3D surfaces to common parameterization by randomly sampling shape function 3D ModelsD2 Shape Distributions Randomly sample shape function Similarity Measure Distance Probability

Which Shape Function? Implementation: simple shape functions based on angles, distances, areas, and volumes A3 (angle) D1 (distance) [Ankerst 99] D2 (distance) D3 (area)   D4 (volume)

D2 Shape Distribution Properties Concise to store? Quick to compute? Invariant to transforms? Efficient to match? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Discriminating?

D2 Shape Distribution Properties  Concise to store?  Quick to compute? Invariant to transforms? Efficient to match? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Discriminating? 512 bytes (64 values) 0.5 seconds (10 6 samples) Distance Probability Skateboard

D2 Shape Distribution Properties Concise to store Quick to compute  Invariant to transforms? Efficient to match? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Discriminating? Translation Rotation Mirror { Normalized Means Scale (w/ normalization) Skateboard Porsche

Distance Probability Skateboard D2 Shape Distribution Properties Concise to store Quick to compute Invariant to transforms  Efficient to match? Insensitive to noise? Insensitive to topology? Robust to degeneracies? Discriminating? Porsche

D2 Shape Distribution Properties Concise to store Quick to compute Invariant to transforms Efficient to match  Insensitive to noise?  Insensitive to topology?  Robust to degeneracies? Discriminating? 1% Noise

D2 Shape Distribution Properties Concise to store Quick to compute Invariant to transforms Efficient to match Insensitive to noise Insensitive to topology Robust to degeneracies  Discriminating?

D2 Shape Distribution Results Question How discriminating are D2 shape distributions? Test database 133 polygonal models 25 classes 4 Mugs 6 Cars 3 Boats

D2 Shape Distribution Results D2 distributions are different across classes D2 shape distributions for 15 classes of objects

D2 Shape Distribution Results D2 distributions for 5 tanks (gray) and 6 cars (black) Distance Probability

D2 Shape Distribution Results Similarity Matrix Darkness represents similarity Blocks Tanks, cars Airplanes Humans Helicopters

D2 Retrieval Experiment Test database is Viewpoint household collection 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

D2 Retrieval Results Precision-recall curves (mean for all queries) Recall Precision Harmonic Shape Descriptor D2 Shape Distribution [Osada et al.] Shape Histogram [Ankerst et al.] EGI [Horn] Moments [Elad et al.] Random 1

Shape Distributions Next steps: Better shape functions Better comparsion methods Analysis apps

D2 Shape Distribution Results D2 shape distributions for 15 classes of objects Line Segment Recognizing gross shapes with D2 distributions

D2 Shape Distribution Results Recognizing gross shapes with D2 distributions D2 shape distributions for 15 classes of objects Circle

D2 Shape Distribution Results Recognizing gross shapes with D2 distributions D2 shape distributions for 15 classes of objects Cylinder

D2 Shape Distribution Results Recognizing gross shapes with D2 distributions D2 shape distributions for 15 classes of objects Sphere

D2 Shape Distribution Results Recognizing gross shapes with D2 distributions D2 shape distributions for 15 classes of objects Two Spheres

Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,... Point descriptors

Taxonomy of Shape Descriptors Structural representations Skeletons Part-based methods Feature-based methods Statistical representations Voxels, moments, wavelets, … Attributes, histograms,...  Point descriptors Next Time!