Harmonic 3D Shape Matching Michael Kazhdan Thomas Funkhouser Princeton University Michael Kazhdan Thomas Funkhouser Princeton University

Motivation Large databases of 3D models Mechanical CAD (National Design Repository) Molecular Biology (Audrey Sanderson) Computer Graphics (Princeton 3D Search Engine)

Goal Find 3D models with similar shapes 3D ModelShape Descriptor Model Database Nearest Neighbor

Research Challenge Need shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match Need shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match 3D ModelShape Descriptor Model Database Nearest Neighbor

Research Challenge Finding a 3D shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match Finding a 3D shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match 3D ModelShape Descriptor Nearest Neighbor Model Database

Research Challenge Finding a 3D shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match Finding a 3D shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match Model Database Nearest Neighbor Shape Descriptor 3D Model

Research Challenge Finding a 3D shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match Finding a 3D shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match 3D Model Nearest Neighbor Shape Descriptor Model Database

Research Challenge Finding a 3D shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match Finding a 3D shape descriptor that is: DiscriminatingDiscriminating Concise to storeConcise to store Quick to computeQuick to compute Efficient to matchEfficient to match 3D Model Nearest Neighbor Shape Descriptor Model Database Many possible alignments

3D Model Matching Approaches Search over all possible alignments Too slow for large databaseToo slow for large database Search over all possible alignments Too slow for large databaseToo slow for large database min

3D Model Matching Approaches Search over all possible alignments Too slow for large databaseToo slow for large database Normalize alignment (e.g., with moments) OK for translation and scale, not for rotationOK for translation and scale, not for rotation Search over all possible alignments Too slow for large databaseToo slow for large database Normalize alignment (e.g., with moments) OK for translation and scale, not for rotationOK for translation and scale, not for rotation PCA Aligned Models

3D Model Matching Approaches Search over all possible alignments Too slow for large databaseToo slow for large database Normalize alignment (e.g., with moments) OK for translation and scale, not for rotationOK for translation and scale, not for rotation Build alignment invariance into descriptor Previous methods not very discriminatingPrevious methods not very discriminating Search over all possible alignments Too slow for large databaseToo slow for large database Normalize alignment (e.g., with moments) OK for translation and scale, not for rotationOK for translation and scale, not for rotation Build alignment invariance into descriptor Previous methods not very discriminatingPrevious methods not very discriminating Shape Histograms [Ankerst et al., 1999] Shape Histograms [Ankerst et al., 1999]

Outline IntroductionApproachImplementation Experimental Results Conclusion and Future Work IntroductionApproachImplementation Experimental Results Conclusion and Future Work

Our Approach Harmonic 3D shape descriptor Decompose 3D shapes into irreducible set of rotation independent componentsDecompose 3D shapes into irreducible set of rotation independent components Store “how much” of the model resides in each componentStore “how much” of the model resides in each component Harmonic 3D shape descriptor Decompose 3D shapes into irreducible set of rotation independent componentsDecompose 3D shapes into irreducible set of rotation independent components Store “how much” of the model resides in each componentStore “how much” of the model resides in each component 3D Model Shape Descriptor Rotation Independent Components

Our Approach Harmonic 3D shape descriptor Decompose 3D shapes into irreducible set of rotation independent componentsDecompose 3D shapes into irreducible set of rotation independent components Store “how much” of the model resides in each componentStore “how much” of the model resides in each component Harmonic 3D shape descriptor Decompose 3D shapes into irreducible set of rotation independent componentsDecompose 3D shapes into irreducible set of rotation independent components Store “how much” of the model resides in each componentStore “how much” of the model resides in each component Concentric Spheres 3D Model Shape Descriptor Rotation Independent Components

Our Approach Harmonic 3D shape descriptor Decompose 3D shapes into irreducible set of rotation independent componentsDecompose 3D shapes into irreducible set of rotation independent components Store “how much” of the model resides in each componentStore “how much” of the model resides in each component Harmonic 3D shape descriptor Decompose 3D shapes into irreducible set of rotation independent componentsDecompose 3D shapes into irreducible set of rotation independent components Store “how much” of the model resides in each componentStore “how much” of the model resides in each component Frequency Decomposition 3D Model Shape Descriptor Rotation Independent Components

Our Approach Harmonic 3D shape descriptor Decompose 3D shapes into irreducible set of rotation independent componentsDecompose 3D shapes into irreducible set of rotation independent components Store “how much” of the model resides in each componentStore “how much” of the model resides in each component Harmonic 3D shape descriptor Decompose 3D shapes into irreducible set of rotation independent componentsDecompose 3D shapes into irreducible set of rotation independent components Store “how much” of the model resides in each componentStore “how much” of the model resides in each component 3D Model Shape Descriptor Rotation Independent Components Amplitudes

Outline IntroductionApproachImplementation Experimental Results Conclusion and Future Work IntroductionApproachImplementation Experimental Results Conclusion and Future Work

Voxelization Convert polygonal model to 3D voxel grid Rasterize surfaces (no solid reconstruction)Rasterize surfaces (no solid reconstruction) Normalize for translation and scaleNormalize for translation and scale Convert polygonal model to 3D voxel grid Rasterize surfaces (no solid reconstruction)Rasterize surfaces (no solid reconstruction) Normalize for translation and scaleNormalize for translation and scale 3D Model3D Voxel Grid

Spherical Decomposition Intersect with concentric spheres

Frequency Decomposition Represent each spherical function as a sum of different frequencies Spherical Functions Frequency Components

Fourier Analysis Circular Function

Fourier Analysis +++=+ … Cosine/Sine Decomposition Circular Function

Fourier Analysis = +++ Constant =+ … Frequency Decomposition Circular Function

Fourier Analysis = Constant1 st Order =+ … + Frequency Decomposition Circular Function

Fourier Analysis = Constant1 st Order2 nd Order =+ … + Frequency Decomposition Circular Function

Fourier Analysis = Constant1 st Order2 nd Order3 rd Order =+ … + … + Frequency Decomposition Circular Function

++++ … + Fourier Analysis = +++ Constant1 st Order2 nd Order3 rd Order + … Frequency Decomposition = Amplitudes invariant to rotation Circular Function

Harmonic Analysis Spherical Function

Harmonic Analysis +++=+ … Spherical Function Harmonic Decomposition

Harmonic Analysis = Constant1 st Order2 nd Order3 rd Order =+ … + … Spherical Function

Building Shape Descriptor Store “how much” (L 2 -norm) of the shape resides in each frequency of each sphere Frequency Decomposition Amplitudes Harmonic Shape Descriptor

Matching Model similarity defined as L 2 -distance between their descriptors Bounds proximity of voxel grids over all rotationsBounds proximity of voxel grids over all rotations Model similarity defined as L 2 -distance between their descriptors Bounds proximity of voxel grids over all rotationsBounds proximity of voxel grids over all rotations, = Sim

Outline IntroductionApproachImplementation Experimental Results Conclusion and Future Work IntroductionApproachImplementation Experimental Results Conclusion and Future Work

Princeton 3D Search Engine Query

Retrieval Experiment Viewpoint “household” database 1,890 models, 85 classes 153 dining chairs25 livingroom chairs16 beds12 dining tables 8 chests28 bottles39 vases36 end tables

Retrieval Results Precision-recall curve (mean for all queries) Recall Precision Random 3D Harmonics Query

Retrieval Results Precision versus recall (mean for all queries) Recall Precision 3D Harmonics (Our Method) D2 Shape Distributions [Osada et al., 2001] Shape Histograms [Ankerst, 1999] EGI [Horn, 1984] Moments [Elad et al., 2001] Random 1

Retrieval Results Precision versus recall (mean for all queries) Recall Precision 3D Harmonics (Our Method) D2 Shape Distributions [Osada et al., 2001] Shape Histograms [Ankerst, 1999] EGI [Horn, 1984] Moments [Elad et al., 2001] Random 1

Retrieval Results Precision versus recall (mean for all queries) Recall Precision 3D Harmonics (Our Method) D2 Shape Distributions [Osada et al., 2001] Shape Histograms [Ankerst, 1999] EGI [Horn, 1984] Moments [Elad et al., 2001] Random 1

Retrieval Results Precision versus recall (mean for all queries) Recall Precision 3D Harmonics (Our Method) D2 Shape Distributions [Osada et al., 2001] Shape Histograms [Ankerst, 1999] EGI [Horn, 1984] Moments [Elad et al., 2001] Random 1

Summary and Conclusion Harmonic shape descriptor is a rotation invariant representation that is: Discriminating (46%-245% better than others tested)Discriminating (46%-245% better than others tested) Concise to store (2048 bytes)Concise to store (2048 bytes) Quick to compute (1-5 seconds)Quick to compute (1-5 seconds) Efficient to match (0.45 seconds: 20,000 model DB)Efficient to match (0.45 seconds: 20,000 model DB) Harmonic shape descriptor is a rotation invariant representation that is: Discriminating (46%-245% better than others tested)Discriminating (46%-245% better than others tested) Concise to store (2048 bytes)Concise to store (2048 bytes) Quick to compute (1-5 seconds)Quick to compute (1-5 seconds) Efficient to match (0.45 seconds: 20,000 model DB)Efficient to match (0.45 seconds: 20,000 model DB)

Future Work Extensions Partial object matching?Partial object matching? Other 3D shape functions?Other 3D shape functions? Other applications Molecular biologyMolecular biology MedicineMedicine PaleontologyPaleontology ForensicsForensicsExtensions Partial object matching?Partial object matching? Other 3D shape functions?Other 3D shape functions? Other applications Molecular biologyMolecular biology MedicineMedicine PaleontologyPaleontology ForensicsForensics Query

Thank You Funding National Science FoundationNational Science Foundation Sloan FoundationSloan FoundationPeople Bernard Chazelle, David Dobkin, David Jacobs, David Kazhdan, Allison Klein, Patrick Min, Szymon Rusinkiewicz, Peter Sarnak, Julianna TymoczkoBernard Chazelle, David Dobkin, David Jacobs, David Kazhdan, Allison Klein, Patrick Min, Szymon Rusinkiewicz, Peter Sarnak, Julianna TymoczkoFunding National Science FoundationNational Science Foundation Sloan FoundationSloan FoundationPeople Bernard Chazelle, David Dobkin, David Jacobs, David Kazhdan, Allison Klein, Patrick Min, Szymon Rusinkiewicz, Peter Sarnak, Julianna TymoczkoBernard Chazelle, David Dobkin, David Jacobs, David Kazhdan, Allison Klein, Patrick Min, Szymon Rusinkiewicz, Peter Sarnak, Julianna Tymoczko

Analysis The Harmonic Descriptor is not invertible Different spheres rotate independentlyDifferent spheres rotate independently Different orders rotate independentlyDifferent orders rotate independently Rotations are not transitive within an orderRotations are not transitive within an order The Harmonic Descriptor is not invertible Different spheres rotate independentlyDifferent spheres rotate independently Different orders rotate independentlyDifferent orders rotate independently Rotations are not transitive within an orderRotations are not transitive within an order

Analysis The Harmonic Descriptor is not invertible Different spheres rotate independentlyDifferent spheres rotate independently Different orders rotate independentlyDifferent orders rotate independently Rotations are not transitive within an orderRotations are not transitive within an order The Harmonic Descriptor is not invertible Different spheres rotate independentlyDifferent spheres rotate independently Different orders rotate independentlyDifferent orders rotate independently Rotations are not transitive within an orderRotations are not transitive within an order l=0 l=1 l=2 l=3

Inter-Radial Coherence Force same orders on different spheres to rotate together by setting up the rotation invariant matrix M k with: where f k,j is the k-th order component of the restriction of f to the j-th radius. (The diagonal is precisely the collection of L 2 -norms.) Force same orders on different spheres to rotate together by setting up the rotation invariant matrix M k with: where f k,j is the k-th order component of the restriction of f to the j-th radius. (The diagonal is precisely the collection of L 2 -norms.)

Polygon Rasterization Rasterize using the Euclidean Distance Transform to measure how much models miss by: Polygonal Model Voxel Model