Methods for 3D Shape Matching and Retrieval Marcin Novotni & Reinhard Klein University of Bonn Computer Graphics Group

Our Aim #1 … , , , Given an example: Find the most similar object(s) in a database … , , , Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Motivation Lots of 3D archives: Search engines for data: WWW Proprietary databases ... Search engines for data: Text, 2D images, music (MIDI), … Emerging since 1998 for 3D Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Our Aim #2 Direct matching Alignment Establishing correspondences Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Motivation Partial matching/retrieval Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Motivation Partial matching/retrieval Statistical shape analysis Morphing Texture transfer Registration Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

General Problem Abstract representation facilitating: identification of salient features of 3D objects description of features comparison (matching) Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Overview Matching for 3D Shape Retrieval Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Matching for 3D Shape Retrieval Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

→ D( ) D : General Problem We need a Descriptor Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

d( , ) d( , ) D( ) D( ) : = General Problem We need a Distance Measure Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

d( , ) d( , ) d( , ) ≤ ≤ General Problem We need a Distance Measure : Close to (application driven) notion of resemblance Computationally cheap and robust d( , ) d( , ) d( , ) ≤ ≤ Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

D( ) ≡ x1 xn 3D Zernike Descriptors Feature vectors Xi : 3D Zernike Descriptors [Canterakis ’99, Novotni & Klein ’03, ’04] Distance Measure: Euclidean Distance D( ) ≡ x1 xn Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Retrieval performance [Novotni & Klein ‘03 ’04] Slightly better than [Funkhouser et al. ’02] Object class dependent performance! Class dependent coefficient importance! Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Faces Chairs Airplanes Importance Coeff No. (Frequency) Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Relevance feedback: Learning Machines: User selects relevant / irrelevant items Distance measure is tuned Learning Machines: SVM (Support vector machines) [Vapnik ‘95] One class SVM [Schölkopf et al. ’99] (K)BDA ((Kernel) Biased Discriminant Analysis) [Zhou et al. ‘01] Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Geometric Similarity Estimation Idea [Novotni & Klein 2001]: Definition of „geometric“ similarity in terms of a geometric distance Intuitive, simple, robust. Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Geometric Similarity Estimation Database objects example Normalized volumetric error 0.00 6.78 8.85 30.29 38.09 67.53 Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Geometric Similarity Estimation Classification by user set threshold Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Geometric Similarity Estimation Measures deformation magnitude Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching ? Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching ? Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching ? Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching ? Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Ideally: dense mapping ? Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Ideally: dense mapping Deformation by mapping semantics [D’Arcy Thompson 1917: On Growth and Form ] Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Ideally: dense mapping Easier: mapping salient points Curvature extremes Corners (Harris points in 2D) Etc… Scale space extremes Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Ideally: dense mapping Easier: mapping salient points Curvature extremes Corners (Harris points in 2D) Etc… Scale space extremes Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Scale Space extremes [Lindeberg ‘94] Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching We have: Salient points Spatial position Size of local blobs How to match??? Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Criteria for correspondences: Similar Local geometries Constellations of points Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Criteria for correspondences: Similar Local geometries Constellations of points Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Local description Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Local description Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Assumption: Similar local descriptors Similar local geometries Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Criteria for correspondences: Similar Local geometries Constellations of points Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Similar constellations of points Smooth mappings leave constellations consistent Idea Constellations are consistent if mapping is smooth Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Similar constellations of points Idea: Constellations are consistent if mapping is smooth Thin Plate Spline interpolation [Brookstein ’89]  minimize: Total curvature Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching  minimize: Minimizer (Thin Plate Spline interpolator): Affine part Nonlinear deformation Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching  minimize: Minimizer (Thin Plate Spline interpolator): 2D Thin Plate Spline Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching  minimize: Minimizer (Thin Plate Spline interpolator): Can be computed by a (N+4)x(N+4) matrix inversion Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Find (sub)sets of correspondences: Small local descriptor distances Small deformation energy Hierarchical pruning and clustering Using: Local descriptors Geometrical constellation consistency Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching New avenues: Local Descriptions for retrieval Online Learning for local descriptions Dense matching from salient points Etc. Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Danke, DFG! Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Basis functions in the unit sphere: SH on the sphere We use SH on the sphere to retain rotation invariance… Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Basis functions in the unit sphere: SH on the sphere Function of the radius … and a radial function other than sampling. Since the radial function depends only on the radius, the rotation invariance is preserved. Rotation invariant! Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Object function, e.g. voxel grid 3D Zernike Descriptors Basis functions in the unit sphere: 3D Zernike Moments [Canterakis ‘99]: … and a radial function other than sampling. Since the radial function depends only on the radius, the rotation invariance is preserved. Object function, e.g. voxel grid Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors 3D Zernike Descriptors: Amplitudes of the Zernike decomposition Rotation invariant And the zernike descriptors are defined similarly to the SH amplitudes: we gather the zernike moments with upper index from –l to l for a given lower indices n and l. Due to the properties of SH, the norms of these vectors will be rotation invariant. Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Basis functions in the unit sphere: SH on the sphere We use SH on the sphere to retain rotation invariance… Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Basis functions in the unit sphere: SH on the sphere Function of the radius … and a radial function other than sampling. Since the radial function depends only on the radius, the rotation invariance is preserved. Rotation invariant! Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Object function, e.g. voxel grid 3D Zernike Descriptors Basis functions in the unit sphere: 3D Zernike Moments [Canterakis ‘99]: … and a radial function other than sampling. Since the radial function depends only on the radius, the rotation invariance is preserved. Object function, e.g. voxel grid Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors 3D Zernike Descriptors: Amplitudes of the Zernike decomposition Rotation invariant And the zernike descriptors are defined similarly to the SH amplitudes: we gather the zernike moments with upper index from –l to l for a given lower indices n and l. Due to the properties of SH, the norms of these vectors will be rotation invariant. Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors For N=22 : 155 floats as search key Timings (1.8 GHz Pentium): Voxelization: 0.3 – 10.0 sec / object Computation: 0.2 sec / object Retrieval (1814 objects): 0.3 sec Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Retrieval performance [Novotni & Klein ’04] Slightly better than [Funkhouser et al. ’02] Object class dependent performance! Class dependent coefficient importance! Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Faces Chairs Airplanes Importance Coeff No. Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors 3D Zernike functions [Canterakis ‘99] are polynomials such that are orthonormal within the unit ball Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors 3D Zernike functions [Canterakis ‘99] are polynomials such that are orthonormal within the unit ball 3D Zernike Moments: The 3D Zernike moments are defined as inner products of the object function with the elements of the basis. Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors 3D Zernike Descriptors: Amplitudes of the Zernike decomposition Rotation invariant And the zernike descriptors are defined similarly to the SH amplitudes: we gather the zernike moments with upper index from –l to l for a given lower indices n and l. Due to the properties of SH, the norms of these vectors will be rotation invariant. Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors For N=20 : 121 floats as search key Timings (1.8 GHz Pentium): Voxelization: 0.3 – 10.0 sec / object Computation: 0.2 sec / object Retrieval (1814 objects): 0.3 sec Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors Retrieval performance [Novotni & Klein ’04] Slightly better than [Funkhouser et al. ’02] Object class dependent performance! Class dependent coefficient importance! Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Matching should be: Independent of topology Robust Suitable for partial matching Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Local description Local shape histograms  Not rotation invariant Rotation invariance  Amplitudes of the Fourier Transform Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

3D Zernike Descriptors 155 floats as search key Retrieval (1814 objects): 0.3 sec Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Stuff to remember: Salient points simplify the problem Smooth mapping iff consistent constellations Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Stuff to remember: Salient points simplify the problem Volumetric On the surface Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Stuff to remember: Salient points simplify the problem Smooth mapping iff consistent constellations Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching New avenues: Local Descriptions for retrieval Retrieval by part selection & recognition Retrieval from large scenes Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching New avenues: Local Descriptions for retrieval Online Learning for local descriptions Adopting pattern recognition methods Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching New avenues: Local Descriptions for retrieval Online Learning for local descriptions Dense matching from salient points Morphing, registration, object statistics Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Our Aim #2 Direct matching Alignment Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Scale space extremes [Lindeberg ‘94] Blob detection by localizing extremes of Laplacian … … in scale and space Size of the blob Position of the blob Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Maxima of Laplacian over scales Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group

Correspondence Matching Spatial maxima Marcin Novotni  Reinhard Klein University of Bonn  Computer Graphics Group