1. Methods for 3D Shape Matching and Retrieval
Marcin Novotni & Reinhard Klein
University of Bonn
Computer Graphics Group

2. Our Aim #1 … , , , Given an example: Find the most similar object(s)
in a database … , , ,
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

3. Motivation Lots of 3D archives: Search engines for data: WWW
Proprietary databases
...
Search engines for data:
Text, 2D images, music (MIDI), …
Emerging since 1998 for 3D
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

4. Our Aim #2 Direct matching Alignment Establishing correspondences
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

5. Motivation Partial matching/retrieval Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

6. Motivation Partial matching/retrieval Statistical shape analysis
Morphing
Texture transfer
Registration
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

7. General Problem Abstract representation facilitating:
identification of salient features of 3D objects
description of features
comparison (matching)
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

8. Overview Matching for 3D Shape Retrieval Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

9. Matching for 3D Shape Retrieval
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

10. → D( ) D : General Problem We need a Descriptor
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

11. d( , ) d( , ) D( ) D( ) : = General Problem We need a Distance Measure
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

12. d( , ) d( , ) d( , ) ≤ ≤ General Problem We need a Distance Measure :
Close to (application driven) notion of resemblance
Computationally cheap and robust
d( , )
d( , )
d( , ) ≤ ≤
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

13. D( ) ≡ x1 xn 3D Zernike Descriptors Feature vectors
Xi : 3D Zernike Descriptors
[Canterakis ’99, Novotni & Klein ’03, ’04]
Distance Measure: Euclidean Distance
D( ) ≡ x1 xn
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

14. 3D Zernike Descriptors Retrieval performance [Novotni & Klein ‘03 ’04]
Slightly better than [Funkhouser et al. ’02]
Object class dependent performance!
Class dependent coefficient importance!
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

15. 3D Zernike Descriptors Faces Chairs Airplanes Importance
Coeff No. (Frequency)
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

16. 3D Zernike Descriptors Relevance feedback: Learning Machines:
User selects relevant / irrelevant items
Distance measure is tuned
Learning Machines:
SVM (Support vector machines) [Vapnik ‘95]
One class SVM [Schölkopf et al. ’99]
(K)BDA ((Kernel) Biased Discriminant Analysis) [Zhou et al. ‘01]
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

17. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

18. Geometric Similarity Estimation
Idea [Novotni & Klein 2001]:
Definition of „geometric“ similarity in terms of a geometric distance
Intuitive, simple, robust.
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

19. Geometric Similarity Estimation
Database objects
example
Normalized volumetric error
0.00
6.78
8.85
30.29
38.09
67.53
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

20. Geometric Similarity Estimation
Classification by user set threshold
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

21. Geometric Similarity Estimation
Measures deformation magnitude
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

22. Correspondence Matching?
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

23. Correspondence Matching?
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

24. Correspondence Matching?
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

25. Correspondence Matching?
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

26. Correspondence Matching
Ideally: dense mapping ?
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

27. Correspondence Matching
Ideally: dense mapping
Deformation by mapping semantics
[D’Arcy Thompson 1917: On Growth and Form ]
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

28. Correspondence Matching
Ideally: dense mapping
Easier: mapping salient points
Curvature extremes
Corners (Harris points in 2D)
Etc…
Scale space extremes
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

29. Correspondence Matching
Ideally: dense mapping
Easier: mapping salient points
Curvature extremes
Corners (Harris points in 2D)
Etc…
Scale space extremes
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

30. Correspondence Matching
Scale Space extremes [Lindeberg ‘94]
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

31. Correspondence Matching
We have:
Salient points
Spatial position
Size of local blobs
How to match???
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

32. Correspondence Matching
Criteria for correspondences:
Similar
Local geometries
Constellations of points
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

33. Correspondence Matching
Criteria for correspondences:
Similar
Local geometries
Constellations of points
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

34. Correspondence Matching
Local description
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

35. Correspondence Matching
Local description
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

36. Correspondence Matching
Assumption:
Similar local descriptors
Similar local geometries
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

37. Correspondence Matching
Criteria for correspondences:
Similar
Local geometries
Constellations of points
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

38. Correspondence Matching
Similar constellations of points
Smooth mappings leave constellations consistent
Idea
Constellations are consistent if mapping is smooth
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

39. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

40. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

41. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

42. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

43. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

44. Correspondence Matching
Similar constellations of points
Idea:
Constellations are consistent if mapping is smooth
Thin Plate Spline interpolation [Brookstein ’89]
 minimize:
Total curvature
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

45. Correspondence Matching
 minimize:
Minimizer (Thin Plate Spline interpolator):
Affine part
Nonlinear deformation
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

46. Correspondence Matching
 minimize:
Minimizer (Thin Plate Spline interpolator):
2D Thin Plate Spline
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

47. Correspondence Matching
 minimize:
Minimizer (Thin Plate Spline interpolator):
Can be computed by a (N+4)x(N+4) matrix inversion
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

48. Correspondence Matching
Find (sub)sets of correspondences:
Small local descriptor distances
Small deformation energy
Hierarchical pruning and clustering
Using:
Local descriptors
Geometrical constellation consistency
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

49. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

50. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

51. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

52. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

53. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

54. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

55. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

56. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

57. Correspondence Matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

58. Correspondence Matching
New avenues:
Local Descriptions for retrieval
Online Learning for local descriptions
Dense matching from salient points
Etc.
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

59. Danke, DFG! Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

60. 3D Zernike Descriptors Basis functions in the unit sphere:
SH on the sphere
We use SH on the sphere to retain rotation invariance…
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

61. 3D Zernike Descriptors Basis functions in the unit sphere:
SH on the sphere
Function of the radius
… and a radial function other than sampling. Since the radial function depends only on the radius, the rotation invariance is preserved.
Rotation invariant!
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

62. Object function, e.g. voxel grid
3D Zernike Descriptors
Basis functions in the unit sphere:
3D Zernike Moments [Canterakis ‘99]:
… and a radial function other than sampling. Since the radial function depends only on the radius, the rotation invariance is preserved.
Object function, e.g. voxel grid
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

63. 3D Zernike Descriptors 3D Zernike Descriptors:
Amplitudes of the Zernike decomposition
Rotation invariant
And the zernike descriptors are defined similarly to the SH amplitudes: we gather the zernike moments with upper index from –l to l for a given lower indices n and l. Due to the properties of SH, the norms of these vectors will be rotation invariant.
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

64. 3D Zernike Descriptors Basis functions in the unit sphere:
SH on the sphere
We use SH on the sphere to retain rotation invariance…
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

65. 3D Zernike Descriptors Basis functions in the unit sphere:
SH on the sphere
Function of the radius
… and a radial function other than sampling. Since the radial function depends only on the radius, the rotation invariance is preserved.
Rotation invariant!
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

66. Object function, e.g. voxel grid
3D Zernike Descriptors
Basis functions in the unit sphere:
3D Zernike Moments [Canterakis ‘99]:
… and a radial function other than sampling. Since the radial function depends only on the radius, the rotation invariance is preserved.
Object function, e.g. voxel grid
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

67. 3D Zernike Descriptors 3D Zernike Descriptors:
Amplitudes of the Zernike decomposition
Rotation invariant
And the zernike descriptors are defined similarly to the SH amplitudes: we gather the zernike moments with upper index from –l to l for a given lower indices n and l. Due to the properties of SH, the norms of these vectors will be rotation invariant.
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

68. 3D Zernike Descriptors For N=22 : 155 floats as search key
Timings (1.8 GHz Pentium):
Voxelization: 0.3 – 10.0 sec / object
Computation: 0.2 sec / object
Retrieval (1814 objects): 0.3 sec
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

69. 3D Zernike Descriptors Retrieval performance [Novotni & Klein ’04]
Slightly better than [Funkhouser et al. ’02]
Object class dependent performance!
Class dependent coefficient importance!
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

70. 3D Zernike Descriptors Faces Chairs Airplanes Importance Coeff No.
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

71. 3D Zernike Descriptors 3D Zernike functions [Canterakis ‘99]
are polynomials such that are orthonormal within the unit ball
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

72. 3D Zernike Descriptors 3D Zernike functions [Canterakis ‘99]
are polynomials such that are orthonormal within the unit ball
3D Zernike Moments:
The 3D Zernike moments are defined as inner products of the object function with the elements of the basis.
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

73. 3D Zernike Descriptors 3D Zernike Descriptors:
Amplitudes of the Zernike decomposition
Rotation invariant
And the zernike descriptors are defined similarly to the SH amplitudes: we gather the zernike moments with upper index from –l to l for a given lower indices n and l. Due to the properties of SH, the norms of these vectors will be rotation invariant.
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

74. 3D Zernike Descriptors For N=20 : 121 floats as search key
Timings (1.8 GHz Pentium):
Voxelization: 0.3 – 10.0 sec / object
Computation: 0.2 sec / object
Retrieval (1814 objects): 0.3 sec
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

75. 3D Zernike Descriptors Retrieval performance [Novotni & Klein ’04]
Slightly better than [Funkhouser et al. ’02]
Object class dependent performance!
Class dependent coefficient importance!
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

76. Correspondence Matching
Matching should be:
Independent of topology
Robust
Suitable for partial matching
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

77. Correspondence Matching
Local description
Local shape histograms
 Not rotation invariant
Rotation invariance
 Amplitudes of the Fourier Transform
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

78. 3D Zernike Descriptors 155 floats as search key
Retrieval (1814 objects): 0.3 sec
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

79. Correspondence Matching
Stuff to remember:
Salient points simplify the problem
Smooth mapping iff consistent constellations
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

80. Correspondence Matching
Stuff to remember:
Salient points simplify the problem
Volumetric
On the surface
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

81. Correspondence Matching
Stuff to remember:
Salient points simplify the problem
Smooth mapping iff consistent constellations
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

82. Correspondence Matching
New avenues:
Local Descriptions for retrieval
Retrieval by part selection & recognition
Retrieval from large scenes
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

83. Correspondence Matching
New avenues:
Local Descriptions for retrieval
Online Learning for local descriptions
Adopting pattern recognition methods
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

84. Correspondence Matching
New avenues:
Local Descriptions for retrieval
Online Learning for local descriptions
Dense matching from salient points
Morphing, registration, object statistics
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

85. Our Aim #2 Direct matching Alignment Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

86. Correspondence Matching
Scale space extremes [Lindeberg ‘94]
Blob detection by
localizing extremes of Laplacian …
… in scale and space
Size of the blob
Position of the blob
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

87. Correspondence Matching
Maxima of Laplacian over scales
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group

88. Correspondence Matching
Spatial maxima
Marcin Novotni  Reinhard Klein
University of Bonn  Computer Graphics Group