1 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Feature-based methods and shape retrieval problems © Alexander & Michael Bronstein, © Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Advanced topics in vision Processing and Analysis of Geometric Shapes EE Technion, Spring 2010

2 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Structure Local Feature descriptors Global Metric

3 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Combining local and global structures BBK 2008; Keriven, Torstensen 2009; Dubrovina, Kimmel 2010; Wang, B 2010 Pair-wise stress (global) Point-wise stress (local) Local structure can be geometric or photometric

4 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Photometric stress Thorstenstein & Keriven 2009

5 Numerical Geometry of Non-Rigid Shapes Diffusion Geometry Heat kernels, encore Brownian motion on X starting at point x, measurable set C probability of the Brownian motion to be in C at time t Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005 Heat kernel represents transition probability

6 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Intrinsic descriptors Sun, Ovsjanikov & Guibas 2009 Multiscale local shape descriptor (Heat kernel signature) can be interpreted as probability of Brownian motion to return to the same point after time (represents “stability” of the point) Time (scale)

7 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Sun, Ovsjanikov & Guibas 2009 for small t Relation to curvature

8 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Heat kernel signature Heat kernel signatures represented in RGB space Sun, Ovsjanikov & Guibas 2009 Ovsjanikov, BB & Guibas 2009

9 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Heat kernel descriptors Invariant to isometric deformations Localized sensitivity to topological noise J. Sun, M. Ovsjanikov, L. Guibas, SGP 2009 M. Ovsjanikov, BB, L. Guibas, 2009

10 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Scale invariance Original shapeScaled by HKS= Not scale invariant!

11 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Scale-invariant heat kernel signature B, Kokkinos CVPR 2010 Log scale-space Scaling = shift and multiplicative constant in HKS log + d/d  Undo scaling Fourier transform magnitude Undo shift    =2k  /T

12 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Scale invariance B, Kokkinos 2009 Heat Kernel SignatureScale-invariant Heat Kernel Signature

13 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Bending invariance B, Kokkinos CVPR 2010 Heat Kernel SignatureScale-invariant Heat Kernel Signature

14 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Bending invariance Wang, B 2010 Geodesic+HKSDiffusion+HKSCommute+SI-HKS

15 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Topology invariance Geodesic+HKSDiffusion+HKS Wang, B 2010

16 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Scale invariance Wang, B 2010 Geodesic+HKSCommute+SI-HKS

17 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Invariance Geodesic metric RigidInelasticTopology Diffusion metric Scale Wang, B 2010 Commute-time metric Heat kernel signature (HKS) Scale-invariant HKS (SI-HKS)

18 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems

19 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Tagged shapes Shapes without metadata Man, person, humanPerson Text search Content-based search 3D warehouse

20 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems ? Content-based search problems Invariant shape retrieval Shape classification ? Semantic Variability of shape within category Geometric Variability of shape under transformation

21 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Image vs shape retrieval IlluminationViewMissing data DeformationTopologyMissing data

22 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Bags of words Notre Dame de Paris is a Gothic cathedral in the fourth quarter of Paris, France. It was the first Gothic architecture cathedral, and its construction spanned the Gothic period. construction architecture Italy France cathedral church basilica Paris Rome Gothic Roman St. Peter’s basilica is the largest church in world, located in Rome, Italy. As a work of architecture, it is regarded as the best building of its age in Italy. Notre Dame de Paris is a Gothic cathedral in the fourth quarter of Paris, France. It was the first Gothic architecture cathedral, and its construction spanned the Gothic period. St. Peter’s basilica is the largest church in world, located in Rome, Italy. As a work of architecture, it is regarded as the best building of its age in Italy.

23 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Bags of features Visual vocabulary Feature detector + descriptor Invariant to changes of the image Discriminative (tells different images apart)

24 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Advantages “Shape signature” Easy to store Easy to compare Partial similarity possible

25 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Images vs shapes ImagesShapes Many prominent featuresFew prominent features Affine transforms, illumination, occlusions, resolution Non-rigid deformations, topology, missing parts, triangulation SIFT, SURF, MSER, DAISY, …Curvature, conformal factor, local distance histograms

26 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems “ShapeGoogle” Feature descriptor Geometric words Bag of words Geometric expressions Spatially-sensitive bag of features “ ” “ ”

27 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Geometric vocabulary M. Ovsjanikov, BB, L. Guibas, 2009

28 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Bags of features Geometric vocabulary M. Ovsjanikov, BB, L. Guibas, 2009 Nearest neighbor in the descriptor space

29 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Bags of features Geometric vocabulary M. Ovsjanikov, BB, L. Guibas, 2009 Weighted distance to words in the vocabulary

30 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Bags of features Shape distance = distance between bags of features M. Ovsjanikov, BB, L. Guibas, 2009 Statistics of different geometric words over the entire shape

31 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Index in vocabulary 164 M. Ovsjanikov, BB, L. Guibas, 2009 Bags of features

32 Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval Statistical weighting Query qDatabase D syzygy in astronomy means alignment of three bodies of the solar system along a straight or nearly straight line. a planet is in syzygy with the earth and sun when it is in opposition or conjunction. the moon is in syzygy with the earth and sun when it is new or full. Sivic & Zisserman 2003 BB, Carmon & Kimmel 2009 Frequent in document = important in is or syzygy Rare in database = discriminative with a of the andwhen

33 Michael Bronstein Shape Google: geometric words and expressions for invariant shape retrieval Statistical weighting Query qDatabase D Significance of a term t Term frequencyInverse document frequency Weight bags of features by tf-idf Reduce the influence of non-important terms in dense descriptor Sivic & Zisserman 2003 BB, Carmon & Kimmel 2009

34 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Expressions In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem. In biological science, decomposition is the process of organisms to break down into simpler form of matter. Usually, decomposition occurs after death. Matrix is a science fiction movie released in Matrix refers to a simulated reality created by machines in order to subdue the human population. matrix decomposition matrix factorization science fiction canonical form In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem. In biological science, decomposition is the process of organisms to break down into simpler form of matter. Usually, decomposition occurs after death. Matrix is a science fiction movie released in Matrix refers to a simulated reality created by machines in order to subdue the human population. matrix decomposition is a the of in to by science form In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem. Matrix is a science fiction movie released in Matrix refers to a simulated reality created by machines in order to subdue the human population. M. Ovsjanikov, BB, L. Guibas, 2009

35 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Expressions In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem. matrix decomposition is a the of in to by science form In particular matrix used type a some science, decomposition form a factorization of is canonical. matrix math decomposition is in a Each problem. into of matrix decomposition matrix factorization science fiction canonical form M. Ovsjanikov, BB, L. Guibas, 2009

36 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Visual expressions “Inquisitor King”Inquisitor, King“King Inquisitor” Giuseppe Verdi, Don Carlo, Metropolitan Opera

37 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Geometric expressions M. Ovsjanikov, BB, L. Guibas, 2009 “Yellow Yellow”Yellow No total order between points (only “far” and “near”) Geometric expression = a pair of spatially close geometric words

38 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Spatially-sensitive bags of features M. Ovsjanikov, BB, L. Guibas, 2009 is the probability to find word at point and word at point Proximity between points and Distribution of pairs of geometric words Shape distance is the statistic of geometric expressions of the form

39 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems M. Ovsjanikov, BB, L. Guibas, 2009 Spatially-sensitive bags of features

40 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems SHREC 2010 dataset

41 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems SHREC 2010 dataset BB et al, 3DOR 2010

42 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems ShapeGoogle with HKS descriptor (mAP %) BB et al, 3DOR 2010

43 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems ShapeGoogle with SI-HKS descriptor (mAP %) BB et al, 3DOR 2010

44 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Scale 0.7Heat Kernel Signature ? Scale-Invariant Heat Kernel Signature Scale-invariant retrieval Kokkinos, B 2009

45 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Scale 1.3Heat Kernel Signature Scale-Invariant Heat Kernel Signature Kokkinos, B 2009 Scale-invariant retrieval

46 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Heat Kernel Signature Local scale Scale-Invariant Heat Kernel Signature Kokkinos, B 2009 Scale-invariant retrieval

47 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Structure Local Feature descriptors Global Metric

48 Michael Bronstein Diffusion geometry for shape recognition Beylkin & Niyogi 2003 Coifman, Lafon, Lee, Maggioni, Warner & Zucker 2005 Rustamov 2007 Laplacian embedding Represent the shape using finite-dimensional Laplacian eigenmap Ambiguities!

49 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Osada, Funkhouser, Chazelle & Dobkin 2002 Rustamov 2007 Global point signature (GPS) embedding Deformation- and scale-invariant No ambiguities related to eigenfunction permutations and sign No need to compare multidimensional embeddings Represent the shape using distribution of Euclidean distances in the Laplacian embedding space (=commute time distances)

50 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Diffusion distance distributions Mahmoudi & Sapiro 2009 Represent the shape using distribution of diffusion distances Deformation-invariant  How to select the scale? x 10 -3

51 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Spectral shape distance KernelDistanceDistributionDissimilarity Aggregation BB 2010

52 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Spectral shape distance KernelDistanceDistributionDissimilarityAggregation BB 2010

53 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Spectral shape distance KernelDistanceDistributionDissimilarityAggregation Diffusion distance BB 2010

54 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Particular case I: Rustamov GPS embedding KernelDistanceDistributionDissimilarityAggregation BB 2010

55 Numerical Geometry of Non-Rigid Shapes Feature-based methods & shape retrieval problems Particular case II: Mahmoudi&Sapiro KernelDistanceDistributionDissimilarityAggregation BB 2010