Feature-based methods Numerical geometry of non-rigid objects Feature-based methods Alexander Bronstein Michael Bronstein 1

Two perspectives Metric geometry Computer vision Features and local descriptors Metric geometry Shapes as metric spaces 2

Invariant retrieval Illumination View Missing data Deformation Topology Missing data 3

Bags of words Notre Dame de Paris is a Gothic cathedral in the fourth quarter of Paris, France. It was the first Gothic architecture cathedral, and its construction spanned the Gothic period. Notre Dame de Paris is a Gothic cathedral in the fourth quarter of Paris, France. It was the first Gothic architecture cathedral, and its construction spanned the Gothic period. construction architecture Italy France cathedral church basilica Paris Rome Gothic Roman St. Peter’s basilica is the largest church in world, located in Rome, Italy. As a work of architecture, it is regarded as the best building of its age in Italy. St. Peter’s basilica is the largest church in world, located in Rome, Italy. As a work of architecture, it is regarded as the best building of its age in Italy. 4

Feature detector + descriptor Bags of features Visual vocabulary Feature detector + descriptor Invariant to changes of the image Discriminative (tells different images apart) 5

Images vs shapes Images Shapes Many prominent features Few prominent features Affine transforms, illumination, occlusions, resolution Non-rigid deformations, topology, missing parts, triangulation SIFT, SURF, MSER, DAISY, … Curvature, conformal factor, local distance histograms 6

Diffusion scale space can be interpreted as probability of Brownian motion to remain at the same point after time (represents “stability” of the point) Multiscale descriptor Time (scale) J. Sun, M. Ovsjanikov, L. Guibas, SGP 2009 M. Ovsjanikov, BB, L. Guibas, 2009 7

Heat kernel descriptor represented in RGB space Heat kernel descriptors Heat kernel descriptor represented in RGB space J. Sun, M. Ovsjanikov, L. Guibas, SGP 2009 M. Ovsjanikov, BB, L. Guibas, 2009 8

Our descriptor is… Dense Defined at every point of the shape No feature detection Statistical weighting (reduce the influence of “dull” points) Intrinsic Based on diffusion geometry, hence intrinsic Invariant to isometric deformations Insensitive to topological noise Consistent Uses a geometrically consistent discretization of the Laplace-Beltrami operator Insensitive to different sampling and triangulations M. Ovsjanikov, BB, L. Guibas, 2009

Heat kernel descriptors Invariant to isometric deformations Localized sensitivity to topological noise J. Sun, M. Ovsjanikov, L. Guibas, SGP 2009 M. Ovsjanikov, BB, L. Guibas, 2009 10

Geometric vocabulary M. Ovsjanikov, BB, L. Guibas, 2009 11

Nearest neighbor in the descriptor space Bags of features Given a geometric vocabulary for each point with local descriptor find representation Hard quantization Soft quantization Weighted distance to words in vocabulary Nearest neighbor in the descriptor space M. Ovsjanikov, BB, L. Guibas, 2009 12

Bags of features Statistics of different geometric words over the entire shape Shape distance = distance between bags of features M. Ovsjanikov, BB, L. Guibas, 2009 13

Bags of features 1 Index in vocabulary 64 14 M. Ovsjanikov, BB, L. Guibas, 2009 14

Expressions In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem. In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem. In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem. In biological science, decomposition is the process of organisms to break down into simpler form of matter. Usually, decomposition occurs after death. In biological science, decomposition is the process of organisms to break down into simpler form of matter. Usually, decomposition occurs after death. Matrix is a science fiction movie released in 1999. Matrix refers to a simulated reality created by machines in order to subdue the human population. Matrix is a science fiction movie released in 1999. Matrix refers to a simulated reality created by machines in order to subdue the human population. Matrix is a science fiction movie released in 1999. Matrix refers to a simulated reality created by machines in order to subdue the human population. matrix decomposition is a the of in to by science form matrix decomposition matrix factorization science fiction canonical form M. Ovsjanikov, BB, L. Guibas, 2009 15

Expressions In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem. In particular matrix used type a some science, decomposition form a factorization of is canonical. matrix math decomposition is in a Each problem. into of matrix decomposition is a the of in to by science form matrix decomposition matrix factorization science fiction canonical form M. Ovsjanikov, BB, L. Guibas, 2009 16

Visual expressions “Inquisitor King” Inquisitor, King Giuseppe Verdi, Don Carlo, Metropolitan Opera 17

Geometric expressions Yellow “Yellow Yellow” No total order between points (only “far” and “near”) Geometric expression = a pair of spatially close geometric words M. Ovsjanikov, BB, L. Guibas, 2009 18

Spatially-sensitive bags of features Distribution of pairs of geometric words is the probability to find word at point and word at point Proximity between points and is the statistic of geometric expressions of the form Shape distance M. Ovsjanikov, BB, L. Guibas, 2009 19

Spatially-sensitive bags of features M. Ovsjanikov, BB, L. Guibas, 2009 20

Query Positives Negatives Iso Topo Triang Part Query Positives Negatives

Shape Google Query Match 1 Match 2 Match 3 Match 25 Iso Iso 0.060 Null 0.072 Noise 0.382 Iso Iso+Topo 0.011 Iso 0.019 Iso+Topo 0.023 Iso 0.073 M. Ovsjanikov, BB, L. Guibas, 2009 22

Shape Google Query Match 1 Match 2 Match 3 Match 25 Part Part 0.292 Iso+Topo 0.372 Null 0.377 Triang 0.575 Iso+Topo Iso 0.014 Iso+Topo 0.108 Noise 0.114 Iso+Topo 0.459 M. Ovsjanikov, BB, L. Guibas, 2009 23

Bags of features 100% 98% 96% True Positive Rate (TPR) 90% 80% 0.1% 1% 10% 100% M. Ovsjanikov, BB, L. Guibas, 2009 False Positive Rate (FPR) 24

Spatially-sensitive bags of features 100% 98% 96% True Positive Rate (TPR) 90% 80% 0.1% 1% 10% 100% M. Ovsjanikov, BB, L. Guibas, 2009 False Positive Rate (FPR) 25

Comparison SS-BoF 8% BoF Shape DNA [Reuter et al.] Equal Error Rate (EER) 4% 0% All Null Iso. Part. Topo. Triang. Iso.+Topo. M. Reuter et al., 2006 M. Ovsjanikov, BB, L. Guibas, 2009 26