MSRI Program: Mathematical, Computational and Statistical Aspects of Vision Introductory workshop, Jan 24-28, 2005 Modeling Shape David Mumford

The problem of shape 1D vs. 2D signals: in 1D, boundaries decompose the signal domain into intervals; in 2D, the boundaries and the parts are less trivial. One can often recognize an object in an image by its shape alone. By a ‘shape’ in n-dimensions, let’s mean an open subset S  R n with not too convoluted a boundary and usually meaning topologically a ball. Remarkably, people find it natural to answer the question: are two shapes S and T similar? The first key problem in a computational theory is to define similarity, i.e. to put a metric on the space S of all shapes.

What is the space of shapes? S 2 ≈ set of all “smooth” connected plane curves, no self-intersections (“simple closed curves”) Infinite dimensional! Not a vector space BUT, locally linear, i.e. a manifold Tangent vector to S n at shape R = normal vector field along bdry of R

Think of S n geometrically A curve on S n is a warping of one shape to another. On S 2, the set of ellipses forms a surface: The geometric heat equation: is a vector field on S n

2D shapes often come in categories Typical shapes and examples of desired clustering in computer vision experiments. Top right: samples from the NIST handwritten zip code database often used in statistical learning theory; the ‘hat’ is Saint-Exupery’s pattern recognition challenge. Such categories should be subsets U  S 2 and datasets give point clouds in S 2 One seeks probability measures on S 2 to model these clouds, to do Bayesian inference.

Why more than one metric is needed The central shape is similar in various respects to all 5 of the shapes around in – but in different metrics! Distance between shapes can be measured via averages (L 1 ), worst cases (L ∞ ) or by mean squares (L 2 ) and using points, 1-jets, 2-jets, etc. In L 1, distances are: A < B,C < D,E In L , distances are: B < C,D <A,E In L  with 1-jets: D < B,C < A,E In L 1 with 2-jets: D < A,B < C,E To make E close, need ‘robust’ non-convex metrics that discard outliers (e.g. L 1/2 ).

Two simple metrics L 1 -metric leading to set of meas. subsets S ⊂ R 2 : infinitesimally: leads to path length: leads to global metric: Frechet metric (like Hausdorff metric) on cont. maps f:S 1 →R 2 : infinitesimally: leads to path length: leads to global metric: Neither metric has good geodesics – balls are like boxes, but they stack well, can measure ‘volume’ (K.Leonard, using  -entropy)

Advantages of L 2 metrics Can define gradient flows of a function. Have a beautiful theory of locally unique geodesics, thus a warping of one shape to another. Can define the Riemannian curvature tensor. If non-positive, have a good theory of means. Can expect a theory of diffusion, of Brownian motion, hence Gaussian-type measures and their mixtures. WHERE DO THEY COME FROM? 1.Local: 2.Global: use G n = gp. of diffeomorphisms of R n and S n ≈ G n / subgp fixing unit ball, take quotient of metric on G n 3.Conformal (n=2): use S 2 ~ diffeos of S 1

The simplest local metric Define the function L(R)=(n-1)-volume of boundary, then  L = heat eqn vector fld.a = κ ∂R, i.e. BUT These are inf.dim.Riem. manifolds for which forward geodesics, curvature, etc. work fine; But 2 pt. bdry value problem fails, geod eqn is hyperbolic.

The Michor metric – the simplest Riemannian metric Infinitesimally: Globally: If A=0, get ‘geodesic spray’, positive curvature, but infimum of path lengths is zero If A>0, the metric controls the change in length(C) and gives interesting geodesics – not always unique.

A geodesic triangle Consider an ellipse rotated through 0, 60 and 120 degrees. These 3 ellipses form a triangle in S 2. Using the metrics with A=1.0, 0.1 and 0.01, we join them with 3 geodesics. The path in S 2 forming one of these edges is shown in the first row for the 3 metrics. The second row shows the whole triangle of shapes. When A=1, we have negative curvature, the angle sum is 102 o and the shapes on the edges fall back towards the unit circle. When A=.01, we have positive curvature, the angle sum is 207 o and the edge shapes tend towards parallograms, away from the unit circle.

2 nd Riemannian metric: diffeomorphisms of  n Write the space of shapes   as a homogeneous space w.r.t.   Diff(  n ):     subgp  fixing unit sphere) Put an inner product on vector fields: and define the length of a path {θ t } in  by: If {θ t } is thought of a fluid flow, then ν t (x) is its velocity. This is right invariant, so  acts on right by isometries, so    is a ‘Riemannian submersion’,

A fluid flow equation gives geodesics in these metrics Geodesics on  are geodesics on  starting, and hence continuing,  to cosets θ  If {θ t } is thought of a fluid flow, then v t = velocity, u t = Lv t = ‘momentum’ in this metric. Geodesics now are solutions to a regularized compressible form of Euler’s equation ( Arnold,Vishik ): Treating u as a section of  1   n (so makes intrinsic sense), the equation says the momentum u is constant along the flow given by v. The equation is linear in u, so u can be a generalized function! To get geodesics for shapes, u should be supported on their bdry To get geodesics for finite sets of points, u should be a sum of delta fcns

A geodesic in G whose momentum is concentrated at four points ( Younes,Miceli ) This is a fast and efficient tool for dealing with shapes via finite sets of landmark points ( Kendall ).

Elasticity – the solid approach In a liquid, particles have no memory of where they were initially. In a solid, they do. On the group G, consider the strain matrix: Depending on the material, there will be a strain energy density and a total strain energy a function on G, measuring distance from I. e may be inhomogeneous and anisotropic or it may have a generic form: Minimizing the strain energy plus an image mismatch term used the face warping.

2 examples of image warping via diffeomorphisms Faces warped using strain energy as prior (P.Hallinan, “2 and 3D Patterns of the Face”) Heart warped using geodesic length as prior (Miller et al)

Shape via complex analysis In dimension 2 only, can replace the real coordinates x,y by a single complex coordinate z=x+iy. A basic construction from complex analysis puts nearly unique global coordinates on any shape: Apply this twice, to the inside and outside of a shape: The fingerprint of the shape is:

Two examples An ellipse and a kidney shaped object, with the conformal parametrization of their interiors and exteriors marked. The interior map has been chosen to carry 0 to 0, (but it may take 0 to any other interior point.)

Good things about the conformal approach The fingerprint determines the shape up to translation and scaling, i.e. there is a bijection: We get an action of the group Diff(S 1 ) on the space of shapes, hence can approximate shapes via words in elementary diffeomorphisms. There is a unique Riemannian metric on for which the group action is made of isometries. (Note analogy with ordinary distances on R n.) The curvature of this metric is non-positive, so we have unique geodesics, means, etc. This representation leads to a simple construction of an axis, hence to a decomposition of into cells.

2 geodesics in the W-P metric ( E.Sharon )

Axes: the royal road to shape description Humans perceive shapes as having ‘parts’, linked in a combinatorial pattern. The axis gives this ( and even bit length compression, Leonard 2004).

Axes in three dimensions Axes in 3D are trickier: Yan Cao’s definition: Given a shape S, or even an arbitrary measure m with support S, consider the functional on potential axes: An anatomical example:

Axes via the fingerprint Minima of  ’ correspond (roughly) to points on C nearest to  0 (0). Combinatorial structure of the axis leads to a natural cell decomposition of S 2.