1. Stochastic Systems Group Some Rambling (Müjdat) and Some More Serious Discussion (Ayres, Junmo, Walter) on Shape Priors

2. Stochastic Systems Group Desire to use Shape Priors in Segmentation The most common (implicit) prior used is the curve length penalty: Segmenting curve Observed image data The posterior: Want to be able to use better prior models

3. Stochastic Systems Group Challenges and remarks Need probabilistic descriptions in the space of shapes –A non-linear, infinite-dimensional manifold –Distance (similarity) measures in the shape space Of course, a statistical description for shapes has uses other than segmentation as well –Sampling from a shape density –Recognition of objects –Completion of incomplete shapes

4. Stochastic Systems Group The PCA World Cootes & Taylor –Shape representation using marker points Leventon, Tsai –PCA and level sets More… Want a more principled approach

5. Stochastic Systems Group Some Literature “Active shape models - their training and application,” T. Cootes, C. J. Taylor, D.H. Cooper, J. Graham, 1995. “Embedding Gestalt laws in Markov random fields,” Song-Chun Zhu, 1999. “On the incorporation of shape into geometric active contours,” Y. Chen, H.D. Tagare et al., 2001. “Image segmentation based on prior probabilistic shape models,” A. Litvin and W. C. Karl, 2002. “Shape priors for level set representations,” M. Rousson and N. Paragios, ECCV, 2002. “Nonlinear shape statistics in Mumford-Shah based segmentation,” D. Cremers, T. Kohlberger, and C. Schnorr, 2002. “Geometric analysis of constrained curves for image understanding,” A. Srivastava, W. Mio, E. Klassen, X. Liu, 2003. “Analysis of planar shapes using geodesic paths on shape spaces,” E. Klassen, A. Srivastava, and W. Mio (in review) “Gaussian distributions on Lie groups and their application to stat. shape analysis,” P. T. Fletcher, S. Joshi, C. Lu, and S. Pizer, 2003.

6. Stochastic Systems Group Overview of Anuj Srivastava’s Work Specify a space of continuous curves with constraints (e.g. simple closed) and exploit the differential geometry of this space Use geodesic paths for deformations between curves and to compute distances – do not have analytical exps. for geodesics To move in the shape manifold, first move in a linear space and then project back (using tangents/normals) Given two curves, solve an optimization problem to find the geodesic path between them (find a local min) Build statistical descriptions based on Karcher means; covariance of (Fourier coefficients of) tangent vectors Use such descriptions as priors (very preliminary) Some current limitations: –Cannot handle topological changes –Extension to surfaces in 3-D not straightforward

7. Stochastic Systems Group Outline Some metrics proposed for shape similarity (Junmo) Anuj Srivastava’s work –Geometric representations of curves and shape spaces (Walter) –Tangents, normals, geodesics (Ayres) –Statistical models and application to segmentation (Junmo) Brief highlights from Pizer’s work (Walter) Discussion on all of this

8. Stochastic Systems Group Some Metrics on the Space of Shapes Notion of “similarity” between shapes – Basic task of vision system is to recognize similar objects which belong to the same category. Describing shapes : landmarks, level set Shape can be described as There are several metrics for two shapes

9. Stochastic Systems Group Hausdorff Metric Given Very sensitive to any outlier points in and

10. Stochastic Systems Group Template Metric Totally insensitive to outliers

11. Stochastic Systems Group Transport Metric Fill with ‘stuff’ and find the shortest paths along which to move this ‘stuff’ so that it now fills

12. Stochastic Systems Group Optimal Diffeomorphism If and are topologically different, the distance would be infinite. –E.g. is minus a pinhole –E.g. A small break cuts a shape into two

13. Stochastic Systems Group Geometric Representations of Curves and Shape Spaces Restrict attention to closed curves in R 2. Classify curves which differ only by orientation preserving rigid motions (rotation and translation) and uniform scaling as the same shape. Consider two different representations of planar curves for simple closed curves: –Using direction functions. –Using curvature functions.

14. Stochastic Systems Group Preliminaries First, the issue of scaling is resolved by fixing the length of all curves to 2 . Curves are parameterized by arc length with period 2  & Define the unit tangent function where S 1 is the unit circle, & where  (s) is the direction function.  (s+2  ) -  (s) = 2  n (n = rotation index { = 1})

15. Stochastic Systems Group Curves to Consider Consider entire set of curves with rotation index 1 because this set is complete. –Contains its own limit points. Note that the set of simple closed curves is an open subset of this larger set.

16. Stochastic Systems Group Shape Using Direction Functions Direction function for S 1 is  0 (s) = s. All other closed curves have direction function  =  0 + f, where f is L 2 periodic on [0,2  ]. To adjust for rigid rotations, restrict attention to  s.t. To ensure curve closure, require

17. Stochastic Systems Group More Direction Functions Define a map by The pre-shape space C 1 is  1 -1 ( ,0,0). Multiple elements of C 1 may denote the same shape. An adjustment of the reference point (s=0) handles this.

18. Stochastic Systems Group Geometry of Shape Manifolds Constraints define a manifold embedded in  0 + L 2 Move along manifold by moving in tangent space and projecting back to manifold Tangent space is infinite dimensional, but normal space is characterized by three constraints defined in  

19. Stochastic Systems Group Tangents and Normals The derivative of   in the direction of f at  is: Implies d   is surjective If f is orthogonal to {1, sin , cos  }, then d   =0 in the direction of f and hence f is in the tangent space

20. Stochastic Systems Group Projections Want to find the closest element in C 1 to an arbitrary    0 + L 2 Basic idea: move orthogonal to level sets so projections under  form a straight line in R 3 For a point b  R 3, we define the level set as: Let b 1 =( ,0,0). Then its level set is the preshape space C 1

21. Stochastic Systems Group Approximate Projections If points are close to C 1, then one can use a faster method Let d  be the normal vector at  for which  (  +d  )=b 1. Can do first order approximation to compute this Approximate Jacobian as:

22. Stochastic Systems Group Iterative algorithm Define the residual (error) vector as Then: where Iteratively update  d   until the error goes to zero Call this projection operator P

23. Stochastic Systems Group Example Projections Fig. 1: Projections of arbitrary curves into C 1

24. Stochastic Systems Group Geodesics Definition: For a manifold embedded in Euclidean space, a geodesic is a constant speed curve whose acceleration vector is always perpendicular to the manifold Define the metric between two shapes as the distance along the manifold between the shapes with respect to the L 2 inner product Nice features: –Defined for all closed curves –Interpolants are closed curves Finds geodesics in a local sense, not necessarily global

25. Stochastic Systems Group Paths from initial conditions Assume we have a  in C 1 and an f in the tangent space Approximate geodesic along manifold by moving to  +f  t and projecting that back onto the manifold (  t is step size) So  (t+  t) = P(  (t) +f (t)  t)

26. Stochastic Systems Group Transporting the tangent vector Now f (t) is not in the tangent space of  (t+  t) Two conditions for a geodesic: –The acceleration vector must be perpendicular to the manifold: simply project f into the next tangent space –The curve must move at constant speed: renormalize so ||f (t+1) ||=||f (t) || h k is the orthonormal basis of the normal space

27. Stochastic Systems Group Geodesics on shape spaces S 1 is a quotient space of C 1 under actions of S 1 by isometries, so finding geodesics in S 1 equivalent to finding geodesics in C 1 which are orthogonal to S 1 orbits S 1 acting by isometries implies that if a geodesic in preshape space is orthogonal to one S 1 orbit, it’s orthogonal to all S 1 orbits which it meets So now normal space has one additional component spanned by The algorithm is the same as detailed earlier except with an expanded normal space

28. Stochastic Systems Group Geodesics between shapes We know how to generate geodesic paths given  and f Now we want to construct a geodesic path from  1 to  2 So we need to find all f that lead from  1 to an S 1 orbit of  2 in unit time, and then choose the one that leads to the shortest path Let  define the geodesic flow, with  (  1,0,f)=  1 as the initial condition We then want  (  1,1,f)=  2

29. Stochastic Systems Group Finding the geodesic Define an error functional which measures how close we are to the target at t=1: Choose the geodesic as the flow  which has the smallest initial velocity ||f|| i.e., min ||f|| s.t. H[f]=0 Hard because infinite dimensional search

30. Stochastic Systems Group Fourier decomposition f  L 2, so it has a Fourier decomposition Approximate f with its first m+1 cosine components and its first m sine components: Let a be the vector containing all of the Fourier coefficients Now optimization problem is min ||a|| s.t. H[a]=0

31. Stochastic Systems Group Geodesic paths Fig. 2: Geodesic paths between two shapes

32. Stochastic Systems Group Statistical Modeling of Shapes Given example shapes –Mean shape –Shape variation –Shape prior –Sampling from the prior –Using shape prior for segmentation of occluded images

33. Stochastic Systems Group Mean Shape Given the geodesic distance function, Karcher mean of shapes is defined to be a shape for which the variance function is a local minimum The Karcher mean exists, but may not be unique

34. Stochastic Systems Group Variances on Shape Spaces Model the variation from the mean shape as, an element of the tangent space at the mean shape Represent by its Fourier expansion: Model as multivariate normal with mean 0 and covariance matrix

35. Stochastic Systems Group Shape Sampling Sample Fourier coefficients of tangent vectors from the multivariate normal distribution Move along the geodesic path starting from the mean shape in the direction of by distance

36. Stochastic Systems Group Shape Sampling Examples Observed shapes Mean shape Random samples from the Gaussian model

37. Stochastic Systems Group Shape Prior : the space of curves (larger than shape space) can be represented as pairs – : parameters for translation, rotation, and scaling – : the shape Gaussian density with a mean shape with the shape dispersion

38. Stochastic Systems Group Bayesian Discovery of Objects

39. Stochastic Systems Group Some closing thoughts on Anuj Srivastava’s work Most energy has been spent on manipulating the shape manifold Work on using these models as priors preliminary –Non-diagonal covariance – explore modes of variation –Non-Gaussian? –Mean in manifold? Other thoughts –Non-Fourier representations for tangents – KL expansion? –Could similar ideas be used with representations based on signed distance functions?

40. Stochastic Systems Group Overview of Steve Pizer’s Work Medial representations (m-reps) are used to model the geometry of anatomical objects. Medial parameters are not in a Euclidean space; so, PCA cannot be used. However, m-reps model parameters are elements of a Lie group. Gaussian distributions on this Lie group are considered, with the max likelihood estimates of mean and covariance derived. Similar to PCA for Euclidean spaces, principal geodesic analysis (PGA) on Lie groups are defined for the study of anatomical variability. Framework is applied to hippocampi in a schizophrenia study. –86 m-rep figures are first aligned (translation/rotation/scaling) –Intrinsic mean is then computed –PGA (modes of variability) are then computed –Results yield smoother deformations when compared with PCA

41. Stochastic Systems Group Medial Representation Introduced by Blum (1978), a 3-D object is represented by a set of connected continuous medial manifolds formed by the centers of all spheres are are interior to the object and tangent to the object boundary at two or more points. The figure below illustrates this:

42. Stochastic Systems Group Medial Atom & Lie Groups A medial atom is represented by –The location in space (R 3 ) –The radius of the sphere (R + ) –The local frame (SO(3)) –The object angle (SO(2)) R 3 is a Lie group under vector addition, R + is a multiplicative Lie group, and SO(2) & SO(3) are Lie groups under composition of rotations. The direct product of Lie groups is a Lie group.

43. Stochastic Systems Group Lie Groups and Lie Algebras A Lie group is a group G that is a finite-dim manifold such that the two group operations of G, multiplication and inverse, are C 2 mappings. If e is the identity of G, the tangent space at e forms a Lie algebra. The exponential map provides a method for mapping vectors in the tangent space into the Lie group.

44. Stochastic Systems Group Alignment and PGA Translation: each model is situated so that the average of its medial atoms is at the origin Rotation and scaling are done in a manner which minimizes the total sum-of-squared distances between m-rep figures. After alignment, principal directions in the geodesic are computed, and the analog to PCA is performed.

45. Stochastic Systems Group Results The analysis, performed on 86 aligned hippocampus m-reps, shows smooth deformations (compared with PCA). The mean shape is top left, the medial atoms are overlaid lower left, and the first 3 PGA modes are shown right.

46. Stochastic Systems Group Shape Using Curvature Functions Alternatively, curves can be represented by curvature fcns. Since the rotation index is 1, Using, the closure condition

47. Stochastic Systems Group More Curvature Functions Define a map by then the pre-shape space C 2 is  2 -1 (2 ,0,0). As with direction functions, different placements of s=0 result in different C 2 shapes. Thus, re-parameterization is needed.