1 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Non-Rigid Correspondence and Calculus of Shapes Of bodies changed to various forms, I sing. Ovid, Metamorphoses Alexander Bronstein, Michael Bronstein © 2008 All rights reserved. Web: tosca.cs.technion.ac.il

2 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Does a “natural” correspondence exist?

3 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Correspondence accurate ‘‘ ‘‘ makes sense ‘‘ ‘‘ beautiful ‘‘ ‘‘ Geometric Semantic Aesthetic

4 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Correspondence Correspondence is not a well-defined problem!  Chances to solve it with geometric tools are slim.  If objects are sufficiently similar, we have better chances. Correspondence between nonrigid deformations of the same object.

5 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 1D motivation: correspondence between curves Two curves, Arclength parametrization Unique up to initial point. Reparametrize and to canonical parametrization. Find correspondence between intervals Correspondence between and

6 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes The curse of higher dimension We relied on existence of a “canonical” arclength parametrization. Was possible due to existence of total ordering of points in 1D. Surfaces (2D objects) do not have a total ordering. Hence, no analogy of arclength parametrization for surfaces. We can still find an invariant parametrization.

7 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Invariant parametrization

8 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Invariant parametrization Ingredients: Parametrization domain. Group of deformations. Shape. Parametrization procedure, constructing given the shape. Desideratum: commutativity of the parametrization procedure with the deformation: How to construct such an invariant parametrization procedure?

9 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Compute minimal distortion embeddings Define intrinsic parametrizations Find rigid motion between parametrizations Define correspondence between shapes Canonical forms, bis

10 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Canonical forms Embedding into the plane is not distortionless. Invariance of parametrization holds only approximately Generally, there exists no rigid motion bringing and into perfect correspondence. Relax assumptions on : allow to be any bijection.

11 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Image processing insight Given two grayscale images and, find the optical flow (a.k.a. disparity map, motion field, etc.) minimizing the error Local image misalignment Given two shapes parametrized by and, find minimizing measures mismatch between and.

12 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Problem: functional depends on parametrization. Make it intrinsic replacing with. has also to be parametrization-independent. Example: normal misalignment Problem: not isometry-invariant. Make an intrinsic quantity, e.g., Not limited to geometric quantities. May include photometric information. Image processing approach

13 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Minimization problem is ill-posed! Add a regularization term Tikhonov Total variation Healthy solution to ill-posed problems

14 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Regularizer has to be parametrization-invariant. Frobenius norm is replaced by the Hilbert-Schmidt norm is an intrinsic quantity in parametrization coordinates is correspondence between shapes. is the intrinsic gradient on. is the norm in the tangent space of. Intrinsic regularization

15 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Physical insight Cauchy-Green deformation tensor Square of local change of distance due to elastic deformation. measures average distance deformation. = elastic energy (a.k.a. Dirichlet energy) of thin rubber sheet pressed against a mold.

16 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Dirichlet energy We have been looking for a regularizer… …but found a good measure for shape mismatch! is an intrinsic quantity. Minimizing gives a minimum deformation correspondence. Minimizer is a harmonic map of to. Some harmony

17 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Define a general energy functional is intrinsic, hence can be expressed in terms of the metric Correspondence problem becomes GMDS with generalized stress. Minimum distortion correspondence

18 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Minimum distortion correspondence

19 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes “Harmonic stress”: gives the norm of the Cauchy-Green tensor Our good old L 2 stress gives “as isometric as possible” correspondence. Generalized stress

20 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Minimum distortion correspondence Defined up to intrinsic symmetry of and.

21 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Partial correspondence

22 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Minimum distortion correspondence MATLAB ® intermezzo

23 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes TIME ReferenceTransferred texture Texture transfer

24 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Virtual body painting

25 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Texture substitution I’m Alice.I’m Bob. I’m Alice’s texture on Bob’s geometry

26 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Given two shapes and, and a correspondence. We can define a convex combination of the two shapes as a new shape, where the extrinsic location of each point is given by Alternatively Define deformation field transforming into and express. We can create new shapes by adding or subtracting other shapes. We have a calculus of shapes. Calculus of shapes

27 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Calculus of shapes in shape space Extrapolation Interpolation

28 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Temporal super-resolution TIME

29 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Motion-compensated interpolation

30 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Metamorphing 100% Alice 100% Bob 75% Alice 25% Bob 50% Alice 50% Bob 75% Alice 50% Bob

31 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Face caricaturization EXAGGERATED EXPRESSION

32 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes The quest for trajectory In our definition of linear trajectory between corresponding points was used. If and are extrinsically similar, this gives good result. Generally, there is no guarantee that is a valid shape: Not a manifold Self-intersecting Even if shape is valid, it is not necessarily isometric to.

33 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Given two shapes and, and a correspondence, we want to find intermediate shapes. For each point, define a trajectory for such that The big question: How to select trajectories? No self-intersections of intermediate meshes. No distortion of intrinsic geometry in intermediate meshes. The quest for trajectory

34 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Define deformation field Tangent to the trajectory In order for intermediate shapes to be isometric to, must hold for all and. Deformation field

35 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Killing field: deformation field preserving the metric. Satisfies for all and May not exist, even if and are isometric! Remember: not every nonrigid shape is continuously bendable… The Killing field Wilhelm Karl Joseph Killing ( )

36 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes As isometric as possible deformation field. Define inner product between deformation fields of Induces a norm Problem: vanishes for being a rigid motion. Solution: add stiffening term: Metric for deformation fields

37 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes We have a Riemannian metric on the space of shapes. Find a minimal geodesic connecting between and. Boundary conditions. Minimum deviation from Killing field along the path. As isometric as possible morph. As isometric as possible morph

38 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Summary and suggested reading Invariant surface parametrization G. Zigelman, R. Kimmel, and N. Kiryati, Texture mapping using surface flattening via multi-dimensional scaling, IEEE TVCG 9 (2002), no. 2, 198–207. An image processing insight to correspondence B. K. P. Horn and B. G. Schunck, Determining optical flow, Artificial Intelligence 17 (1981), no. 1-3, 185–203. Harmonic embeddings N. Litke, M. Droske, M. Rumpf, and P. Schroder, An image processing approach to surface matching. Minimum distortion correspondence Calculus of shapes A.M. Bronstein, M.M. Bronstein, R. Kimmel, Calculus of non-rigid surfaces for geometry and texture manipulation, IEEE TVCG 13 (2007), no. 5, 903–913.

39 Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes Summary and suggested reading Morphing M. Alexa, Recent advances in mesh morphing, Computer Graphics Forum 21 (2002), no. 2, 173–196. V. Surazhsky and C. Gotsman, Controllable morphing of compatible planar triangulations, ACM Trans. Graphics 20 (2001), no. 4, 203–231. M. Kilian, N. J. Mitra, and H. Pottmann, Geometric modeling in shape space, ACM Trans. Graphics 26 (2007), no. 3.