1. 1 Applications Shmuel Peleg and Joshua Herman, “Panoramic Mosaics by Manifold Projection”, Computer Vision and Pattern Recognition (CVPR), 1997 Wolfgang Heidrich and Hans-Peter Seidel, “View-independent environment maps”, SIGGRAPH / Eurographics Workshop on Graphics Hardware, 1998 Matthew Brand, “Charting a Manifold”, Mitsubishi tech report, 2003 K.Grochow, S. Martin, A. Hertzmann, and Z. Popovic Style-based Inverse Kinematics, Siggraph 2004

2. 2 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Applications using manifolds Many problems can be phrased in manifold terminology Provides an alternative way of viewing the problem Can also provide some formalism

3. 3 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Applications Surface fitting Consistent parameterization Image-based rendering Environment mapping Animation

4. 4 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Surface fitting CT, MRI data, noisy, densely sampled in widely-spaced contours Boundary conditions less problematic Fitting Manifold Surfaces To 3D Point Clouds Cindy Grimm, David Laidlaw, and Joseph Crisco J. of Biomedical Engineering Feb. 2002

5. 5 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Consistent parameterization Build manifold once Fit to multiple bone point data sets

6. 6 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Aligning parameterization Identify similar points on meshes Pin parameterization

7. 7 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Application: Panoramas Problem statement: Given images from a known camera movement Rotation about camera axis “Push-broom” pan (assumes negligible depth) “Glue” images together into a single image Rover, nasa.gov

8. 8 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Camera rotation Final image can be rendered on a cylinder No parallax Each image samples some number of pixels on cylinder (manifold) image Peleg and Herman

9. 9 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Push broom/vertical slit camera Translation of camera Image slit perpendicular to camera motion Need not travel in straight line Depth differences negligible Parallax Manifold is part of ground plane viewed by camera Ground plane Direction of travel Peleg and Herman

10. 10 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Practical problem How to line up individual images to create one seamless image? Manifold: Final image (3D function RGB on 2D manifold) Charts: Individual images (2D charts) Overlap regions/transition functions: Unknown Assume translation (Account for optical effects of camera) Note: Only works for these two camera motions

11. 11 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html General solution Define a format for the transition function E.g., translation in x,y Define an error metric that measures how well two overlap regions agree E.g., pixel difference Optimize over free parameters in transition function E.g., x,y shift between all pairs of overlapping images

12. 12 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Solving for overlaps, transitions Find translation that minimizes pixel differences Find  that minimizes || I 0 (s) – I 1 (  (s))||  (s) = s +  s, where  s is unknown 0 1 ss

13. 13 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Final image Transition functions align images (abstract manifold) Final image colors? (RGB function on manifold) Blend and embedding functions for each chart Embedding function: Original image Blending function: How much to use of each overlapping image Usually favor very short blend regions

14. 14 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Constructing final image Define sampling for final image 0 1 I f (a 0 -1 (s)) = B 0 (s) I 0 (s) + B 1 (  (s)) I 1 (  (s))

15. 15 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Application: Environment mapping Place scene/model inside sphere Light intensity/color found by intersecting normal with sphere 1-1 mapping between normal direction and sphere Every point on sphere assigned light intensity/color Implementation Store colors in one (or more) texture maps (2D)

16. 16 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Parameterization Surface normal (point on sphere) to point in texture map Atlas/local parameterization Desirable properties Even sampling of sphere Adaptive Partition Overlap (mip mapping, continuity) Simple to compute Amenable to GPU implementation

17. 17 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Approach 1 Single texture map Not unique (poles) Poor sampling Simple to compute

18. 18 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Approach II Cube mapping Six charts Discontinuities at edges Sampling better at center of faces than edges Simple (plane) computation Which plane?

19. 19 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Approach III Parabolic mapping Chart functions use parabolic function Better sampling Slightly more computation Less-noticeable seams Heidrich and Hans-Peter Seidel

20. 20 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Proposal Use chart approach Allows for adaptive sampling (more detail where needed) Chart sizes uniform: Tile texture map Include overlap Minimal extra texture map Mip-mapping/down sampling

21. 21 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Application: Animation Human configuration space lies on a manifold of dimension n embedded in m dimensional space, where n << m Articulated skeleton: over 40 degrees of freedom (shoulders, knees, hips, etc., each 1-3 degrees of rotation) Individual motions (reaching, walking) certainly lie on lower dimension manifolds End-point of reach plus time Shape of manifold of all possible human motions? Who knows? K.Grochow, S. Martin, A. Hertzmann, and Z. Popovic

22. 22 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Overview Manifold learning Data samples (e.g., motion capture, key frames) Interpolation equals manifold construction Editing equals manifold editing High-level review of manifold learning techniques What kind of manifolds can you learn/construct from data? 2D animation example Manifolds in animation

23. 23 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html 2D illustration Two joint angles Circle X Circle manifold (torus) Animation Repetitive motion Joint angle plot Circle manifold Animation is a 1D manifold embedded in 2D

24. 24 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html 2D illustration Two joint angles Circle X Circle manifold (torus) Animation Repetitive motion Joint angle plot Circle manifold Animation is a 1D manifold embedded in 2D

25. 25 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Manifold learning Input: Sample points in R m E.g., Motion capture sequence, each pose is a data point, m is number of dof of joints 2D example: ,  for each pose Assume data lies on a manifold of dimension n Constraints on manifold shape/geometry (e.g., linear, no self-intersections) Goal: Parameterize manifold Single chart

26. 26 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Manifold learning, linear techniques Principal components analysis (PCA), Independent components analysis (ICA) Assumes shape of manifold in space is linear/planar Learn vector(s) that span the hyper-plane

27. 27 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Manifold learning, deformed linear Support vector machines (SVM) Deform space points are in first Deformation is “nice” Warps points to new places “Guess” which deformation will work Learn linear/planar function Deformation Linear parameterization

28. 28 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Manifold learning, local shape Isomap, Local linear embedding (LLE), Semi-definite embedding (SDE) Define a distance metric between points Assumptions: K closest neighbors are also closest K neighbors on the manifold Disk around point in manifold would contain all K neighbors, but not others Know dimension n of manifold Possible to deform K neighbors into n dimensions while maintaining (relative) distances Output: Single chart/parameterization of entire data set

29. 29 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Isomap, LLE, SDE cont. Non-obvious failure modes Circular/repetitive data sets Self-intersections Raw result: 2D embedding Modified : 1D embedding

30. 30 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Manifold construction as learning Use K neighbors to define chart domains (U c ) Charts are “squished” Gaussians Center, tangent vectors Find transition functions (affine transformations) Transformation takes tangent vectors into R n Aligns free vectors with neighbors UcUc Matthew Brand, “Charting a Manifold”, Mitsubishi tech report, 2003

31. 31 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Uses of animation manifold General idea: Construct (implicitly or explicitly) a manifold representing valid human poses Create a new animation sequence Foot must touch here, reach here, etc. Not sufficient to constrain all degrees of freedom (dof) Project on to manifold to fill in remaining dof K. Grochow, S. L. Martin, A. Hertzmann, Z. Popovic, Style-Based Inverse Kinematics, Siggraph 2004

32. 32 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Re-sequencing as embedded manifold Goal: Given existing sequence (samples), add more/change samples Assumptions: Samples come from some smooth manifold Some form of interpolation gives new samples on manifold Current approaches: Interpolation between neighboring samples in sequence for given new time

33. 33 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Re-phrasing problem Manifold learning or sequence timing provides parameterization/abstract manifold Embed manifold with smooth function Parameterization Use function fitting Re-sequencing: Evaluate embedding function E(M)

34. 34 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Caveats What makes animation data difficult? “Distance” loses meaning in >> 10 dimensions Every point equally far away Can’t enumerate Noise Error in capture process Skeleton only approximates human motion Joint angle representation Don’t explicitly deal with manifold, parameterization

35. 35 Siggraph 2005, 8/1/2005 www.cs.wustl.edu/~cmg/Siggraph2005/siggraph.html Summary Manifolds provide a formalism for breaking a problem into manageable pieces Charts provide local parameterization Planar domains Overlaps: Natural mechanism for moving between parameterizations Blend functions instead of geometric constraints No boundary condition problems Explicitly encapsulating/representing manifold is beneficial Cleaner algorithm specifications