Real-Time Enveloping with Rotational Regression Robert Wang Kari Pulli Jovan Popović

Enveloped (skinned) characters are pervasive. Skeletons are often used to control meshes. skeletonmesh

Physically based modeling provides realistic deformations.  Realistic deformations – Finite-element based[Teran et al. 2005] – Anatomy based[Scheepers et al. 1997] – Elastically deformable[Capell et al. 2002, 2005] – Used in movie production – Off-the-shelf commercial tools  Slow evaluation [Teran et al. 2005] [Absolute Character Tools 1.6]

We learn a fast model from exported examples. Exported Examples (skeleton-mesh pairs) Fast Model Our method Black Box Simulation

Artists can still use existing modeling tools or scanned data. Exported Examples (skeleton-mesh pairs) Fast Model 3-D Scan Data

This is analogous to mesh simplification. High-resolution mesh Low-resolution mesh mesh simplification  Higher quality  Used in movie production  Faster to render  Optimized for interactive applications Physical simulation Rotational Regression Enveloping learning

How do we map a skeleton to a mesh? ? What parameters should we learn? How to model muscle deformations for fast evaluation?

Linear blend skinning linearly maps joint rotations to vertex positions.  Most popular enveloping technique for games  Coarse modeling parameters (but very simple)  Not very expressive (but very fast) + y y Figure from [Wang and Phillips 2002]

Linear blend skinning has many names.  Also known as, – Single-Weight Enveloping – Skeletal Subspace Deformation (SSD) – Or just, “Skinning”  We will use “Linear Blend Skinning” or “SSD.”

The two steps of our work are deformation gradients prediction and mesh reconstruction. Mesh reconstruction Deformation gradients prediction (Rotational Regression)

We present a replacement for linear blend skinning.  Coarse modeling parameters.  Can’t handle certain types of deformations.  Fast  Lets you use your existing modeling tool.  Good for muscle bulges.  Fast Whenever you have an existing model, you should use our technique instead of linear blend skinning. +

 Rigid components move with the bone rotation  Other surfaces rotate in the opposite direction Our model is inspired by the behavior of a flexing bicep. Surface rotation Bone rotation

Angle is scaled by u. Axis is offset by rotation W. source rotation (bone) target rotation (surface)

We map a sequence of bone rotations to a sequence of surface rotations. source rotation sequence (bone) target rotation sequence (surface) ……

We fit parameters u and W by regression. Skeleton rotations Surface rotations u’,W’ Best-fit parameters

Rotational regression is good at capturing muscle bulges.

Mesh reconstruction stitches deformation gradients together. Deformation gradients prediction Mesh reconstruction

 Least-squares problem equivalent to linear system.  Computation is matrix- multiplication. Mesh reconstruction solved with least-squares. deformation gradients vertex positions Least –squares C D(q)D(q)

Near-rigid vertices help eliminate low-frequency errors at extremities.  Low-frequency errors can accumulate at extremities of mesh  We fix a set of near-rigid vertices to their SSD predictions  Still a least squares problem

We build upon existing mesh reconstruction work.  Mesh IK[Sumner et al. 2005], [Der et al. 2006]  SCAPE[Anguelov et al. 2005]  Similar formulation, faster evaluation. [Anguelov et al. 2005]

Here’s a review of what we’ve covered. Rotational Regression Deformation Gradients Prediction Mesh Reconstruction Least-squares problem C Dk(q)Dk(q)

Model reduction lowers the dimensionality of problem.  Large multiplication on CPU  Smaller multiplications on GPU Dk(q)Dk(q) C  C’C’  Dl(q)Dl(q) SSD

Model reduction uses greedy clustering.  Vertices clustered into proxy-bones.  Per-triangle deformation gradients clustered into “key” deformation gradients. P =

Mesh reconstruction reduced to the following matrix-multiplications. C’C’  Dl(q)Dl(q) SSD weights “key” deformation gradients Map from “key” deformation gradients to proxy-bones All on GPU:  Computation in fragment program

Skinning Mesh Animations is a an alternative approach to model reduction.  The method from Skinning Mesh Animations uses mean-shift clustering and is more robust to errors. [James and Twigg 2005]  Our method minimizes vertex error and is faster

Deformation gradients prediction is now on “key” deformation sequences.  Fewer deformation gradient sequences to predict rotational regression.

Mesh reconstruction step reduced to matrix-multiplications on GPU.  Smaller matrix- multiplications  Supported on graphics hardware C’C’  Dl(q)Dl(q)

Our Technical Contributions: Rotational Regression Accurate and GPU-Ready Poisson Reconstruction Model Reduction

Results

Our work approximates the training examples better than SSD and also generalizes well.

Our model is suitable for interactive techniques.  Evaluation speed within a factor of two of SSD  Off-line training preprocess is usually less than half an hour

How does our work fit with previous work?

Our work is complementary to displacement correcting techniques.  Previous work provide corrective displacements. – Pose space deformation[Lewis et al. 2000], – Shape by example[Sloan et al. 2001], – Eigenskin[Kry et al. 2002]  Our work provides better approximation of rotations.  Our work complements approaches that build upon SSD. Figure from [Kry et al. 2001]

Displacement correcting approaches fail when SSD is very wrong.

Our work builds upon previous ideas on enriching the SSD model.  Multi-weight enveloping[Wang and Phillips 2002]  Additional joints[Mohr and Gleicher 2003]  Our technique has more parameters than SSD and generalizes the additional-bones model.

A more expressive model is useful here.

Our model doesn’t do a perfect job.  Not perfect reproduction – Inspired by muscle bulging and twisting. Other behaviors empirically validated. – Displacement correcting technique can be used for exact reproduction of examples.

Conclusion: Fast and accurate enveloping.  Fast evaluation of physical simulations through learning. – Within a factor of two of SSD on most models  Accurate reproduction of details – Better approximation and generalization – Complementary to previous work  A replacement for linear blend skinning

Acknowledgements  Funding – Nokia Research Center – National Science Foundation – Pixar Animation Studios  Hardware/Software – NVIDIA Corporation – Autodesk  Data – Drago Anguelov – Joel Anderson – Michael Comet, Comet Digital, LLC – Mark Snoswell, CG Character – Joseph Teran, Ron Fedkiw  MIT Graphics Group – Ilya Baran – Jiawen Chen – Sylvain Paris

Questions?  Thank you for coming to our talk!

Learning tasks trade expressiveness and simplicity. More Expressive: Captures more types of deformation. Simpler: Easier to fit Fewer training examples needed. Less likely to overfit. Rotational Regression

Linear blend skinning (SSD) is a rough and ready map from joint rotation matrices to vertex positions.  Most popular enveloping technique for games  Coarse modeling parameters (but very simple)  Not expressive enough (but very fast) desired deformation SSD deformation

Model Reduction  True optimization not as tractable  We approximate it with a greedy algorithm inspired by mesh simplification. difficult to solve simultaneously discrete optimization

Our work builds upon previous ideas on  Additional joints[Mohr and Gleicher 2003]  Multi-weight enveloping[Wang and Phillips 2002]  Our technique generalize the additional- bones model  We evaluate cross-validation error to test for overfitting [Wang and Phillips 2002]

Rotational regression is good at capturing muscle bulges.