Learning Human Pose and Motion Models for Animation Aaron Hertzmann University of Toronto

Animation is maturing … … but it’s still hard to create

Keyframe animation

q1q1 q2q2 q3q3 q (t)

Characters are very complex Woody: facial controls controls in his body

Motion capture [Images from NYU and UW]

Motion capture

Mocap is not a panacea

Problem Animation is very time-consuming Fine for big studios Problem for:

Goal: model human motion What motions are likely? Applications: Computer animation Computer vision

Related work: physical models Accurate, in principle Too complex to work with (but see [Liu, Hertzmann, Popović 2005]) Computationally expensive

Related work: motion graphs Input: raw motion capture “Motion graph” (slide from J. Lee)

Approach: statistical models of motions Learn a PDF over motions, and synthesize from this PDF [Brand and Hertzmann 1999] What PDF do we use?

Style-Based Inverse Kinematics with: Keith Grochow, Steve Martin, Zoran Popović

Motivation

Body parameterization Pose at time t : q t Root pos./orientation (6 DOFs) Joint angles (29 DOFs) Motion X = [ q 1, …, q T ]

Forward kinematics Pose to 3D positions: qtqt [x i,y i,z i ] t FK

Problem Statement Generate a character pose based on a chosen style subject to constraints Constraints Degrees of freedom (DOFs) q

Real-time Pose Synthesis Off-Line Learning Approach Motion Learning Style Synthesis Pose Constraints

Style Representation Objective function –given a pose evaluate how well it matches a style –allow any pose Probability Distribution Function (PDF) –principled way of automatically learning the style

y(q) = q orientation(q) velocity(q) [ q 0 q 1 q 2 …… r 0 r 1 r 2 v 0 v 1 v 2 … ] Features

Goals for the PDF Learn PDF from any data Smooth and descriptive Minimal parameter tuning Real-time synthesis

Mixtures-of-Gaussians

GPLVM y1y1 y2y2 y3y3 x1x1 x2x2 Latent Space Feature Space Gaussian Process Latent Variable Model [Lawrence 2004] GP   -1 x ~ N (0,I) y ~ GP(x;  ) Learning: arg max p(X,  | Y) = arg max p(Y | X,  ) p(X)

Scaled Outputs Different DOFs have different “importances” Solution: RBF kernel function k(x,x’) k i (x,x’) = k(x,x’)/w i 2 Equivalently: learn x  Wy where W = diag(w 1, w 2, … w D )

Style Learning y1y1 y2y2 y3y3 x1x1 x2x2

Precision in Latent Space  2 (x)

Pose Synthesis y1y1 y2y2 y3y3 x1x1 x2x2 arg min x,q p(y(q),x|X,Y,  ) s.t. C(q) = 0

Pose Synthesis arg min x,q p(y(q),x|X,Y,  ) s.t. C(q) = 0 Constraints Degrees of freedom (DOFs) q

SGPLVM Objective Function y1y1 y2y2 y3y3 x1x1 x2x2

Baseball Pitch

Track Start

Jump Shot

The Active Set All training dataActive set data Training Data

Annealing Original Style High Variance Medium Variance Original Style

Style interpolation Given two styles  1 and  2, can we “interpolate” them? Approach: interpolate in log-domain

Style interpolation (1-s)s

Style interpolation in log space (1-s) s

Applications

Interactive Posing

Multiple motion style

Realtime Motion Capture

Style Interpolation

Trajectory Keyframing

Posing from an Image

Modeling motion GPLVM doesn’t model motions Velocity features are a hack How do we model and learn dynamics?

Gaussian Process Dynamical Models with: David Fleet, Jack Wang

Dynamical models x t+1 xtxt

Hidden Markov Model (HMM) Linear Dynamical Systems (LDS) [van Overschee et al ‘94; Doretto et al ‘01] Switching LDS [Ghahramani and Hinton ’98; Pavlovic et al ‘00; Li et al ‘02] Nonlinear Dynamical Systems [e.g., Ghahramani and Roweis ‘00] Dynamical models

Gaussian Process Dynamical Model (GPDM) Marginalize out, and then optimize the latent positions to simultaneously minimize pose reconstruction error and (dynamic) prediction error on training data. pose reconstruction latent dynamics Latent dynamical model : Assume IID Gaussian noise, and with Gaussian priors on and

Reconstruction where contains the th -dimension of each training pose is a kernel matrix with entries for kernel function (with hyperparameters ) scales different pose dimensions The data likelihood for the reconstruction mapping, given centered inputs has the form:

Reconstruction The data likelihood for the reconstruction mapping, given centered inputs has the form: where is a kernel matrix with entries for kernel function (with hyperparameters ) scales different pose dimensions

Dynamics The latent dynamic process on has a similar form: where is a kernel matrix defined by kernel function with hyperparameters

Subspace dynamical model : Markov Property Remark: Conditioned on, the dynamical model is 1 st -order Markov, but the marginalization introduces longer temporal dependence.

Learning To estimate the latent coordinates & kernel parameters we minimize with respect to and. GPDM posterior: reconstruction likelihood priorsdynamics likelihood training motions hyperparameterslatent trajectories

Motion Capture Data ~2.5 gait cycles (157 frames)Learned latent coordinates (1st-order prediction, RBF kernel) 56 joint angles + 3 global translational velocity + 3 global orientation from CMU motion capture database

3D GPLVM Latent Coordinates large “jumps’ in latent space

Reconstruction Variance Volume visualization of. (1 st -order prediction, RBF kernel)

Motion Simulation Animation of mean motion (200 step sequence) initial state Random trajectories from MCMC (~1 gait cycle, 60 steps)

Simulation: 1 st -Order Mean Prediction Red: 200 steps of mean prediction Green: 60-step MCMC mean Animation

Linear Kernel Dynamics Animation 200 steps of mean prediction

Missing Data 50 of 147 frames dropped (almost a full gait cycle) spline interpolation

Missing Data: RBF Dynamics

Missing Data: Linear Dynamics

Determining hyperparameters GPDMNeil’s parametersMCEM Data: six distinct walkers

Where do we go from here? Let’s look at some limitations of the model 60 Hz120 Hz

What do we want? Phase Variation x1x1 x2x2 A walk cycle

Branching motions WalkRun

Stylistic variation

Current work: manifold GPs Latent space (x) Data space (y)

Summary GPLVM and GPDM provide priors from small data sets Dependence on initialization, hyperpriors, latent dimensionality Open problems modeling data topology and stylistic variation