Constraint-based Motion Optimization Using A Statistical Dynamic Model Jinxiang Chai Texas A&M University

Goal: generate natural human motion from spatial- temporal constraints Human Animation from User Constraints

Goal: generate natural human motion from spatial- temporal constraints Key framesResulting animation Human Animation from User Constraints

Goal: generate natural human motion from spatial- temporal constraints Key framesResulting animation Key trajectories Resulting animation Human Animation from User Constraints

Applications Shrek Xtreme Martial Arts (Dicsovery channel) The Lord of the Rings Polar Express Friday Night 3D Bowling

Works for various spatial-temporal constraints Challenges: Human Animation Control

Works for various spatial-temporal constraints Human motions are high-dimensional while constraints are not Challenges: Human Animation Control

Works for various spatial-temporal constraints Human motions are high-dimensional while constraints are not People are experts on natural human motion Challenges: Human Animation Control

Problem Formulation θ MAP = argmax θ p(θ|C) Find the most likely human motion θ MAP from user constraints C: (Maximum a posterior)

Problem Formulation θ MAP = argmax θ p(θ|C) Find the most likely human motion θ MAP from user constraints C: = argmax θ p(C|θ) × p(θ) (Bayes rule) (Maximum a posterior)

Problem Formulation θ MAP = argmax θ p(θ|C) Find the most likely human motion θ MAP from user constraints C: = argmax θ p(C|θ) × p(θ) (Bayes rule) How well motion matches Constraints? How natural motion is generated? (Maximum a posterior)

Generate natural human motion from user-defined constraints Algorithm Overview Human motion database Human motion analysis Trajectory optimization Human motion prior p(θ) User-defined constraints C θ MAP θ MAP = argmax θ p(C|θ)p(θ)

[Witkin and Kass SIG88] [Cohen SIG92][Liu et al SIG94] [Fang and Pollard SIG03] Related Work: Physically Based Trajectory Optimization [Safonova et al. SIG04] [Sulejmanpasic and Popovic SIG 05] [Liu and Popovic SIG02] [Popovic and Witkin SIG99]

Reordering motion clip [Lee et al. SIG02, Kovar et al. SIG02, Pullen & Bregler SIG02, Arikan et al. SIG03] Learning model from human motion [Brand & Hertzmann SIG00, Li et al. SIG02] Interpolating motions [Rose et al. CG&A98, Kovar et al. SIG04] Related Work: Animation from Mocap Data

Generate natural motion from a small set of user-defined constraints Algorithm Overview Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints

A prerecorded large and heterogeneous human motion database 12 MX-40 Vicon cameras (120 hz) Full-body movement ( 1 hour ) and facial movement (10 minutes) Human Motion Database

Generate natural motion from user-defined constraints Algorithm Overview Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints

User-defined Constraints Any kinematic constraints throughout the motion Position cons. Orientation cons. Distance cons. Joint angle cons.

User input: Keyframe constraints Body Animation: Example Constraints

User input: Keyframe constraints Continuous trajectory constraints Body Animation: Example Constraints

Startt= 6st=3s User input: Face Animation: Example Constraints

Startt= 6st=3s t=1st=1.5st=3s Face Animation: Example Constraints User input:

Startt= 6st=3s User input: t=1st=1.5st=3s StartWidth of the mouthEnd Face Animation: Example Constraints

Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints Generate natural human motion from user-defined constraints Algorithm Overview

Statistical Dynamic Model Character pose Low-dimensional pose space y t = C x t + D Human motion database Human motion analysis Statistical dynamic model x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction

Model Complexity Character pose Low-dimensional pose space y t = C x t + D Human motion database Human motion analysis Statistical dynamic model x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction Dim(u t ) Dim(x t )

Model Learning Human motion database Human motion analysis Character pose Low-dimensional pose space y t = C x t + D x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction Dynamic model matrices: A 1, …,A m, B, C, D

Model Learning Human motion database Human motion analysis Dynamic model matrices: A 1, …,A m, B, C, D Dynamic system order: m, dimensionality of x t and u t Character pose Low-dimensional pose space y t = C x t + D x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction

Reconstruction error Full-body walking data Facial data X t = A 1 X t-1 +…+A m X t-m +B u t Statistical dynamic model: m = 3, dim(u t ) = 4, error = 0.7 deg m = 2, dim(u t ) = 1, error = 0.1 mm Y t = C X t + D

Forward Simulation y t = C x t + D x t = A 1 x t-1 + A 2 x t-2 + A 3 x t-3 +B u t Dim(u t ) = 4 Full-body movement:

Forward Simulation y t = C x t + D x t = A 1 x t-1 + A 2 x t-2 + A 3 x t-3 +B u t utut ytyt Dim(u t ) = 4 Full-body movement:

x t = A 1 x t-1 + A 2 x t-2 + B u t Forward Simulation utut YtYt Dim(u t ) = 1 y t = C x t + D Facial movement:

Motion Optimization Human motion database Human motion analysis Human motion prior User-defined constraints Generate natural motion from a small set of user-defined constraints Motion optimization

Human Motion Prior The motion prior of generated motion sequence

Human Motion Prior The motion prior of generated motion sequence p(θ)

Human Motion Prior The motion prior of generated motion sequence p(θ) Assume u t is independent each other x t depends on the x t-1,…,x t-m and u t

Human Motion Prior The motion prior of generated motion sequence p(θ) (Chain rule)

Human Motion Prior The motion prior of generated motion sequence Control priorDynamics prior p(θ) (Chain rule)

Generate natural motion from a small set of user-defined constraints Algorithm Overview Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints

Motion Optimization Find the most likely motion θ MAP from user constraints C: θ MAP = argmin θ –ln p(C|θ) - ∑ln p(u t ) - ∑ ln p(x t |x t-1,…,x t-m,u t )

Motion Optimization Find the most likely motion θ MAP from user constraints C: θ MAP = argmin θ –ln p(C|θ) - ∑ln p(u t ) - ∑ ln p(x t |x t-1,…,x t-m,u t ) Prior for control input Prior from statistical dynamic model Constraint term

Motion Optimization Find the most likely motion θ MAP from user constraints C: θ MAP = argmin θ –ln p(C|θ) - ∑ln p(u t ) - ∑ ln p(x t |x t-1,…,x t-m,u t ) Prior for control input Prior from statistical dynamic model Constraint term - Trajectory optimization - Cubic B-splines for x and u - Sequential Quadratic programming - Random initial guess [0...1]

Motion Optimization Initial guess 1 Initial guess x10 4 Iteration number Objective function Initial guess 3

Results: Climbing over an Obstacle User input Output animation

Results: Mickey Walking User input Output animation

Results: Baby Walking User input Output animation

Results: Running User input Output animation

Results: Walking & Jumping User input Output animation

Motion Generalization I Source motion Retargeted motion

Motion Generalization II Source motion Modified motion

Facial Animation I Start t= 6s t=3s User input:

Facial Animation II StartEndt=1st=1.5st=3s User input:

Algorithm Evaluation I: The Importance of Motion Priors Input constraints Inverse Kinematics (IK) Our method IK in PCA space

Algorithm Evaluation II: Different Motion Priors Input key frames Running prior Walking prior Jumping prior

Algorithm Evaluation III: Different Numbers of Constraints Ground truth motion Four key frames (1.1 deg per joint) Two key frames (1.8 deg per joint) Six key frames (0.9 deg per joint) Walking prior

Discussion An alternative for physically based optimization √ does not need physical models √ low-dimensional model and faster convergence √ can generate slow even stylized motion x requires appropriate mocap data x cannot specify dynamic constraints (e.g., mass)

Discussion An alternative for physically based optimization √ does not need physical models √ low-dimensional model and faster convergence √ can generate slow even stylized motion x requires appropriate mocap data x cannot specify dynamic constraints (e.g., mass) Intuitive Interfaces for specifying spatial-temporal constraints?

Conclusions Sparse spatial-temporal constraints Human motion database Human motion analysis Motion optimization Human motion prior Realistic animation or

Realistic human animation Trajectory optimization with statistical models Conclusions Human motion database User-defined constraints Human Motion prior p(C|θ)p(θ) argmax θ p(θ|C)

Realistic human animation Trajectory optimization with statistical models A new compact spatial-temporal representations for human motion data: statistical dynamic models Conclusions Human motion database User-defined constraints Human Motion prior X t = A 1 X t-1 +…+A m X t-m +B u t Y t = C X t + D p(C|θ)p(θ) argmax θ p(θ|C)

Questions?