Quick! We Need a Dancing Duck! Mike Gleicher & The Graphics Gang Department of Computer Sciences University of Wisconsin, Madison

A Tale of Motion Use and Re-Use Can we use retargeting tools to create animation? Spacetime-based retargeting (1998) sounded good, what happened? Hey Mike – what have you been doing for the past 3 years?

What have I been doing? Welcome to Wisconsin! –(yes the snow is up to the mailbox)

Motion Retargeting in Practice Why haven’t the methods taken off? Retargeting solves only one piece of the character animation puzzle The results aren’t as good as we might like The methods are a pain to deal with Maybe motion re-use isn’t the right problem

The Talk Agenda Help! Show you the complete story –Our whole motion re-use pipeline –How we are dealing with issues –Where the old stuff fits in Show you how far we’re getting –And where we might be going Make use of Andy’s nice slides

The Original Plan… Create animation by re-use Easier to beg, borrow, steal, buy, …

Retargeting Compendium A retrospective Some old stuff –Indulge me – it’s 47 seconds Some new examples of the old stuff –Notice what’s different Found examples

What did you just see Found Animation –Found characters on the web –Found motions on the web –Put them together! Animating Characters –Not just skeletons –Simple skinning methods All examples from motion capture

What did you see? Plenty of problems! Many relating to the fact that we have characters not just skeletons We’ve been lucky to get some good motion, but not all of it is great

Motion is Tough! And it’s not just motion capture’s problem Motion Capture Sample every frame –No structure. –No intent. –lots of data! Dense samples can represent rich frequency content Noise, sensor errors Motion Capture Sample every frame –No structure. –No intent. –lots of data! Dense samples can represent rich frequency content Noise, sensor errors Hand Animation Data at Keyframes –Structured? –Remember why? –Maybe lots of data. Good animators create complex frequency content Quality takes work Hand Animation Data at Keyframes –Structured? –Remember why? –Maybe lots of data. Good animators create complex frequency content Quality takes work Hard for academics to obtain quality hand animation

Mocap Pipeline Motion Capture is a misunderstood technology Retargeting is in the “Process” box –Found Motion, so no control of Shoot –Distinction between Process and Apply might be blurred Similar story for hand animation PlanShootProcessApply

Processing Pipeline Define Skeleton Raw Data Clean Marker Data Clean Skeletal Data cleanup identify clean Solve to Skeleton Clean Skeleta l Data Retargeti ng& Control Mapping Animatio n Data Application Define Character Define Character Controls

Why change a perfectly good motion? Animation: bring something to life –means we want to change something about the performance Actor vs. character Restrictions of realism Performers aren't perfect Need for usable data (loops, reference poses,...) Studio is not virtual world Motion re-use –sometimes we are stuck with what we have

So You Want a Dancing Duck. Our motion pipeline Driven by need to use “found” stuff An overview of the process includes: –Gather model and motion –Match model and motion poses –Bind the model to the motion –Tweak the results using constraint-based motion editing techniques

How did we do that? Get Motion (from library) Annotate Constraints Get Character (from library) Devise skeleton for character Compute Adapted Motion LightingRender Convert Skeleton Get from library (stock CD or Web) Our Software Commercial Animation System (Maya or 3D Studio MAX)

Start With a Model. Off the web Created from scratch We just get geometry (a mesh) (Ant model courtesy Hou Soon Ming)

Grab a Motion. Off the web Freshly captured From company moCap archives Everybody has a different skeleton

Size up the Situation. Our chosen character size and motion size are clearly different

How to connect? Find a skeleton for character –Use topology of original motion Find deformation parameters –Skinning –Vertex Weights (since we’re doing SSD) This is backwards! –Pros design controls with characters –Map animation data on those controls

Skeleton Matching But how does the skeleton connect to the character? What are the character’s sizes? Where should the joints go? What about orientations?

Find Character’s Skeleton Not simply a geometric operation –Spine down center or back? –How many joints? –Which d.o.f. where? Skeleton is an abstraction

Skeletons for humans Animals are very complex We understand humans pretty well Copyrighted Image deleted (skeleton and muscles from an anatomy text) Copyrighted Image deleted (functional diagram from An anatomy textbook)

We have favorite abstractions 206 bones, muscles, fat, organs, clothing, … 206 bones, complex joints 53 bones Kinematic joints

This is not how humans work Simple Pin Joint Complex tendon and bone system Representation of complex human structure with varying degrees of simplification

This can’t be automated! (without lots of data) Artistic decisions in how the character’s geometry responds to movement! Need to design the set of controls for moving the geometry –Get the right expressitivity –Get the right response –Get the right ease of use We’re limiting ourselves to given format

Building Skeleton From Scratch? Appears to be very easy The joint coordinate systems may not match – Very Tedious

Rotating Skeleton Into Place? Tedious process Difficult to get certain rotations correct (wrist, shoulders, etc)

Shoulder Example 1. A -90 degree rotation around the Z axis in this case yields the following:

Shoulder Example 2. In this case, a 90 degree rotation around the X axis yields the same result as before:

FindSkel: the skeleton finder Given: Geometry (a mesh in “rest pose”) Desired skeleton Our conventions Find: The right skeleton parameters The “binding pose” relative to the “zero pose”

Our Approach. Instead of a skeleton, create a point cloud of joints in the mesh Easy for the user Works with nearly any 3d modeling package out there

This is about usability Points are easy to specify Take whatever the user gives us –Guess the rest –Symmetry –Conventions –Heuristics –Assumptions Pass 1: make a full cloud of points Pass 2: find skeleton and pose

Rotate the Motion’s Skeleton. Rotate each bone in the motion to match model point cloud information Relies heavily on knowledge of humanoid kinematics Maintains coordinate systems

Our Approach - Prepare Motion. Need to pose the motion in a way which allows intuitive adjustments to match model’s bind pose Zombie (or Frankenstein) pose

Limb Forward Direction Hands, elbows, feet, and knees especially problematic Feet have a free guide – the toes point forward

Limb Forward Direction Contd. Hands have no forward vector Easy to add forward point in cloud, but somewhat unintuitive

Motion Simplification Contd. When rotating shoulder into position, it is very difficult to take all rotational degrees of freedom into account We only allow for rotation around two axes

Non-Humanoid Motion. Unless you know at least some of the complexities behind the creature’s degrees of freedom, all bets are off

Finally We Can Bind! Despite all the gotcha’s, our technique still works With our matching skeletons, any bone-based skinning technique can be applied Skinning depends on the skeleton

Now Are We Done? Most certainly not! The model is not interacting with its environment as it should Its feet are nowhere near the ground

Maybe that paper wasn’t useless… I knew we needed Retargeting! Model’s root is still following old root translations A simple translation to the floor will yield sliding feet and too much bounce

How Do We Fix This Mess? Constraint-based motion editing to the rescue! Finally something from the paper!

Constraints and Our Example. In our case, adding constraints where the motion’s feet hit the floor would probably be sufficient As seen before, our new mesh is clearly not satisfying its foot-hits-floor constraints

Solving for the Constraints. A quick solve for the constraints yields our model walking nicely along the floor

What’s wrong with the story? Philosophically misguided –Force old controls onto new character –Requires convention in controls –Randomly chose hard to use controls Ignores MoCap Wisdom –Begin with the ends in mind –Solve problems earlier, not later Dealing with clips

Processing Pipeline Define Skeleton Raw Data Clean Marker Data Clean Skeletal Data cleanup identify clean Solve to Skeleton Clean Skeleta l Data Retargeti ng& Control Mapping Animatio n Data Application Define Character Define Character Controls

What’s wrong with the story? Results aren’t as good as we’d like –Get better constraints Solver is too hard to implement –Find new solution paradigm High-level control still hard –Build abstracted representations Off-line –See upcoming ToG paper Now for the quick tour of the first 3

Better Constraints/Objectives Feature of constraint-based approach –Just need to find those right ones –Dates back to Witkin&Kass ’88 –We still don’t have the grace functional I expected to do more of this –Lots of mileage from using a few constraints well –Figure out how to use what you have

Want Better Results? Devise Better Constraints! Self-intersection Higher-level properties Style ???? Form ???? ????

Collision Constraints Get your hands out of your pockets!

Constraints don’t fix everything… This arm does well This arm doesn’t go through the body, but…

Wins and Losses of Spacetime Fast, practical –Linear complexity? Solves real problems Flexibility in: –Spatial Constraints –Objective Functions ? –Temporal Constraints ? Widely applicable Nice results Hard to implement Poor integration Off-line No guarantees Spatial constraints not enforced Flexibility not exploited Rely on constraints

Is there an alternative? Need to deal with spatial and temporal constraints Don’t want the messiness of “whole motion” computation Handle spatial and temporal constraints separately! Per-Frame IK Plus Filter –Lee & Shin ’99 -- first version –Gleicher ’00 & ’01 -- generalization

Per- Frame IK + Filter (PFIK+F) IK per frame to solve spatial constraints –But this messes up temporal constraints Filter changes to enforce temporal constraints –But this messes up spatial constraints Iterate until converges, or …

PFIK+F vs. Spacetime Fast, practical Solves real problems Flexibility in: –Spatial Constraints –Objective Functions (?) –Temporal Constraints (?) Widely applicable Nice results Yes! (requires fast IK) Yes! –(depends on IK) –(limited, unexplored) Yes! Um, it’s a matter of taste, and IK quality

PFIK+F vs. Spacetime Use standard pieces! Choice in which last Solve spatial constraints last Need good IK Hard to implement Poor integration Off-line No guarantees Spatial constraints not enforced Flexibility not exploited Rely on constraints

My PFIK+F solver Use pieces I have lying around Non-linear optimizing solver for IK FIR linear filters for temporal constraints Propagation of results for temporal coherence

Downsides of PFIK+F? No global decisions –Doesn’t handle “don’t cares” as well –Order dependence –No interframe constraints Reliance on quality of IK solver Not necessarily faster (or slower)

Higher Level Control Need to find abstractions of motion Can’t just work with details –Even constraints aren’t enough better Path Editing (a factoring approach) –Gleicher ’01 Motion Graphs –Kovar & Gleicher ‘??

The Idea Original MotionNew Path

What is Path Editing? Factor Motion into Path And Detail Edit Path as Needed (use curve editing tools) Apply Detail to new Path

Why Edit Paths? (because previous work can’t do it) Can’t Capture All Possibilities Can’t Synthesize All Styles Can’t Simulate Complex Motions Current Editing Tools Don’t Apply Need tools for sculpting motion INTERACTIVELY! Need easier to implement tools!

How to Edit a Path Determine what the path is –Easy to manipulate, conceptual representation Factor motion into path and detail Edit the path Put path and detail back together Clean up the details

Motion Use Ideal Capture a large corpus of motion Everything you need is there –Somewhere – but where –Maybe it needs some adaptation Have some specification of needs Generate motion from this

Current approaches Move Trees –Specially designed motions that connect –Requires planning to create –Requires tricks to fit together –Games use 1000s (yes thousands) of motions! –Graphs/motions done manually Style Machines –???

Goal Generate motion, on demand, using an abstracted representation Corpus of Motions Preprocessing Abstracted Representation Requests

First (working) Example A few notable failures –Best Linear Unbiased Estimators –Dense tables Simple seems to work –State matching –Rigid factoring (timescaling soon) Graph structures are useful –Today: random walks –Tomorrow: ???

Tricks up our sleeves General Transitions are hard –Can be arbitrarily hard –Can be really easy (if similar motions) –Identify when basic techniques work Factored representations –More things to match –More ways to apply things

Thanks! UW Graphics Gang Mainframe Entertainment, House of Moves Studios, Digital Domain This research is supported by: Microsoft, Intel, Pixar, IBM NSF Career Award