Animating (human) motion Presented by: –Yoram Atir –Simon Adar

Applications of computer animation Movies Advertising Games Simulators …

General goals of the work presented -New methods aimed to save time/money/skills needed. -Study motion (texture).

Agenda -Basic concepts -Motion Synthesis/texture using motion capture -Physics/Biomechanics Motion Synthesis -Cartoon Motion Retargeting.

Basic Concepts Basic concepts Animation world (3D) Skeletal model representation Model positioning Keyframes Motion capture Frequency bands Correlations

Basic Concepts 3D animation world -(Human) model is animated in Object space -Animated model projected into “global” space -Camera is placed and rotated -Perspective is set -Other…

Basic Concepts Skeletal representation -Each model has its own Default Pose -DOF’s – joint angles/translations relative to Default Pose -Hierarchical (tree) skeletal representation of model Picture from Lecture in Computer Graphics course Department of computer science University of Washington

Basic Concepts Creating motion -Skeletal variations between frames -Overall rotation/Translation between frames -Correlate. General Problem: A LOT of work due to the large number of DOFS & high frame rate

Basic Concepts Figure positioning -Forward kinematics (simplified): Figure positioning by joint data specification. Problem: -Tedious trial and error.

Basic Concepts Figure positioning Inverse kinematics (simplified) -Joint data is acquired by solving for the final position -In general, This is an optimization problem with a large system of variables and constraints -Problems often are expressed as minimization problems, and solved using standard algorithms (gradient decent etc). -Usually, infinite number of possible solutions. -A “good” solution has to be more than “feasible” -Often one is obtained by embedding specific knowledge as additional constraints, and/or -Using Inverse kinematics as a part of a specific solution.

Basic Concepts Basic methods for saving labor Motion capture KeyFrames

Basic Concepts Keyframes –Specifying only part of DOFs and frames –Computer interpolation between them Problem: “smooth” interpolation looks unreal There are methods to apply “specific noise” –Term has historical roots

Basic Concepts Motion capture –Acquired from “live action” –Copied onto animated character Problem: Hard to adapt. “Motion Editing” – methods to adapt mocap –Done in studios –Mocap libraries exist

Basic Concepts Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control

Basic Concepts Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive

Basic Concepts Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive Detail hard

Basic Concepts Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive Detail hard Many DOF

Basic Concepts Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive Detail hard Many DOF Detail easy

Basic Concepts Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive Detail hard Many DOF Detail easy All DOF

Basic Concepts Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive No control Detail hard Many DOF Detail easy All DOF

Basic Concepts Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive No control Not intuitive Detail hard Many DOF Detail easy All DOF

Keyframe Data vs. Motion Capture Data

Basic Concepts Frequency Bands Right flatRight toeLeft flatLeft toe

Basic Concepts Frequency Bands Simplifies the form of the data –Low frequency Variations: Large scale motions. –Higher frequency variations: individual “noise” / Jitter Both are important to preserve in order to capture the essence of motion

Basic Concepts Correlations Joints angle/translation data is related to each other Joint angles are correlated over time Correlation “plot” is –(somewhat) Specific to the type of motion –Carries “personality” information (style)

More information… INTRODUCTION TO COMPUTER ANIMATION – Rick parent Splines Hash Inc - Animation software (Movies, tutorials…) Google…

Motion Capture Assisted Animation: Texturing and Synthesis Kathy Pullen Chris Bregler SIGGRAPH 2002 Motion Capture Assisted Animation – Pullen/Bregler

Agenda -Basic concepts -Motion Synthesis/texture using motion capture -Physics/Biomechanics Motion Synthesis -Cartoon Motion Retargeting

Goal: Motion Capture Assisted Animation Create a method that allows an artist low- level control of the motion Combine the strengths of keyframe animation with those of mocap Create a method that allows an artist low- level control of the motion Combine the strengths of keyframe animation with those of mocap Motion Capture Assisted Animation – Pullen/Bregler

Goal: Motion Capture Assisted Animation “Sketch” an animation by keyframing Animate only a few degrees of freedom Set few keyframes “Enhance” the result with mocap data Synthesize missing degrees of freedom Texture keyframed degrees of freedom “Sketch” an animation by keyframing Animate only a few degrees of freedom Set few keyframes “Enhance” the result with mocap data Synthesize missing degrees of freedom Texture keyframed degrees of freedom Motion Capture Assisted Animation – Pullen/Bregler

What is a Motion Texture? Every individual’s movement is unique Synthetic motion should capture the texture To “texture” means to add style to a pre- existing motion Technically, texturing is a special case of synthesis

Goal: Motion Capture Assisted Animation Blue = Keyframed Purple = Textured/Synthesized Motion Capture Assisted Animation – Pullen/Bregler

How an Animator Works A few degrees of freedom at first Not in detail Fill in detail with more keyframes later A few degrees of freedom at first Not in detail Fill in detail with more keyframes later Motion Capture Assisted Animation – Pullen/Bregler

The Method in Words Choose degrees of freedom to drive the animation Compare these degrees of freedom from the keyframed data to mocap Find similar regions Look at what the rest of the body is doing in those regions Put that data onto the keyframed animation Choose degrees of freedom to drive the animation Compare these degrees of freedom from the keyframed data to mocap Find similar regions Look at what the rest of the body is doing in those regions Put that data onto the keyframed animation Motion Capture Assisted Animation – Pullen/Bregler

Choices the Animator Must Make 1.Which DOF to use as matching angles 2.Which DOF to texture, which to synthesize 3.Which frequency band to use in matching 4.How many frequency bands to use in texturing 5.How many matches to keep 6.How many best paths to keep 1.Which DOF to use as matching angles 2.Which DOF to texture, which to synthesize 3.Which frequency band to use in matching 4.How many frequency bands to use in texturing 5.How many matches to keep 6.How many best paths to keep Motion Capture Assisted Animation – Pullen/Bregler

Before Beginning: Choose Matching Angles Left Hip x Left Hip y Left Hip z Left Knee x Left Knee y Left Knee z Left Ankle x Left Ankle y Left Ankle z Left Ball x Left Ball y Left Ball z Right Hip x Right Hip y Right Hip z Right Knee x Right Knee y Right Knee z Right Ankle x Right Ankle y Right Ankle z Right Ball x Right Ball y Right Ball z Root x trans Root y trans Root z trans Root x rot Root y rot Root z rot Spine1 x Spine1 y Spine1 z Spine2 x Spine2 y Spine2 z Spine3 x Spine3 y Spine3 z Neck x Neck y Neck z Head x Head y Head z Head Aim x Head Aim y Head Aim z Left Clavicle x Left Clavicle y Left Clavicle z Left Shoulder x Left Shoulder y Left Shoulder z Left Elbow x Left Elbow y Left Elbow z Left Wrist x Left Wrist y Left Wrist z Right Clavicle x Right Clavicle y Right Clavicle z Right Shoulder x Right Shoulder y Right Shoulder z Right Elbow x Right Elbow y Right Elbow z Right Wrist x Right Wrist y Right Wrist z Time

Matching Angles Drive the Synthesis Left Hip x Left Hip y Left Hip z Left Knee x Left Knee y Left Knee z Left Ankle x Left Ankle y Left Ankle z Left Ball x Left Ball y Left Ball z Right Hip x Right Hip y Right Hip z Right Knee x Right Knee y Right Knee z Right Ankle x Right Ankle y Right Ankle z Right Ball x Right Ball y Right Ball z Root x trans Root y trans Root z trans Root x rot Root y rot Root z rot Spine1 x Spine1 y Spine1 z Spine2 x Spine2 y Spine2 z Spine3 x Spine3 y Spine3 z Neck x Neck y Neck z Head x Head y Head z Head Aim x Head Aim y Head Aim z Left Clavicle x Left Clavicle y Left Clavicle z Left Shoulder x Left Shoulder y Left Shoulder z Left Elbow x Left Elbow y Left Elbow z Left Wrist x Left Wrist y Left Wrist z Right Clavicle x Right Clavicle y Right Clavicle z Right Shoulder x Right Shoulder y Right Shoulder z Right Elbow x Right Elbow y Right Elbow z Right Wrist x Right Wrist y Right Wrist z Time

Motion Capture Data Left Hip x Left Hip y Left Hip z Left Knee x Left Knee y Left Knee z Left Ankle x Left Ankle y Left Ankle z Left Ball x Left Ball y Left Ball z Right Hip x Right Hip y Right Hip z Right Knee x Right Knee y Right Knee z Right Ankle x Right Ankle y Right Ankle z Right Ball x Right Ball y Right Ball z Root x trans Root y trans Root z trans Root x rot Root y rot Root z rot Spine1 x Spine1 y Spine1 z Spine2 x Spine2 y Spine2 z Spine3 x Spine3 y Spine3 z Neck x Neck y Neck z Head x Head y Head z Head Aim x Head Aim y Head Aim z Left Clavicle x Left Clavicle y Left Clavicle z Left Shoulder x Left Shoulder y Left Shoulder z Left Elbow x Left Elbow y Left Elbow z Left Wrist x Left Wrist y Left Wrist z Right Clavicle x Right Clavicle y Right Clavicle z Right Shoulder x Right Shoulder y Right Shoulder z Right Elbow x Right Elbow y Right Elbow z Right Wrist x Right Wrist y Right Wrist z Time

Overview Steps in texture/synthesis method Frequency analysis Matching Path finding Joining Steps in texture/synthesis method Frequency analysis Matching Path finding Joining Motion Capture Assisted Animation – Pullen/Bregler

In the following series of slides: Hip angle = matching angle Spine angle = angle being synthesized In the following series of slides: Hip angle = matching angle Spine angle = angle being synthesized Example Motion Capture Assisted Animation – Pullen/Bregler

Frequency Analysis: Break into Bands Motion Capture Assisted Animation – Pullen/Bregler

Frequency Time Band-pass decomposition of matching angles Keyframed DataMotion Capture Data Frequency Analysis Motion Capture Assisted Animation – Pullen/Bregler

Frequency Time Keyframed DataMotion Capture Data Chosen low frequency band Frequency Analysis Motion Capture Assisted Animation – Pullen/Bregler

Keyframed DataMotion Capture Data Hip angle data (a matching angle) Chosen Low Frequency Band Motion Capture Assisted Animation – Pullen/Bregler

Keyframed DataMotion Capture Data Making Fragments Break where first derivative changes sign Motion Capture Assisted Animation – Pullen/Bregler

Keyframed DataMotion Capture Data Making Fragments Step through fragments one by one Motion Capture Assisted Animation – Pullen/Bregler

Matching Keyframed Fragment Motion Capture Assisted Animation – Pullen/Bregler

Matching Keyframed Fragment Motion Capture Data Motion Capture Assisted Animation – Pullen/Bregler

Matching Keyframed Fragment Motion Capture Data Motion Capture Assisted Animation – Pullen/Bregler

Matching Compare to all motion capture fragments Angle in degrees Time Keyframed Mocap

Matching Resample mocap fragments to be same length Angle in degrees Time Keyframed Mocap

Matching Using some metric on all matching angles and on their first derivatives: Keep the K closest matches Angle in degrees Time Keyframed Mocap

Matching Keyframed Fragment Motion Capture Data Motion Capture Assisted Animation – Pullen/Bregler

Matching Keyframed Fragment Motion Capture Data Close Matches Motion Capture Assisted Animation – Pullen/Bregler

Matching Hip Angle (Matching Angle) Spine Angle (For Synthesis) Motion Capture Assisted Animation – Pullen/Bregler

Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized Motion Capture Assisted Animation – Pullen/Bregler

Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized Motion Capture Assisted Animation – Pullen/Bregler

Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized Motion Capture Assisted Animation – Pullen/Bregler

Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized Motion Capture Assisted Animation – Pullen/Bregler

Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized Motion Capture Assisted Animation – Pullen/Bregler

Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized Motion Capture Assisted Animation – Pullen/Bregler

Possible Synthetic Spine Angle Data Angle in degrees Time

Path Finding Angle in degrees Time We would like to: Use as much consecutive fragments as possible Stay as close as possible to best fit

Path Finding Angle in degrees Time

Path Finding Angle in degrees Time

Path Finding Angle in degrees Time

Path Finding Angle in degrees Time

Joining Angle in degrees Time

Enhancing Animations: Texturing and Synthesis Keyframed Textured Synthesized Not keyframed

Texturing Synthesize upper frequency bands Motion Capture Assisted Animation – Pullen/Bregler

Texturing Band-pass decomposition of keyframed data Frequency Time

Texturing Synthesize upper frequency bands Frequency Time

Walking animations Texturing and Synthesis Keyframed Sketch

Walking animations Texturing and Synthesis Motion Capture Data Two different styles of walk

Walking animations Texturing and Synthesis Enhanced Animation Upper body is synthesized Lower body is textured

Otter Animations: Texturing Keyframed data

Otter Animations: Texturing Textured animation

Dance Animations: Texturing and Synthesis Lazy Keyframed Sketch

Dance Animations: Texturing and Synthesis Motion Capture Data

Dance Animations: Texturing and Synthesis Enhanced Animation Blue = Keyframed Purple = Textured/Synthesized

Dance Animations: Texturing Keyframed Sketch With More Detail

Dance Animations: Texturing Textured Animation Blue = Keyframed Purple = Textured

Summary of the Method Keyframed data Mocap Data Keyframed Data Mocap DataPossible Synthetic Data Matching Angles Sketch + Mocap Frequency Analysis Matching Path FindingJoining Enhanced Animation

Choices the Animator Must Make 1.Which DOF to use as matching angles 2.Which DOF to texture, which to synthesize 3.Which frequency band to use in matching 4.How many frequency bands to use in texturing 5.How many matches to keep 6.How many best paths to keep 1.Which DOF to use as matching angles 2.Which DOF to texture, which to synthesize 3.Which frequency band to use in matching 4.How many frequency bands to use in texturing 5.How many matches to keep 6.How many best paths to keep

Conclusions and Applications Appropriate for an artist interested in a very particular style of motion The artist may have a relatively small motion capture set of that style The artist may want precise control over parts of the motion Appropriate for an artist interested in a very particular style of motion The artist may have a relatively small motion capture set of that style The artist may want precise control over parts of the motion

Conclusions and Further Work Direct incorporation of hard constraints Fundamental units of motion Direct incorporation of hard constraints Fundamental units of motion

For more info... Special Thanks to: Reardon Steele, Electronic Arts

Agenda -Basic concepts -Motion Synthesis/texture using motion capture -Physics, Biomechanics Motion Synthesis -Cartoon Motion Retargeting

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Agenda Motivation Scope (very dynamic motions, constrains are very important, automatic constrain detection) Overview of the process Constraint and stage detection. Transition pose generation. Momentum control Objective function generation Putting it all together

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Motivation Generate rapid prototyping of realistic character motion Avoid simulated human models, that are very complex, and don’t always look realistic

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Scope Highly dynamic movement such as jumping, kicking, running, and gymnastics. Less energetic motions such as walking or reaching will not work well in this framework

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Overview of the process Motion sketch Character description Motion DB User interaction Constraint & phase detection Transition pose synthesis Objective functions OptimizationAnimation Momentum control

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Overview of the process The objective is to transforms simple animations into realistic character motion by applying laws of physics and the biomechanics domain The unknowns are: values of joint angles and parameters of angular and linear momentum

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Overview of the process Motion sketch Character description Motion DB User interaction Constraint & phase detection Transition pose synthesis Objective functions OptimizationAnimation Momentum control

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Constraint and stage detection Each input sequence has two parts: –The part that needs to be improved –The part that needs to kept intacked Automatically extract the positional and sliding constrains

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Positional constraint detection A positional constraint fixes a specific point on the character to a stationary location for a period of time We need to find if all these points lie on a line, plane In an articulated character we find the constraints on each body part

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Positional constraint detection The algorithm looks for fixed points (point, line, plane)

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Positional constraint detection The algorithm looks for fixed points (point, line, plane)

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Sliding constraints

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Overview of the process Motion sketch Character description Motion DB User interaction Constraint & phase detection Transition pose synthesis Objective functions OptimizationAnimation Momentum control

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Transition pose generation A transition pose separates constrained and unconstrained stages. Two possibilities: –We ask the animator to draw the transition poses –We have an estimator to suggest a transition pose

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Transition Pose Estimator DB contains examples of different motions The input of that DB are the motion parameters like: flight distance, flight height, takeoff angle, landing angle, spin angle.. The DB has a simplified representation of the transition poses by three COM’s We use IK to obtain the full character’s pose from those three COM’s The KNN - K nearest neighbor algorithm The pose estimator predicts the candidate pose by interpolating the KNN with the weights that describe the similarity to the input.

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Overview of the process Motion sketch Character description Motion DB User interaction Constraint & phase detection Transition pose synthesis Objective functions OptimizationAnimation Momentum control

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Momentum control Transition poses constrain the motion at few key points of the animation Dynamic constraints ensure realistic motion of each segment Linear and angular momentum give us these dynamic constraints

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Momentum during unconstrained and constrained stages linear momentum - During “flight” the only force is gravity Angular momentum - During “flight” there is no change in Angular momentum During “ground” stage we avoid computing the momentums and use empirical characteristics

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Overview of the process Motion sketch Character description Motion DB User interaction Constraint & phase detection Transition pose synthesis Objective functions OptimizationAnimation Momentum control

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Objective functions There are three Objective functions, the basic idea behind them is power consumption –Minimum mass displacement –Minimal velocity of DOFs –Static balance

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Overview of the process Motion sketch Character description Motion DB User interaction Constraint & phase detection Transition pose synthesis Objective functions OptimizationAnimation Momentum control

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Putting it all together Environment constraints (Ce) Transition pose constraints (Cp) Momentum constraints (Cm) Q are character’s DOFs subject to

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Overview of the process Motion sketch Character description Motion DB User interaction Constraint & phase detection Transition pose synthesis Objective functions OptimizationAnimation Momentum control

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Some Results Wide variety of figures: male, female, child 51 DOFs The body dimensions and mass distribution is taken from biomechanics literature In some of the cases the animator selects the body parts to be constraints The animator can change relative timing between each phase The optimization was solved by using SNOPT a general nonlinearly-constrained optimization package The optimization time depends on the duration of the animation All of the simple animation took less than five minutes to sketch For all examples the synthesis process took less than five minutes

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Broad jump Only 3 keyframes at takeoff, peak and landing

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Running The angular momentum constraint creates a counter- body movement by the shoulders and arms to counteract the angular momentum generated by the legs. Keyframing 7 DOFs

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Hopscotch Each hop requires 3 keyframes and has fewer than 7 DOFs

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Handspring There were no handstands within the DB so the user had to modify the result

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" High-bar Two constraints stages: the bar and ground

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Karate kick A second synthesis add a keyframe in the peak

C. Karen Liu & Zoran Popovi´c "Synthesis of Complex Dynamic Character Motion from Simple Animations" Twist jumps

Agenda -Basic concepts -Motion Synthesis/texture using motion capture -Physics/Biomechanics Motion Synthesis -Cartoon Motion Retargeting

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Cartoon motion capture & retargeting What is Cartoon Capture & Retargeting Motivation Scope Challenges Modeling Cartoon motion cartoon capture –contour capture –video capture Retargeting Additional constrains & post processing Performance

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” What is Cartoon Capture & Retargeting Cartoon Capture –Track the motion From 2D Animation –Represent the motion & save Retargeting –Translate the motion representation to another output media

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Digitized video Key shapes Cartoon capture Output corresponding key shapes Motion representation retargeting Output video Cartoon motion capture & retargeting scheme

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Motivation

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Motivation (cont.) Animation has high exaggeration drawings & motion, which cannot be extracted from the motion capture Highly trained Animators are rare Fast motion style creation

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Scope The pink area

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Challenges - Capture Cartoon characters have no markers Standard skeletal model-based capture techniques are not suitable The low frame rate makes tracking difficult

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Challenges - Retargeting Current retargeting are based on skeletal models Retarget 2D information to 3D

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Modeling Cartoon motion Digitized video Key shapes Cartoon capture Output corresponding key shapes Motion representation retargeting Output video

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Modeling Cartoon motion Two types of deformations –Affine deformation –Key shape deformation

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Affine Deformation Affine parameters y x

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Key-Shape Deformation S k are the key shapes

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Modeling Cartoon motion In total there are 6+K variables that represent the motion

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Cartoon motion capture Digitized video Key shapes Cartoon capture Output corresponding key shapes Motion representation retargeting Output video

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Cartoon motion capture contour capture: the input is a sequence of contours video capture: the input is the video sequence

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” contour capture Two step minimization: –Find Affine parameters –Find Key-Shape weights Iterate

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Video capture The output is the same as in contour capture The input is the video sequence instead of the contour sequence

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Retargeting Digitized video Key shapes Cartoon capture Output corresponding key shapes Motion representation retargeting Output video

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Retargeting For each Input key-shape an Output key- shape is drawn.

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Retargeting Process Key shapes Interpolation Apply Affine transformation From motion capture Retargeted media Retarget Motion capture

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Examples

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Additional constrains & post processing Undesirable effects may still appear Determine constraints that force the character go through certain position at certain time Apply ad-hoc global transformation that fulfill these constraints

C.Bregler, L.Loeb, E.Chuang, H.Deshpande; "Turning to the Masters: Motion Capturing Cartoons” Performance Quantative performance wasn’t mentioned The more complex the motion of the character is, the more key-shapes are needed Many of the animations contain jitter, but the overall exaggerated motion dominates