March 1, 20021ICT Virtual Human Workshop HUMAN FIGURE ANIMATION Norman I. Badler Center for Human Modeling and Simulation University of Pennsylvania Philadelphia, PA USA Norman I. Badler Center for Human Modeling and Simulation University of Pennsylvania Philadelphia, PA USA

March 1, 20022ICT Virtual Human Workshop Outline Generative Methods and Tradeoffs Generative Methods and Tradeoffs Re-use and Retargetting Re-use and Retargetting Parametric Models and Controls Parametric Models and Controls Real-time or Offline? (i.e., Hard problems) Real-time or Offline? (i.e., Hard problems) Model Availability Model Availability Body Requirements and Assumptions Body Requirements and Assumptions Object – Virtual Human Interactions Object – Virtual Human Interactions Generative Methods and Tradeoffs Generative Methods and Tradeoffs Re-use and Retargetting Re-use and Retargetting Parametric Models and Controls Parametric Models and Controls Real-time or Offline? (i.e., Hard problems) Real-time or Offline? (i.e., Hard problems) Model Availability Model Availability Body Requirements and Assumptions Body Requirements and Assumptions Object – Virtual Human Interactions Object – Virtual Human Interactions

March 1, 20023ICT Virtual Human Workshop The “Realism Ceiling” for Human Models Visual Realism Time to createBehavioral Realism RealismSpecialeffectsInanimateobjectsLife-likevirtualhumans Real-timeAgents

March 1, 20024ICT Virtual Human Workshop What Approaches Push the Curve Toward More Realism? Visual Realism Inanimateobjects MotionCapture Parameterization (face, gait, gesture) Life-likevirtualhumans Time to createBehavioral Realism Realism Real-time

March 1, 20025ICT Virtual Human Workshop Why is Realism Still Hard? Visual Realism Inanimateobjects MotionCapture Parameterization (face, gait, gesture) Difficult to generalize What are the “right” parameters? Life-likevirtualhumans Time to createBehavioral Realism Realism Real-time

March 1, 20026ICT Virtual Human Workshop Procedures Use models; Generalize control; Fast Find models; Specific code; Integration w/others Motion Capture Natural;Nuance;Physics Actors; Acting; Motion specificity; Retargetting Dynamics (Physics- based simulation)Physics; Apparent interlimb coordination; Free fall (rag doll) Control laws; Model complexity; Retargetting Collision response Trade-Offs Advantages Disadvantages

March 1, 20027ICT Virtual Human Workshop Human Movement Categories Voluntary (task-oriented, reach, look-at) Voluntary (task-oriented, reach, look-at) Involuntary (breathing, balancing, blinking) Involuntary (breathing, balancing, blinking) Subconscious Subconscious – Low level motor functions (fingers, legs, lips) – Communicative acts (facial expressions, limb gestures, body posture) What generative techniques work for these? Voluntary (task-oriented, reach, look-at) Voluntary (task-oriented, reach, look-at) Involuntary (breathing, balancing, blinking) Involuntary (breathing, balancing, blinking) Subconscious Subconscious – Low level motor functions (fingers, legs, lips) – Communicative acts (facial expressions, limb gestures, body posture) What generative techniques work for these?

March 1, 20028ICT Virtual Human Workshop Human Movement Implementations Voluntary (procedures [IK], mocap) Voluntary (procedures [IK], mocap) Involuntary (procedures [physiological or dynamic models]) Involuntary (procedures [physiological or dynamic models]) Low level motor functions (procedures, mocap, dynamics) Low level motor functions (procedures, mocap, dynamics) Communicative acts (mocap, procedures) Communicative acts (mocap, procedures) category X technique = {animation paper} Voluntary (procedures [IK], mocap) Voluntary (procedures [IK], mocap) Involuntary (procedures [physiological or dynamic models]) Involuntary (procedures [physiological or dynamic models]) Low level motor functions (procedures, mocap, dynamics) Low level motor functions (procedures, mocap, dynamics) Communicative acts (mocap, procedures) Communicative acts (mocap, procedures) category X technique = {animation paper}

March 1, 20029ICT Virtual Human Workshop MoCap Retargetting Joint angles don’t neatly map to different size or structured figures. Joint angles don’t neatly map to different size or structured figures. Need to maintain constraints (IK) to avoid feet slipping and sliding. Need to maintain constraints (IK) to avoid feet slipping and sliding. Need to add IK to change reach and look-at direction. Need to add IK to change reach and look-at direction. Moving interactions may fit but change dynamics (e.g., figure skating pairs). Moving interactions may fit but change dynamics (e.g., figure skating pairs). Joint angles don’t neatly map to different size or structured figures. Joint angles don’t neatly map to different size or structured figures. Need to maintain constraints (IK) to avoid feet slipping and sliding. Need to maintain constraints (IK) to avoid feet slipping and sliding. Need to add IK to change reach and look-at direction. Need to add IK to change reach and look-at direction. Moving interactions may fit but change dynamics (e.g., figure skating pairs). Moving interactions may fit but change dynamics (e.g., figure skating pairs).

March 1, ICT Virtual Human Workshop Parametric Models (Procedures) IK IK Gait Gait Faces Faces Eyes Eyes Muscles (Deformations) Muscles (Deformations) Gestures (BEAT/MagiCster) Gestures (BEAT/MagiCster) EMOTE EMOTE IK IK Gait Gait Faces Faces Eyes Eyes Muscles (Deformations) Muscles (Deformations) Gestures (BEAT/MagiCster) Gestures (BEAT/MagiCster) EMOTE EMOTE

March 1, ICT Virtual Human Workshop IK at Work Eye view; Note attention Control Instructions translated to parameterized actions.

March 1, ICT Virtual Human Workshop Walking (Sun and Metaxas) Gait from parameterized biomechanical model, original movements from motion capture and interpolated based on terrain angle. movie Gait from parameterized biomechanical model, original movements from motion capture and interpolated based on terrain angle. movie

March 1, ICT Virtual Human Workshop Gesture Control: EMOTE EMOTE: A real-time motion modification system. EMOTE: A real-time motion modification system. Defines qualities of movement with 8 parameters. Defines qualities of movement with 8 parameters. Based on Effort and Shape components of Laban Movement Analysis. Based on Effort and Shape components of Laban Movement Analysis. EMOTE: A real-time motion modification system. EMOTE: A real-time motion modification system. Defines qualities of movement with 8 parameters. Defines qualities of movement with 8 parameters. Based on Effort and Shape components of Laban Movement Analysis. Based on Effort and Shape components of Laban Movement Analysis.

March 1, ICT Virtual Human Workshop Inputs EMOTE Output Key Poses End Effector Goals Motion Capture Procedures Frame Rate Poses 4 Efforts 4 Shapes InverseKinematicsInterpolation

March 1, ICT Virtual Human Workshop Motion Qualities are Important Movements with EMOTE qualities give insight into the agent’s cognitive state. Movements with EMOTE qualities give insight into the agent’s cognitive state. When EMOTE qualities spread from limbs to body, movements appear more sincere. When EMOTE qualities spread from limbs to body, movements appear more sincere. Movements with EMOTE qualities give insight into the agent’s cognitive state. Movements with EMOTE qualities give insight into the agent’s cognitive state. When EMOTE qualities spread from limbs to body, movements appear more sincere. When EMOTE qualities spread from limbs to body, movements appear more sincere.

March 1, ICT Virtual Human Workshop Don’t Animators Know This? Good key pose animators know this and use character deformation to great advantage; EMOTE allows this concept to be applied parametrically (under program or interactive control) to articulated (jointed) figures. Good key pose animators know this and use character deformation to great advantage; EMOTE allows this concept to be applied parametrically (under program or interactive control) to articulated (jointed) figures.

March 1, ICT Virtual Human Workshop An Actor Trying to Act (Apologies to W. Shakespeare) We’ll manually match these movements in our virtual human model, then vary the motion qualities via EMOTE.

March 1, ICT Virtual Human Workshop Manually Matching the Motions

March 1, ICT Virtual Human Workshop The Actor with Neutral Efforts (A Politician?)

March 1, ICT Virtual Human Workshop Actor with Less Rising Shape (Not quite as excited?)

March 1, ICT Virtual Human Workshop Moving the Shapes Inward (Woody Allen?)

March 1, ICT Virtual Human Workshop With Light and Sustained Efforts (More solemn and serious?)

March 1, ICT Virtual Human Workshop Without Torso Movements (Used Car Salesman?)

March 1, ICT Virtual Human Workshop Hit the ball … forcefully. …softly. Manner Variants (adverbs): HIT (with real dynamic response)

March 1, ICT Virtual Human Workshop DI-Guy TM – Being combined with EMOTE (mocap + parameters)

March 1, ICT Virtual Human Workshop Procedural Eye Movement Model from Human Performance Data Source Eyes fixed ahead Eyes moved by statistical model Full MPEG-4 face

March 1, ICT Virtual Human Workshop Important Problems to Solve in Real- Time Constrained reach planning/execution Constrained reach planning/execution Coordination and consistency Coordination and consistency Dynamics on demand (integrated with constraints/goals) Dynamics on demand (integrated with constraints/goals) Individuation (and its specification) Individuation (and its specification) Collision response Collision response Deformable surface Deformable surface Clothing Clothing Constrained reach planning/execution Constrained reach planning/execution Coordination and consistency Coordination and consistency Dynamics on demand (integrated with constraints/goals) Dynamics on demand (integrated with constraints/goals) Individuation (and its specification) Individuation (and its specification) Collision response Collision response Deformable surface Deformable surface Clothing Clothing

March 1, ICT Virtual Human Workshop E.g.: Real-Time Constrained Reach Planning

March 1, ICT Virtual Human Workshop Model Availability “Standard” H-Anim (seamless) avatars (VRML / WEB3D / MPEG-4) “Standard” H-Anim (seamless) avatars (VRML / WEB3D / MPEG-4) DI-Guy (Joint angle control) DI-Guy (Joint angle control) Jack (Toolkit API; plug-ins) Jack (Toolkit API; plug-ins) Application-specific models (Extempo, Ananova, …) Application-specific models (Extempo, Ananova, …) Custom models in Maya, 3DSMax, … Custom models in Maya, 3DSMax, … Issues: Support, Platform, API, Body Swap Issues: Support, Platform, API, Body Swap “Standard” H-Anim (seamless) avatars (VRML / WEB3D / MPEG-4) “Standard” H-Anim (seamless) avatars (VRML / WEB3D / MPEG-4) DI-Guy (Joint angle control) DI-Guy (Joint angle control) Jack (Toolkit API; plug-ins) Jack (Toolkit API; plug-ins) Application-specific models (Extempo, Ananova, …) Application-specific models (Extempo, Ananova, …) Custom models in Maya, 3DSMax, … Custom models in Maya, 3DSMax, … Issues: Support, Platform, API, Body Swap Issues: Support, Platform, API, Body Swap

March 1, ICT Virtual Human Workshop Body Requirements and Assumptions Enough joints, correct placement, DOFs, limits Enough joints, correct placement, DOFs, limits Skin surface shape (deformations) Skin surface shape (deformations) Torso shape and flexibility Torso shape and flexibility Physiology: breathing, itching, eating, … Physiology: breathing, itching, eating, … Rendering: skin, hair, moisture, vascular changes Rendering: skin, hair, moisture, vascular changes Human or caricature? Human or caricature? Resemblance to someone (or not) Resemblance to someone (or not) Clothing Clothing Enough joints, correct placement, DOFs, limits Enough joints, correct placement, DOFs, limits Skin surface shape (deformations) Skin surface shape (deformations) Torso shape and flexibility Torso shape and flexibility Physiology: breathing, itching, eating, … Physiology: breathing, itching, eating, … Rendering: skin, hair, moisture, vascular changes Rendering: skin, hair, moisture, vascular changes Human or caricature? Human or caricature? Resemblance to someone (or not) Resemblance to someone (or not) Clothing Clothing

March 1, ICT Virtual Human Workshop Object – Human Interactions: Challenges Grasping (one / two hands, or with other body parts) Grasping (one / two hands, or with other body parts) Multi-person actions (carrying furniture) Multi-person actions (carrying furniture) Smart objects; Compliant motion (constraints applied to existing motions) Smart objects; Compliant motion (constraints applied to existing motions) Knowledge of how things operate, fit together, disassemble Knowledge of how things operate, fit together, disassemble Grasping (one / two hands, or with other body parts) Grasping (one / two hands, or with other body parts) Multi-person actions (carrying furniture) Multi-person actions (carrying furniture) Smart objects; Compliant motion (constraints applied to existing motions) Smart objects; Compliant motion (constraints applied to existing motions) Knowledge of how things operate, fit together, disassemble Knowledge of how things operate, fit together, disassemble

March 1, ICT Virtual Human Workshop The Agent needs to Know … Action choices cause undesirable effects Actions changed to prevent errors

March 1, ICT Virtual Human Workshop Questions!