Visual Recognition of Human Movement Style Frank E. Pollick Department of Psychology University of Glasgow

Thanks to Chris Atkeson Mitsuo Kawato Josh Hale Armin Bruderlin Harold Hill Andy Calder Helena Paterson Vic Braden Cali Fidopiastis Zoe Kourtzi Stefan Schaal

Outline Introduction Emotion from motion Design of interactions with humanoid robots Spatial & temporal exaggerations based on average movements Style recognition as pattern matching

Biological Motion - point light displays spontaneously organized into the percept of a moving figure Possible to see detailed properties of the actor and action

What properties? Direction of walking movement Naturalness Expression of affect Identity Gender Style

Gender from Gait: A Closer Look

Recognition of movement style Studying the recognition of movement style should inform us of information essential to the representation of movement

Style Recognition: Three problems Low-level: How is biological motion spontaneously organized into the percept of human movement? Mid-level: What information is represented for use to organize movements into categories? High-level: How does context influence recognition?

Currently No theory of low-level motion detection can provide a bottom-up account of the perception of biological motion. No theory of movement categorization can indicate what general properties are crucial Evidence is beginning to accumulate that bidirectional processes are crucial to the recognition of human movement

One Mid-level Account of Biological Motion Perception Kinematic Specification of Dynamics (KSD) –Runeson & Frykholm (1981, 1983) conjectured that the kinematic structure of natural human movements is rich enough to uniquely specify the underlying dynamics of the actions being performed

Definitions Kinematic Properties of Movement –position, velocity, acceleration Dynamic Properties of Movement –force, torque, mass, inertia

Example Perception of Lifted Weight

What contraints can be used to infer dynamics from kinematics Laws of physics Particular configuration of the biomechanical linkages Motor control strategies (can be considered a type of bidirectional model)

My research agenda Currently focuses on a mid-level approach of searching for representation primitives that are useful for movement categorisation

Example: Emotion from motion

Results: Psychological Space Generated from Confusion Matrix

Results: Wrist Kinematics

Results: Correlation between Kinematics & Dimension 1 of Psychological Space

Comparison with Circumplex Model of Affect (Russell, 1980)

Questions Kinematics correlated to dynamics? Spatial or temporal factors important? What role for semantic representations of emotions?

Interaction with an anthropomorphic robot We wish to merge aspects of robot control and perception of human movement in designing the interaction –Practical benefit of creating better interfaces in field of entertainment robotics –Research benefit that exact details are available about movement production

Sticky Hands Exercise

Sticky Hands Game with the Robot Goal –To maintain constant contact force between hands of robot and human while moving together along an arbitrary trajectory –Robot learns and modifies path-following strategy from interactions with humans –Robot as a partner for personal development

Framework Path learning And prediction Motion Augmentation Robot Motor Controller + Position and Force Measurements Motor Control Instructions Desired trajectory Observed trajectory Predicted trajectory

Definitions Forward kinematics –Calculates cartesian position from joint angles Inverse kinematics –Calculates joint angles required to reach cartesian position Inverse dynamics –Calculates joint torques required for a posture

Two Possible Robot Control Strategies Inverse Dynamics Approach Forward Kinematics Approach Place hand at cartesian target using inverse kinematics Measure joint angles Measure joint loads (torque sensors) Estimate expected loads using Inverse Dynamics Subtract estimates from measured loads to estimate force on hand Respond Compliantly by : Moving hand to maintain constant contact force Place hand at cartesian target using inverse kinematics Measure joint angles Estimate hand position using forward kinematics Respond Compliantly by : Changing target position if distance between actual and target positions exceeds a threshold

Current Status Using Forward Kinematic Strategy Path learning and prediction algorithms complete Beginning to implement motion augmentation stage

Example of Interaction

Motion Augmentation Low & Mid level –Modify speed & smoothness based on results from emotion from motion High level –Overlay of high-level processes and context specific gestures

Exaggerated Movements Exaggerate spatial or temporal properties of human movement to see whether recognition is enhanced.

Movement Averages as the Basis for Movement Exaggeration Grand Average Style Average 1 Style Average 2 Style Average 3

Spatial and Temporal Exaggerations Exaggeration of spatial differences enhances recognition of tennis serve style and facial emotion Exaggeration of temporal differences enhances recognition of identity of drinking movements

Example: Spatial Exaggerations of Facial Emotion

Spatial Exaggerations Service Types

Exaggerated Topspin serves InterpolatedAverageExtrapolated

Results: Spatial Exaggerations Categorization Judgments Psychological Space

Temporal Exaggerations of Drinking Movements Define Movement Segments Before ExaggerationAfter Exaggeration

Temporal Exaggerations

Results: Temporal Exaggeration

Conclusion Both spatial and temporal information appear important for recognition –Need to more closely examine spatiotemporal interactions

Pattern-matching and the recognition of human movement

Generalization Task Participants are trained to categorize service movements from one particular direction and then asked to categorize the service movements of the same and a different server from the same and different viewing directions.

Examples of Stimuli

Generalization: Viewpoint

Generalization: IdentityInterferes with Recognition

Generalization Results of Automatic Classifier (Linear Discriminant based on first 2 principal components of the motions)

Movements as complex patterns The errors found in generalizing recognition from a training set to a testing set can be explained by pattern recognition on a low- dimensional representation of the movement. The creation of this representation appears to be quite flexible