1. Visual Recognition of Human Movement Style Frank E. Pollick Department of Psychology University of Glasgow

2. Thanks to Chris Atkeson Mitsuo Kawato Josh Hale Armin Bruderlin Harold Hill Andy Calder Helena Paterson Vic Braden Cali Fidopiastis Zoe Kourtzi Stefan Schaal

3. Outline Introduction Emotion from motion Design of interactions with humanoid robots Spatial & temporal exaggerations based on average movements Style recognition as pattern matching

4. Biological Motion - point light displays spontaneously organized into the percept of a moving figure Possible to see detailed properties of the actor and action

5. What properties? Direction of walking movement Naturalness Expression of affect Identity Gender Style

6. Gender from Gait: A Closer Look

7. Recognition of movement style Studying the recognition of movement style should inform us of information essential to the representation of movement

8. Style Recognition: Three problems Low-level: How is biological motion spontaneously organized into the percept of human movement? Mid-level: What information is represented for use to organize movements into categories? High-level: How does context influence recognition?

9. Currently No theory of low-level motion detection can provide a bottom-up account of the perception of biological motion. No theory of movement categorization can indicate what general properties are crucial Evidence is beginning to accumulate that bidirectional processes are crucial to the recognition of human movement

10. One Mid-level Account of Biological Motion Perception Kinematic Specification of Dynamics (KSD) –Runeson & Frykholm (1981, 1983) conjectured that the kinematic structure of natural human movements is rich enough to uniquely specify the underlying dynamics of the actions being performed

11. Definitions Kinematic Properties of Movement –position, velocity, acceleration Dynamic Properties of Movement –force, torque, mass, inertia

12. Example Perception of Lifted Weight

13. What contraints can be used to infer dynamics from kinematics Laws of physics Particular configuration of the biomechanical linkages Motor control strategies (can be considered a type of bidirectional model)

14. My research agenda Currently focuses on a mid-level approach of searching for representation primitives that are useful for movement categorisation

15. Example: Emotion from motion

16. Results: Psychological Space Generated from Confusion Matrix

17. Results: Wrist Kinematics

18. Results: Correlation between Kinematics & Dimension 1 of Psychological Space

19. Comparison with Circumplex Model of Affect (Russell, 1980)

20. Questions Kinematics correlated to dynamics? Spatial or temporal factors important? What role for semantic representations of emotions?

21. Interaction with an anthropomorphic robot We wish to merge aspects of robot control and perception of human movement in designing the interaction –Practical benefit of creating better interfaces in field of entertainment robotics –Research benefit that exact details are available about movement production

22. Sticky Hands Exercise

23. Sticky Hands Game with the Robot Goal –To maintain constant contact force between hands of robot and human while moving together along an arbitrary trajectory –Robot learns and modifies path-following strategy from interactions with humans –Robot as a partner for personal development

24. Framework Path learning And prediction Motion Augmentation Robot Motor Controller + Position and Force Measurements Motor Control Instructions Desired trajectory Observed trajectory Predicted trajectory

25. Definitions Forward kinematics –Calculates cartesian position from joint angles Inverse kinematics –Calculates joint angles required to reach cartesian position Inverse dynamics –Calculates joint torques required for a posture

26. Two Possible Robot Control Strategies Inverse Dynamics Approach Forward Kinematics Approach Place hand at cartesian target using inverse kinematics Measure joint angles Measure joint loads (torque sensors) Estimate expected loads using Inverse Dynamics Subtract estimates from measured loads to estimate force on hand Respond Compliantly by : Moving hand to maintain constant contact force Place hand at cartesian target using inverse kinematics Measure joint angles Estimate hand position using forward kinematics Respond Compliantly by : Changing target position if distance between actual and target positions exceeds a threshold

27. Current Status Using Forward Kinematic Strategy Path learning and prediction algorithms complete Beginning to implement motion augmentation stage

28. Example of Interaction

29. Motion Augmentation Low & Mid level –Modify speed & smoothness based on results from emotion from motion High level –Overlay of high-level processes and context specific gestures

30. Exaggerated Movements Exaggerate spatial or temporal properties of human movement to see whether recognition is enhanced.

31. Movement Averages as the Basis for Movement Exaggeration Grand Average Style Average 1 Style Average 2 Style Average 3

32. Spatial and Temporal Exaggerations Exaggeration of spatial differences enhances recognition of tennis serve style and facial emotion Exaggeration of temporal differences enhances recognition of identity of drinking movements

33. Example: Spatial Exaggerations of Facial Emotion

34. Spatial Exaggerations Service Types

35. Exaggerated Topspin serves InterpolatedAverageExtrapolated

36. Results: Spatial Exaggerations Categorization Judgments Psychological Space

37. Temporal Exaggerations of Drinking Movements Define Movement Segments Before ExaggerationAfter Exaggeration

38. Temporal Exaggerations

39. Results: Temporal Exaggeration

40. Conclusion Both spatial and temporal information appear important for recognition –Need to more closely examine spatiotemporal interactions

41. Pattern-matching and the recognition of human movement

42. Generalization Task Participants are trained to categorize service movements from one particular direction and then asked to categorize the service movements of the same and a different server from the same and different viewing directions.

44. Examples of Stimuli

45. Generalization: Viewpoint

46. Generalization: IdentityInterferes with Recognition

47. Generalization Results of Automatic Classifier (Linear Discriminant based on first 2 principal components of the motions)

48. Movements as complex patterns The errors found in generalizing recognition from a training set to a testing set can be explained by pattern recognition on a low- dimensional representation of the movement. The creation of this representation appears to be quite flexible