1. Tutorial on the Visual Perception of Human Movement Frank E. Pollick Department of Psychology University of Glasgow & ATR Cyberhuman Project

2. Outline Chronology of Human Movement Research Brain Mechanisms of Human Movement Perception Applications to Humanoid Robots

3. Chronology of Human Movement Research From Johansson ’ s point-light displays to the present Review & Summary

4. Biological Motion - point light displays (Johansson, 1973) spontaneously organized into the percept of a moving figure Possible to see detailed properties of the actor and action

5. References: Biological Motion - point light displays (Johansson, 1973) Johansson, G. (1973). Visual perception of biological motion and a model for its analysis. Perception and Psychophysics, 14, 201-211. Johansson, G. (1975). Visual motion perception. Scientific American, 232, 76-88.

6. Person Perception & Nonverbal Communication (1980s-2001) Numerous studies have applied point-light displays to questions involving: –Gender stereotypes –Visual perception of music performance –Role of IQ in person perception

7. What properties? (late 1970s) Identity Style (e.g. walking versus running) Gender

8. References: Recognizing Gender & Identity from Gait Hoenkamp, E. (1978). Perceptual cues that determine the labeling of human gait. Journal of Human Movement Studies, 4, 59-69. Barclay, C.D., Cutting, J.E. & Kozlowski, L.T. (1978). Temporal and spatial factors in gait perception that influence gender recognition. Perception and Psychophysics, 23, 145-152. Cutting, J.E. & Kozlowski, L.T. (1977). Recognizing friends by their walk: Gait perception without familiarity cues. Bulletin of the Psychonomic Society, 9, 353-356. Cutting, J.E., Proffitt, D.R. & Kozlowski, L.T. (1978). A biomechanical invariant for gait perception. Journal of Experimental Psychology: Human Perception and Performance, 4, 357-372.

9. Closer Look at Recognizing Gender from Point-Light Walkers

10. Tentative Results of a Meta-Analysis on these Gender Recognition Studies Walking - 69% (CI 63-74%) Other Activities - 79% (CI 64-89%) Walking + Other Activities - 71% (CI 67-72%) Performance is not that good for walking Other activities appear to give a boost to performance

11. Theories of Biological Motion Perception (1981 - 1983) Piecewise planarity Hierarchical body model Kinematic specification of dynamics

12. References: Theories of Biological Motion Perception (1981 - 1983) Hoffman, D. D., & Flinchbaugh, B. E. (1982). The Interpretation of Biological Motion. Biological Cybernetics, 42(3), 195-204. Runeson, S., & Frykholm, G. (1981). Visual perception of lifted weight. Journal of Experimental Psychology: Human Perception and Performance, 7, 733-740. Runeson, S., & Frykholm, G. (1983). Kinematic specification of dynamics as an informational basis for person and action perception: Expectation, gender recognition, and deceptive intention. Journal of Experimental Psychology: General, 112, 585-615. Cutting, J.E. & Proffitt, D. (1981). Gait perception as an example of how we may perceive events. In R. Walk & H. Pick, Jr. (Eds.), Intersensory perception and sensory integration. Plenum, New York, New York Bertenthal, B.I. & Pinto, J. (1994). Global processing of biological motions. Psychological Science, 5, 221-225.

13. Kinematic Specification of Dynamics (KSD) Runeson & Frykholm (1981, 1983) conjectured that the kinematic structure of natural human movements is rich enough to uniquely specify the underlying dynamics of the actions being performed

14. Example Perception of Lifted Weight

15. Development of the Ability to Recognize Biological Motion (1982, 1987) Infants as young as 4 months are sensitive to biological motion

16. References: Development of the Ability to Recognize Biological Motion (1982, 1987) Fox, R., & McDaniel, C. (1982). The Perception of Biological Motion by Human Infants. Science, 218(4571), 486-487. Bertenthal, B.I., Proffitt, D.R., Kramer, S.J., & Spetner, N.B. (1987). Infants encoding of kinetic displays varying in relative coherence. Developmental Psychology, 23, 171- 178.

17. Apparent Motion (1991, 1993) When two static images are presented sequentially, motion is perceived if certain spatial and temporal conditions are met. For potentially ambiguous sequences of human movement, the seen movement varies between impossible and possible as time between images increases. Interpretation of low-level motion signal requires high-level cognitive processes

18. Pair of photos Interpretations Impossible - seen when short time between frames Possible - seen when long time between frames Impossible Possible Figure from Stevens et al (2000), Neuroreport

19. References: Apparent Motion (1991, 1993) Shiffrar, M. & Freyd, J.J. (1990). Apparent motion of the human body. Psychological Science, 1, 257-264. Shiffrar, M. & Freyd, J.J. (1993). Timing and apparent motion path choice with human body photographs. Psychological Science, 4, 379-384. Stevens, J.A., Fonlupt, P., Shiffrar, M. & Decety, J. (2000). New aspects of motion perception: selective neural encoding of apparent human movements. Neuroreport, 11, 109-115.

20. Motion Detectors (1992, 1998) At the primary level of visual analysis there exists filters tuned for a variety of visual motion. Is there a detector specifically tuned for detecting biological motion? Specific detectors for biological motion not conclusively found, if they do exist they must integrate visual information over a longer period of time and a greater spatial extent.

21. References: Motion Detectors (1992, 1998) Mather, G., Radford, K., & West, S. (1992). Low-Level Visual Processing of Biological Motion. Proceedings of the Royal Society of London Series B-Biological Sciences, 249(1325), 149-155. Neri, P., Morrone, M. C., & Burr, D. C. (1998). Seeing biological motion. Nature, 395(6705), 894-896.

22. Representing Human Movement for Recognition (1993, 2000) Consistent with view-based theories of static object recognition, the recognition of human movement does not appear invariant over viewing direction.

23. References: Representing Human Movement for Recognition (1993, 2000) Verfaillie, K. (1992). Variant Points-of-View on Viewpoint Invariance. Canadian Journal of Psychology-Revue Canadienne De Psychologie, 46(2), 215-235. Verfaillie, K. (1993). Orientation-Dependent Priming Effects in the Perception of Biological Motion. Journal of Experimental Psychology-Human Perception and Performance, 19(5), 992- 1013. Verfaillie, K. (1997). Transsaccadic memory for the egocentric and allocentric position of a biological-motion walker. Journal of Experimental Psychology-Learning Memory and Cognition, 23(3), 739-760. Verfaillie, K., Detroy, A., & Vanrensbergen, J. (1994). Transsaccadic Integration of Biological Motion. Journal of Experimental Psychology-Learning Memory and Cognition, 20(3), 649-670.

24. Social Perception (1992, 1996, 2001) Locomotion recognised better than social or instrumental actions Emotion can be recognised from point-light displays

25. References: Social Perception (1992, 1996, 2001) Dittrich, W. H. (1993). Action Categories and the Perception of Biological Motion. Perception, 22(1), 15-22. Dittrich, W. H., Troscianko, T., Lea, S. E. G., & Morgan, D. (1996). Perception of emotion from dynamic point-light displays represented in dance. Perception, 25(6), 727-738. Pollick, F.E., Paterson, H., Bruderlin, A. & Sanford, A.J. (2001) Perceiving affect from arm movement. Cognition, 82, B51--B61. Walk, R.D. & Homan, C.P. (1984). Emotion and dance in dynamic light displays. Bulletin of the Psychonomic Society, 22, 437-440

26. Exaggerating the Style of Human Movement (2000, 2001) One can construct spatial and temporal exaggerations of movements that obtain enhanced recognition of movement style Both spatial and temporal information can be used for recognition, the primitive property used by humans is likely some spatiotemporal factor such as velocity.

27. References: Exaggerating the Style of Human Movement (2000, 2001) Pollick, F. E., Fidopiastis, C., & Braden, V. (2001). Recognising the style of spatially exaggerated tennis serves. Perception, 30(3), 323-338. Hill, H., & Pollick, F. E. (2000). Exaggerating temporal differences enhances recognition of individuals from point light displays. Psychological Science, 11(3), 223-228.

28. Review-1 We can recognize properties of actors and actions from human movement but accuracy is not always high There is no complete or experimentally verified theory of biological motion perception Our skill at recognizing human movement starts at a very young age

29. Review-2 It appears that human movement is perceived via interactions between low- level motion measurements and high-level cognitive processes Although the precise nature of these interactions is not understood recent progress in understanding the neural processes holds promise.

30. Brain Mechanisms of Human Movement Perception Brief review of recent results

31. Perception of Human Movement in Macaque Monkeys In 1994 Oram and Perrett reported cells in the superior temporal polysensory area (STPa) of the macaque temporal cortex that were sensitive to biological motion. Oram, M. W., & Perrett, D. I. (1994). Responses of Anterior Superior Temporal Polysensory (Stpa) Neurons to Biological Motion Stimuli. Journal of Cognitive Neuroscience, 6(2), 99-116.

32. Perception of Human movement in man In the past two years several researchers reported that a corresponding brain area (located in the superior temporal sulcus) in humans responded to human movement Figures from Grossman et al (2000), Journal of Cognitive Neuroscience

33. References: Perception of Human movement in man Grezes, J., Fonlupt, P., Bertenthal, B., Delon-Martin, C., Segebarth, C., & Decety, J. (2001). Does perception of biological motion rely on specific brain regions? Neuroimage, 13(5), 775-785. Grossman, E., Donnelly, M., Price, R., Pickens, D., Morgan, V., Neighbor, G., & Blake, R. (2000). Brain areas involved in perception of biological motion. Journal of Cognitive Neuroscience, 12(5), 711-720. Grossman, E. D., & Blake, R. (2001). Brain activity evoked by inverted and imagined biological motion. Vision Research, 41(10-11), 1475-1482. Grossman, E. D., & Blake, R. (1999). Perception of coherent motion, biological motion and form- from- motion under dim-light conditions. Vision Research, 39(22), 3721-3727.

34. Other Brain Areas Involved in Social Cognition The brain area of the superior temporal sulcus (STS) appears involved in the processing of a variety of social signals Figure from Allison et al (2000), Trends in Cognitive Science

35. Proposed Brain Circuit for Social Perception Figure from Allison et al (2000), Trends in Cognitive Science

36. References: Neural Basis of Social Cognition Allison, T., Puce, A. & McCarthy, G. (2000). Social perception from visual cues: Role of the STS region. Trends in Cognitive Science, 4, 267-272. Adolphs, R. (1999). Social cognition and the human brain. Trends in Cognitive Science, 3, 469-479. Adolphs, R. (2001). The neurobiology of social cognition. Current Opinion in Neurobiology, 11, 231-239.

37. Mirror Neurons Neurons in the ventral premotor cortex of macaque monkey discharge both during execution of goal directed hand actions and when the monkey observes similar actions performed by others There is brain imaging evidence for similar interactions in humans

38. References: Mirror Neurons Rizzolatti, G., Fogassi, L., & Gallese, V. (2001). Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Reviews Neuroscience, 2, 661-670.

39. Review Specific brain areas have been found to be active during the perception of human movement Circuits have been implicated for the processing of social signals and imitation This is a new and growing field that will likely yield important results in the next 10 years

40. Applications to Humanoid Robots Specific example of an application

41. Specific Application to Humanoid Robots

42. Problem For this and other applications we would like the robot to move in a way that would express emotion How can research into human movement perception help us?

43. Emotion from Motion Perception of affect from point lights conform to a circumplex model of affect (Pollick, F.E., Paterson, H.M., Bruderlin, A. & Sanford, A.J. (2001) Cognition, 82, B51- B61.)

44. The Circumplex Model of Affect (Russell, 1980) Pleasantness Activation Pleasant Unpleasant ActiveInactive

45. Stimuli: Emotion from motion

46. Results: Psychological Space Generated from Confusion Matrix

47. Results: Correlation between Kinematics & Dimension 1 of Psychological Space

48. Stimuli: Scrambled Displays

49. Results: Psychological Space Generated from Confusion Matrix

50. Results: Correlation between Kinematics & Dimension 1 of Psychological Space

51. Summary of Emotion from Motion For natural movements the perception of affect conforms to a circumplex structure The activation dimension corresponds to a formless cue that relates directly to the movement kinematics. The pleasantness dimension corresponds to phase relations between the different limb segments.

52. Application to Humanoid Robots Currently we are examining how to manipulate the robot control algorithms to obtain independent control of the two variables corresponding to Dimension 1 and Dimension 2.

53. However..... Perception of human movement relies on higher order cognitive factors (expectations). It is not known how seeing a movement produced by a humanoid will influence the processing of the motion.