1. Eye-Hand coordination Stan Gielen Radboud University Nijmegen

2. Why eye-hand coordination ? Visual feedback guidance Exploratory behaviour

3. Some scientific problems 1.The dynamics of eye and hand movements are very different 2.Dynamics for the eye are constant; inertia of the arm varies 3.Different neuronal structures are involved  how to synchronize ? 4.Different frames of reference: retinal coordinate system for visual perception Body-reference system for coordination of arm movements World coordinate system for planning actions 5.Synchronisation of visual (afferent) and motor (efferent) activity due to time delays in efferent and afferent pathways 6.……..

4. Questions 1.Is there a common drive for eye- and hand in aiming movements ? 2.How does the CNS deal with visuo-motor interaction if the dynamics of the eye and hand are very different ? 3.How is information about self-initiated movements used in gaze control ? 4.How does the CNS deal with the fact that dynamics of the arm are different due to variable inertia (arm posture, objects) ? 5.Which coordinate system is used in visuo-motor control ? 6.How does the CNS deal with synchronisation of visual (afferent) and motor (efferent) activity due to time delays in efferent and afferent pathways ? 7.If gaze is used to guide the hand along a path (tracking), does gaze lead the hand by a constant time or a constant distance ?

5. Dynamics of eye and arm movements

6. Introduction in saccadic system van Gisbergen, Robinson, S.Gielen. J. Neurophysiol 45, 417-442, 1981.

7. Position and velocity dependent force components

8. Burster cel

9. Motoneuron behavior

10. Neuronal wiring diagram Saccadic RT

11. Dynamics for limb movements Agonist EMG Antagonist EMG

12. Smooth pursuit : driven by retinal slip Carpenter, Movement of the Eyes, PION, 1977 Solid : predictable target Open: unpredictable target

13. Eye-Hand coordination in aiming movements

14. Eye-hand coordination in aiming movements time position

15. Coordination of eye and hand movements Gielen et al. Exp Brain Res, 56, 154-161, 1984. eye hand

16. Hand and eye positions as a function of RT

17. Conclusion When the eye and hand have to track an object, they are driven by a common drive. The common drive changes continuously as a function of time Gielen et al. Exp Brain Res, 56, 154-161, 1984.

18. Action and perception are coupled: Action-perception cycle Active versus Passive perception

19. Smooth pursuit : anticipation/prediction Carpenter, Movement of the Eyes, PION, 1977 Solid : predictable target Open: unpredictable target Tracking of passively moving target and of target motion due to active head movements

20. Gaze in passive and active conditions

21. Gaze in passive conditions Mean gain for the passive condition for 3 distances (25, 50 and 75 cm), 3 frequencies (0.5, 1 and 1.5 Hz) and 3 stimulus diameters

22. Frequency-dependent gain for active (closed) and passive (open) conditions Gielen et al. Exp Brain Res (2004) 155: 211–219

23. Summary Gaze control is significantly better in active conditions than in passive conditions due to anticipation and/or efference copies

24. Frames of reference in eye-hand coordination

25. Frames of reference viewer centered: relative to the eye, head, or shoulder ?

26. Frames of reference for pointing Desmurget et al., J.Neurophysiol In darkness In light environment

27. Potential error sources in pointing Error in sensory signal (visual, auditory) Error in storing information in memory Error in coordinate transformation from sensory to motor Error in motor command

28. Constant error Variable error

29. Pointing to a remembered target Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

30. Pointing to a remembered target Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

31. Subjects were standing, in a completely dark room Targets were removed during pointing

32. FINGER: no frame, LED on finger DARK: no frame, no LED on finger FRAME: frame and LED on finger 0 1 2 3 4 5 6 7s target cue pointing frame delay

33. DarkFingerFrame Pointing results 1.Errors in perception small for azimuth and elevation, but large for depth direction 2.Errors in motor programming relatively small !

34. Gaze and pointing to remembered visual targets Fixation is on target after target presentation in all conditions

35. Gaze and pointing to remembered visual targets Fixation drifts away from the subject in radial direction in the delay period. The effect is larger in the FRAME condition.

36. Gaze and pointing to remembered visual targets Fixation and pointing do not overlap, but orientation tuning is similar Variable error reflects eye- centered fram of reference

37. Have a cup of coffee and relax !

38. Pointing to a remembered target after egomotion gaze Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

39. DARKFINGER Pointing position FRAME Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

40. DARKFINGER Pointing position FRAME Fixation during pointing Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

41. DARKFINGER Pointing position FRAME Fixation at the end of step Fixation during pointing

42. DARKFINGER Average over all subjects Admiraal, Gielen et al. J Neurophysiol. 2003, 2004 For discrete changes in relative target position: Covariation between gaze and hand position: Hand position is biased to gaze position

43. Conclusions 1.Errors in visual perception are small in azimuth and elevation, but large depth 2.An external frame of reference reduces pointing errors 3.Errors in motor programming are relatively small; main contribution to variable error in pointing is due to errors in visual perception. 4.Updating for ego-motion is incomplete

44. Pointing accuracy in pathology: Parkinson Disease

45. Pointing in Parkinsons Disease Parkinson Disease is primarily a sensory-motor disorder Keijsers, Gielen et al., European Journal of Neuroscience, 21, 239–248, 2005

46. Reference frames for eye-hand coordination For a retinal frame of reference pointing errors should depend on gaze position relative to target position ?

47. Pointing errors in retinal frame of reference ! Beurze, van Pelt en Medendorp J Neurophysiol 96: 352–362, 2006. H, initial hand position; F, fixation point; T, target for movement. B: schematic representation of the paradigm. Horizontal position of the eyes (- - -) and the hand’s pointing direction (—). Boxes indicate location and duration of initial hand position (H, 0–2500 ms), fixation (F, 2500–4500 ms), and target (T, 3500–4500 ms). Visual feedback about index finger position was available until start of movement.

48. Spatial updating operates in gaze-centered coordinates. Medendorp et al. J Neurophysiol 93: 954–962, 2005. Occipital, posterior parietal, and premotor cortex are active for saccades and during pointing movements, retIPS is activated more strongly for saccades than for pointing. The activation associated with pointing was significantly greater when pointing with the unseen hand to targets ipsilateral to the hand. Although there was activation in the left retIPS when pointing to targets on the right with the left hand, the activation was significantly greater when using the right hand (mirror symmetric effect for right retIPS). Each hand is more effective in directing movements to targets in ipsilateral visual space.

49. Hypothetical flow of eye- and hand-related signals to Posterior Parietal Cortex (IPS) A posteriorly directed stream flows to the retIPS, where relative joint angles are converted into a map with the position of possible effectors coded with respect to the eye. In retIPS, this map, with the effectors initially coded in the contralateral cortex having the weaker representation, is integrated with the map of target location. Visual information about target location, travels along an anteriorly directed stream into retIPS, which encodes a map of target location in the contralateral visual field. Proprioceptive information of each effector resides within the primary somatosensory cortex, in the postcentral sulcus, which somatotopically represents the contralateral body parts.

50. Eye-Hand coordination in tracking and tracing

51. Eye and hand tracking in 3D azimuth elevation depth

52. Eye and hand in 3D azimuth elevation depth azimuth elevation depth trackingtracing

53. Interpretation of crosscorrelation : optimal overlap of eye and hand traces Obviously gaze leads hand position. How to quantify the delay between saccadic eye position and hand position ?

54. How to quantify the delay between saccadic eye position and hand position Tramper & Gielen The Journal of Neuroscience 31 (21): 7857-7866, 2011

55. Tracking Gielen et al. Cortex 45: 340 – 355, 2009

56. Tracing Tramper & Gielen The Journal of Neuroscience 31 (21): 7857-7866, 2011

57. Lead time of gaze relative to finger is not constant ! Tramper & Gielen The Journal of Neuroscience 31 (21): 7857-7866, 2011 After correction for saccadic lead time

58. Gaze leads the finger by a constant distance, not a constant lead time ! Δs : 2.2 – 3.7 cm Tramper & Gielen The Journal of Neuroscience 31 (21): 7857-7866, 2011

59. Mean correlation between hand velocity and saccade interval -0.45 (SD 0.10) * * p<0.01 Eye fixations lead hand position by a constant displacement Gielen et al., Cortex 45: 340 – 355, 2009

60. Simplified model for trajectory generation for discrete aiming movements

61. Taken from Wolpert, Kawato et al.

62. Time delays should be incorporated for moving targets ! Δt 1 Δt2Δt2 Comparing signals at different positions in time !

63. Sensory prediction Courtesy Daniel Wolpert

66. Virtual reality

67. Thanks for your attention !