Eye-Hand coordination Stan Gielen Radboud University Nijmegen

Why eye-hand coordination ? Visual feedback guidance Exploratory behaviour

Some scientific problems 1.The dynamics of eye and hand movements are very different 2.Dynamics for the eye are constant; inertia of the arm varies 3.Different neuronal structures are involved  how to synchronize ? 4.Different frames of reference: retinal coordinate system for visual perception Body-reference system for coordination of arm movements World coordinate system for planning actions 5.Synchronisation of visual (afferent) and motor (efferent) activity due to time delays in efferent and afferent pathways 6.……..

Questions 1.Is there a common drive for eye- and hand in aiming movements ? 2.How does the CNS deal with visuo-motor interaction if the dynamics of the eye and hand are very different ? 3.How is information about self-initiated movements used in gaze control ? 4.How does the CNS deal with the fact that dynamics of the arm are different due to variable inertia (arm posture, objects) ? 5.Which coordinate system is used in visuo-motor control ? 6.How does the CNS deal with synchronisation of visual (afferent) and motor (efferent) activity due to time delays in efferent and afferent pathways ? 7.If gaze is used to guide the hand along a path (tracking), does gaze lead the hand by a constant time or a constant distance ?

Dynamics of eye and arm movements

Introduction in saccadic system van Gisbergen, Robinson, S.Gielen. J. Neurophysiol 45, , 1981.

Position and velocity dependent force components

Burster cel

Motoneuron behavior

Neuronal wiring diagram Saccadic RT

Dynamics for limb movements Agonist EMG Antagonist EMG

Smooth pursuit : driven by retinal slip Carpenter, Movement of the Eyes, PION, 1977 Solid : predictable target Open: unpredictable target

Eye-Hand coordination in aiming movements

Eye-hand coordination in aiming movements time position

Coordination of eye and hand movements Gielen et al. Exp Brain Res, 56, , eye hand

Hand and eye positions as a function of RT

Conclusion When the eye and hand have to track an object, they are driven by a common drive. The common drive changes continuously as a function of time Gielen et al. Exp Brain Res, 56, , 1984.

Action and perception are coupled: Action-perception cycle Active versus Passive perception

Smooth pursuit : anticipation/prediction Carpenter, Movement of the Eyes, PION, 1977 Solid : predictable target Open: unpredictable target Tracking of passively moving target and of target motion due to active head movements

Gaze in passive and active conditions

Gaze in passive conditions Mean gain for the passive condition for 3 distances (25, 50 and 75 cm), 3 frequencies (0.5, 1 and 1.5 Hz) and 3 stimulus diameters

Frequency-dependent gain for active (closed) and passive (open) conditions Gielen et al. Exp Brain Res (2004) 155: 211–219

Summary Gaze control is significantly better in active conditions than in passive conditions due to anticipation and/or efference copies

Frames of reference in eye-hand coordination

Frames of reference viewer centered: relative to the eye, head, or shoulder ?

Frames of reference for pointing Desmurget et al., J.Neurophysiol In darkness In light environment

Potential error sources in pointing Error in sensory signal (visual, auditory) Error in storing information in memory Error in coordinate transformation from sensory to motor Error in motor command

Constant error Variable error

Pointing to a remembered target Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

Pointing to a remembered target Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

Subjects were standing, in a completely dark room Targets were removed during pointing

FINGER: no frame, LED on finger DARK: no frame, no LED on finger FRAME: frame and LED on finger s target cue pointing frame delay

DarkFingerFrame Pointing results 1.Errors in perception small for azimuth and elevation, but large for depth direction 2.Errors in motor programming relatively small !

Gaze and pointing to remembered visual targets Fixation is on target after target presentation in all conditions

Gaze and pointing to remembered visual targets Fixation drifts away from the subject in radial direction in the delay period. The effect is larger in the FRAME condition.

Gaze and pointing to remembered visual targets Fixation and pointing do not overlap, but orientation tuning is similar Variable error reflects eye- centered fram of reference

Have a cup of coffee and relax !

Pointing to a remembered target after egomotion gaze Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

DARKFINGER Pointing position FRAME Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

DARKFINGER Pointing position FRAME Fixation during pointing Admiraal, Gielen et al. J Neurophysiol. 2003, 2004

DARKFINGER Pointing position FRAME Fixation at the end of step Fixation during pointing

DARKFINGER Average over all subjects Admiraal, Gielen et al. J Neurophysiol. 2003, 2004 For discrete changes in relative target position: Covariation between gaze and hand position: Hand position is biased to gaze position

Conclusions 1.Errors in visual perception are small in azimuth and elevation, but large depth 2.An external frame of reference reduces pointing errors 3.Errors in motor programming are relatively small; main contribution to variable error in pointing is due to errors in visual perception. 4.Updating for ego-motion is incomplete

Pointing accuracy in pathology: Parkinson Disease

Pointing in Parkinsons Disease Parkinson Disease is primarily a sensory-motor disorder Keijsers, Gielen et al., European Journal of Neuroscience, 21, 239–248, 2005

Reference frames for eye-hand coordination For a retinal frame of reference pointing errors should depend on gaze position relative to target position ?

Pointing errors in retinal frame of reference ! Beurze, van Pelt en Medendorp J Neurophysiol 96: 352–362, H, initial hand position; F, fixation point; T, target for movement. B: schematic representation of the paradigm. Horizontal position of the eyes (- - -) and the hand’s pointing direction (—). Boxes indicate location and duration of initial hand position (H, 0–2500 ms), fixation (F, 2500–4500 ms), and target (T, 3500–4500 ms). Visual feedback about index finger position was available until start of movement.

Spatial updating operates in gaze-centered coordinates. Medendorp et al. J Neurophysiol 93: 954–962, Occipital, posterior parietal, and premotor cortex are active for saccades and during pointing movements, retIPS is activated more strongly for saccades than for pointing. The activation associated with pointing was significantly greater when pointing with the unseen hand to targets ipsilateral to the hand. Although there was activation in the left retIPS when pointing to targets on the right with the left hand, the activation was significantly greater when using the right hand (mirror symmetric effect for right retIPS). Each hand is more effective in directing movements to targets in ipsilateral visual space.

Hypothetical flow of eye- and hand-related signals to Posterior Parietal Cortex (IPS) A posteriorly directed stream flows to the retIPS, where relative joint angles are converted into a map with the position of possible effectors coded with respect to the eye. In retIPS, this map, with the effectors initially coded in the contralateral cortex having the weaker representation, is integrated with the map of target location. Visual information about target location, travels along an anteriorly directed stream into retIPS, which encodes a map of target location in the contralateral visual field. Proprioceptive information of each effector resides within the primary somatosensory cortex, in the postcentral sulcus, which somatotopically represents the contralateral body parts.

Eye-Hand coordination in tracking and tracing

Eye and hand tracking in 3D azimuth elevation depth

Eye and hand in 3D azimuth elevation depth azimuth elevation depth trackingtracing

Interpretation of crosscorrelation : optimal overlap of eye and hand traces Obviously gaze leads hand position. How to quantify the delay between saccadic eye position and hand position ?

How to quantify the delay between saccadic eye position and hand position Tramper & Gielen The Journal of Neuroscience 31 (21): , 2011

Tracking Gielen et al. Cortex 45: 340 – 355, 2009

Tracing Tramper & Gielen The Journal of Neuroscience 31 (21): , 2011

Lead time of gaze relative to finger is not constant ! Tramper & Gielen The Journal of Neuroscience 31 (21): , 2011 After correction for saccadic lead time

Gaze leads the finger by a constant distance, not a constant lead time ! Δs : 2.2 – 3.7 cm Tramper & Gielen The Journal of Neuroscience 31 (21): , 2011

Mean correlation between hand velocity and saccade interval (SD 0.10) * * p<0.01 Eye fixations lead hand position by a constant displacement Gielen et al., Cortex 45: 340 – 355, 2009

Simplified model for trajectory generation for discrete aiming movements

Taken from Wolpert, Kawato et al.

Time delays should be incorporated for moving targets ! Δt 1 Δt2Δt2 Comparing signals at different positions in time !

Sensory prediction Courtesy Daniel Wolpert

Virtual reality

Thanks for your attention !