Sensory-motor integration Bijan Pesaran May 1 st, 2008

Visual response in PMv centered on hand Graziano and Gross (1994)

Visual response in PMv centered on hand Graziano and Gross (1994)

Duhamel et al (1997) Invariant visual response in VIP

Duhamel et al (1997)

PRR neurons are eye-centered Batista et al (1999)

Updating of response in PRR

Mechanisms for encoding multiple variables Centered tuning curve

Eye position influences visual response of 7a neurons Andersen et al (1985)

Mechanisms for encoding multiple variables Centered tuning curve Multiplicative encoding - gain field Tuning curveGain field

Visual receptive fields in VIP partially shift with eye position Avillac et al (2005) Eye positions

Mechanisms for encoding multiple variables Centered tuning curve Multiplicative encoding - gain field Partial shift encoding – intermediate reference frame Tuning curveGain field

Eye position encoding Widespread phenomenon throughout visual and motor cortices Peaked response Eye left Eye middle Eye right Response Target position

Eye position encoding Difficulty addressing flanking responses Peaked response Flanking response Eye left Eye middle Eye right Response Target position

Gain model

Shifting model

Geometry of visually-guided reaching Eye-centered Hand-centered Relative position vectors Nonlinear 3D geometry and muscle dynamics H T E T-E E-H H-E T-H

Geometry of visually-guided reaching Nonlinear 3D geometry and muscle dynamics H T E T-E E-H H-E T-H MIP PMd Batista et al (1999) Eye-centered Graziano et al (1997) Hand-centered PMv What is code? T-E T-H Pesaran et al (2006)

Decompose the 3-D response matrix

Distinguish between different reference frames

Define multiple effector reference frames

Multiple reference frames Direct visuomotor transformation –Supports Area 5 data (Buneo et al 2002) Direct Transformation Serial Transformation Head-on-body position Eye-in-head position Eye- centered target location Head- centered target location Body- centered target location Hand- centered target location Body-centered hand position + - Motor output Eye- centered target location Hand- centered target location Eye-centered hand location + - Nonlinear 3D geometry and muscle dynamics Motor output Nonlinear 3D geometry and muscle dynamics

MIP PMd Nonlinear muscle dynamics Encoding relative position Nonlinear 3D geometry and muscle dynamics Batista et al 1999 Graziano et al 1997 H T E T-E E-H H-E T-H PMv Coordinating the hand and the eye Eye-centered Hand-centered Relative hand- eye position Vectors in abstract space

Design reference frame task 0 12 Start Cue Delay Go Hold Time (sec)

Design reference frame task 0 12 Start Cue Delay Go Hold Time (sec)

Design reference frame task Four initial eye positions

Design reference frame task Four initial eye and hand positions

Design reference frame task Four initial eye, hand and target positions 111 PMd neurons and 48 MIP neurons

Example PMd neuron

Example MIP neuron

Sensory-motor transformations Head-on-body position Eye-in-head position Eye- centered target location Head- centered target location Body- centered target location Hand- centered target location Body-centered hand position + - Serial Transformation

Sensory-motor transformations Eye-centered hand location Eye-centered target location Hand-centered target location - + Direct Transformation Buneo et al. (2002)

Sensory-motor transformations Eye-centered target location Hand-centered target location - + Relative position Transformations Eye-centered target location Hand-centered target location - + Relative location of hand and eye Eye-centered hand location Hand-centered eye location

Relative position particularly useful for coordination Relative hand-eye position could be important for hand-eye coordination –Can map hand and eye onto each other –Computationally efficient No need to code for all combinations of locations H T E T-E E-H H-E T-H

Relative position particularly useful for coordination Relative hand-eye position could be important for hand-eye coordination –Can map hand and eye onto each other –Computationally efficient No need to code for all combinations of locations H T E T-E E-H H-E T-H

Relative position particularly useful for coordination Relative hand-eye position could be important for hand-eye coordination –Can map hand and eye onto each other –Computationally efficient No need to code for all combinations of locations H T E T-E E-H H-E T-H