Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1 Michael Arbib: CS564 - Brain Theory and Artificial Intelligence University of Southern California, Fall 2001 Lecture 18. The FARS model of Control of Reaching and Grasping Reading Assignments: Motor Schemas and Cortical Regions: TMB 2, Sections 2.2, 5.3, 6.3* FARS Model: Fagg, A. H., and Arbib, M. A., 1998, Modeling Parietal- Premotor Interactions in Primate Control of Grasping, Neural Networks, 11: * Caution: Most of the neuroanatomy in 6.3 is still reliable, but much research has updated our understanding of cortical correlates of motor control since 1989.

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 2 Perceptual And Motor Schemas A perceptual schema embodies the process whereby the system determines whether a given domain of interaction is present in the environment. {Recall our discussion of VISIONS, TMB2 §5.2} A schema assemblage combines an estimate of environmental state with a representation of goals and needs The internal state is also updated by knowledge of the state of execution of current plans made up of motor schemas which are akin to control systems but distinguished by the fact that they can be combined to form coordinated control programs

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 3 Preshaping While Reaching to Grasp

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 4 Hypothetical coordinated control program for reaching and grasping Dashed lines — activation signals; solid lines — transfer of data. (Adapted from Arbib 1981) Perceptual Schemas Motor Schemas

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 5 "What" versus "How” in Human Visual Cortex Parietal Cortex Inferotemporal Cortex How (dorsal) What (ventral) reach programming grasp programming AT: Goodale and Milner Lesion here: Inability to verbalize or pantomime size or orientation DF: Jeannerod et al. Lesion here: Inability to Preshape (except for objects with size “in the semantics” Monkey Data: Mishkin and Ungerleider on “What” versus “Where”

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 6 Reaching for an object by a patient with a lesion of the parietal cortex: Jeannerod, M., Michel, F., Prablanc, C., 1984, The Control of Hand Movements in a Case of Hemianaesthesia Following a Parietal Lesion, Brain107: Consider the implications for Project 1.

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 7 Virtual Fingers Arbib, Iberall and Lyons

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 8 Opposition Spaces and Virtual Fingers The goal of a successful preshape, reach and grasp is to match the opposition axis defined by the virtual fingers of the hand with the opposition axis defined by an affordance of the object (Iberall and Arbib 1990)

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 9 Planning for the reach must take account of the planned grasp

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 10 Somatosensory areas SMA FEF (saccades) SMA = pre-SMA + SMA-proper

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 11 Somatosensory data: A key to motor control

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 12 Do data support the idea of virtual fingers? Iberall and Arbib 1988 suggest that multiple, seemingly non-somatotopic, representations of the digits could be due to a virtual finger representation. Using the tentative identifications: VF1 involves palm and thumb areas VF2 involves the index with or without other fingers VF3 involves finger combinations excluding the thumb and the index yields a possible mapping of virtual fingers onto the caudal kinesthetic map that Strick and Preston (1962) found in the squirrel monkey.

Iberall and Arbib’s 1988 view of cortical contributions to the coordinated control program for reaching and grasping. Use it as an evaluation point as we develop the FARS and MNS models. What do we gain, what have we lost? See TMB2 §6.3 for the details.

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 14 Introducing AIP and F5 (Grasping) in Monkey F5 - grasp commands in premotor cortex Giacomo Rizzolatti AIP - grasp affordances in parietal cortex Hideo Sakata A key theme of visuomotor coordination: parietal affordances (AIP) drive frontal motor schemas (F5)

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 15 Grasp Specificity in an F5 Neuron Precision pinch (top) Power grasp (bottom) (Data from Rizzolatti et al.)

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 16 The Sakata Protocol

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 17 Grip Selectivity in a Single AIP Cell A cell that is selective for side opposition (Sakata)

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 18 Differential Timing of Activity Peaks in Different AIP Neurons Note the need for a broad database of many cells within each region to see that cells are not just “pattern recognizers” but also have a relationship to the time course of the ongoing behavior.

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 19 Size Specificity in a Single AIP Cell This cell is selective toward small objects, somewhat independent of object type ( Hideo Sakata) Note: Some cells show size specificity; others do not.

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 20 FARS (Fagg-Arbib-Rizzolatti-Sakata) Model Overview Task Constraints (F6) Working Memory (46?) Instruction Stimuli (F2) AIP Dorsal Stream: Affordances IT Ventral Stream: Recognition Ways to grab this “thing” “It’s a mug” PFC AIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it.

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 21 Secondary Somatosensory Cortex (SII) In the grasp versus point comparison in a PET study of humans, we found a marked increase of activity in the secondary somatosensory cortex (SII). Ablation of SII in non-human primates results in decrements in tactile discrimination and impaired tactile learning. Focal lesions of the parietal operculum in humans characteristically produce tactile agnosia without loss of simple tactile sensation, or motor control. This deficit can include the inability to sort objects based on size or shape, although sorting on texture is preserved. The model relates the augmented response to higher order tactile feedback or tactile expectation.

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 22 Motor Commands, Expectations, and Feeedback F5 (grasp type) MI hand (muscle assemblies) (elementary sensory features) SI SII expectation motor commands sensory info (sensory hyperfeatures) A7 (internal model)

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 23 Interaction of AIP and F5 During the Sakata Task Activation Connection Inhibitory Connection Priming Connection AIP precision-related cell AIP power-related cell

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 24 The Problem of Serial Order in Behavior (Karl Lashley) If we tried to learn a sequence like A  B  A  C by reflex chaining, what is to stop A triggering B every time, to yield the performance A  B  A  B  A  ….. (or we might get A  B+C  A  B+C  A  …..) A solution: Store the “action codes” (motor schemas) A, B, C, … in one part of the brain (F5 in FARS) and have another area (pre-SMA in FARS) hold “abstract sequences” and learn to pair the right action with each element: (pre-SMA): x1  x2  x3  x4 abstract sequence (F5): A B C action codes/motor schemas Hypothesis: The “Sakata-Protocol Sequencing” is not mediated within F5 --Sequences are stored in pre-SMA and administered by the Basal Ganglia (BG)

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 25 Basal Ganglia Anatomy in the Rat From Prescott et al., HBTNN 2e, to appear

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 26 A 2-Function View of the Basal Ganglia Skeletomotor Pathway Cortex Indirect Pathway Direct Pathway

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 27 Bischoff-Grethe Sequencing Model Pre-SMA SMA-Proper Motor Cortex Basal Ganglia

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 28 The “Visual Front End” of the FARS Model Visual Cortex Parietal Cortex VIP PIPAIP F4 How (dorsal) IT What (ventral) (position) (shape, size, orientation) (object/grasp transform) F5 (grasp type) (arm goal position)

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 29 Positioning F2, F6 and Areas 46 and SII in Monkey

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 30 Prefrontal Influences on F5 F4 Inferior Premotor Cortex F5 (grasp type) F6 46 (arm goal position) F2 (abstract stimuli) pre-SMA Dorsal premotor cortex Frontal Cortex

Michael Arbib CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 31 The Complete FARS Model