Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 1 The Aims of the Course: We will use the challenge of understanding the mechanisms of visuomotor coordination and action recognition in the monkey brain to provide a structured set of goals for our mastery of Brain Theory: modeling interactions of components of the brain, especially more or less realistic biological neural networks localized in specific brain regions and Connectionism in both Artificial intelligence (AI) and Cognitive psychology: modeling artificial neural networks -- networks of trainable “quasi-neurons” -- to provide “parallel distributed models” of intelligence in humans, animals and machines Tools: Gaining an understanding of the use of NSLJ (Neural Simulation Language in Java) for simulating and analyzing neural networks Michael Arbib: CS564 - Brain Theory and Artificial Intelligence

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 2 CS 564: Brain Theory and Artificial Intelligence URL: for syllabus, instructor and TA information, handouts, homework and grades DEN URL: Instructor: Michael Arbib; (Office hours: Tuesdays, HNB 03) TAs: Erhan Oztop, Salvador Marmol, This course provides a basic understanding of brain function, of artificial neural networks which provide tools for a new paradigm for adaptive parallel computation, and of the Neural Simulation Language NSLJ which allows us to simulate biological and artificial neural networks No background in neuroscience is required, nor is specific programming expertise, but knowledge of Java will enable students to extend the NSLJ functionality in interesting ways

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 3 Texts Texts: MA Arbib, 1989, The Metaphorical Brain 2: Neural Networks and Beyond, Wiley-Interscience TMB2 is being sent over to "The paper clip" for duplication. Students can purchase it there for $16 plus tax. A Weitzenfeld, MA Arbib and A Alexander, 2002, NSL Neural Simulation Language, MIT Press (in press) [ An old version is at Book/TOC.htm. A new version will be posted in a week or two.] Other Required Reading: will be posted on the Website – starting with the Mirror Neuron Proposal Supplementary reading: MA Arbib, Ed, 1995, The Handbook of Brain Theory and Neural Networks, MIT Press (paperback) Michael A Arbib, and Jeffrey Grethe, Editors, 2001, Computing the Brain: A Guide to Neuroinformatics, San Diego: Academic Press (in press)

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 4 Grading One mid-term and a final will cover the entire contents of the readings as well as the lectures Students will be organized into 5 groups, each working together on a semester-long project The final exam will cover all of the course, but emphasizing material not covered in the mid-term Distribution of Grades:  NSL assignments and other homework: 25%;  Mid-term: 20%;  Project 30%;  Final Exam: 25%

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 5 5 Specific Aims for the Mirror Neuron Proposal 5 Specific Group Projects for CS 564 Development of the Mirror System: 1. Development of Grasp Specificity in F5 Motor and Canonical Neurons 2. Visual Feedback for Grasping: A Possible Precursor of the Mirror Property Recognition of Novel and Compound Actions and their Context: 3. The Pliers Experiment: Extending the Visual Vocabulary 4. Recognition of Compounds of Known Movements 5. From Action Recognition to Understanding: Context and Expectation

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 6 Send to by Noon on Tuesday September 4, 2001 Line 1: Your name then a colon (:) then your address Line 2: Your Department (and your location if you are off-campus) Line 3: Any special skills you bring to the course Lines 4+: Your top 2 or 3 choices for a Project topic from the list: 1. Development of Grasp Specificity 2. Visual Feedback for Grasping 3. The Pliers Experiment 4. Recognition of Compound Movements 5. Context and Expectation

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 7 Michael Arbib:CS564 - Brain Theory and Artificial Intelligence Lecture 1 Introduction and Brain Overview Reading Assignments: TMB2: Chapter 1 Mirror Neuron Proposal* * Read this “lightly” the first week of class We will master it and much more as the class proceeds Your initial task is to review the 5 specific aims, and pick the 2 you find most interesting One of these will probably become your project goal for the semester

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 8 Opposition Spaces and Virtual Fingers The goal of a successful preshape, reach and grasp is to match the opposition axis defined by the virtual fingers of the hand with the opposition axis defined by an affordance of the object (Iberall and Arbib 1990)

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 9 Visual Control of Grasping in Macaque Monkey F5 - grasp commands in premotor cortex Giacomo Rizzolatti AIP - grasp affordances in parietal cortex Hideo Sakata A key theme of visuomotor coordination: parietal affordances (AIP) drive frontal motor schemas (F5)

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 10 Grasp Specificity in an F5 Neuron Precision pinch (top) Power grasp (bottom) (Data from Rizzolatti et al.)

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 11 FARS (Fagg-Arbib-Rizzolatti-Sakata) Model Overview Task Constraints (F6) Working Memory (46?) Instruction Stimuli (F2) AIP Dorsal Stream: Affordances IT Ventral Stream: Recognition Ways to grab this “thing” “It’s a mug” PFC F5 AIP extracts affordances - features of the object relevant to physical interaction with it. Prefrontal cortex provides “context” so F5 may select an appropriate affordance

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 12 Syllabus Overview 1  Introduction [Proposal] {Background: TMB Chapter 1}  Charting the Brain 1 [TMB 2.4]  The Brain as a Network of Neurons [TMB Section 2.3]  Visual Preprocessing [TMB 3.3]  Adaptive networks: Hebbian learning, Perceptrons; Landmark learning [TMB 3.4] [NSLbook]  Hebbian Learning and Visual plasticity; Self-organizing feature maps; [NSLJ] Kohonen maps  Higher level vision 1: object recognition {Background TMB 5.2}  Introduction to NSL: modules; SCS schematic capture system; Maxselector model[NSLbook] {Homework}

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 13 Syllabus Overview 2  Schemas for Reaching and Grasping; Affordances [TMB 2.2, 5.3] {Background TMB 2.1, 52}  Charting the Brain 2  The FARS model of control of grasping 1: Population coding  The FARS model of control of grasping 2: Sequence learning and the basal ganglia  The FARS model of control of grasping 3: Working Memory

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 14 Mirror Neurons Rizzolatti, Fadiga, Gallese, and Fogassi, 1995: Premotor cortex and the recognition of motor actions Mirror neurons form the subset of grasp-related premotor neurons of F5 which discharge when the monkey observes meaningful hand movements made by the experimenter or another monkey. F5 is endowed with an observation/execution matching system [The non-mirror grasp neurons of F5 are called F5 canonical neurons.]

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 15 Computing the Mirror System Response The FARS Model: Recognize object affordances and determine appropriate grasp. The Mirror Neuron System (MNS) Model: We must add recognition of  trajectory and  hand preshape to  recognition of object affordances and ensure that all three are congruent. There are parietal systems other than AIP adapted to this task.

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 16 The Mirror Neuron System (MNS) Model

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 17 STS hand shape recognition Model Matching Precision grasp Hand Configuration Classification Step 2: The feature vector generated by the first step is used to fit a 3D-kinematics model of the hand by the model matching module. The resulting hand configuration is sent to the classification module. Color Coded Hand Feature Extraction Step 1 of hand shape recognition: system processes the color-coded hand image and generates a set of features to be used by the second step

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 18 Virtual Hand/Arm and Reach/Grasp Simulator A power grasp and a side grasp A precision pinch

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 19 Core Mirror Circuit Hand state Mirror Neurons (F5mirror) Association (7b) Neurons Mirror Feedback Object affordance Mirror Neuron Output Motor Program (F5 canonical) Hand shape recognition & Hand motion detection Hand-Object spatial relation analysis Object affordance - hand state association Object Affordances Action recognition (Mirror Neurons) Motor program Motor execution Mirror Feedback Integrate temporal association Motor program F5canonical F5mirror

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 20 Power and precision grasp resolution (a) (b) Power Grasp Mirror Neuron Precision Pinch Mirror Neuron

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 21 Syllabus Overview 3  Reinforcement learning and motor control; [NSLJ] Conditional motor learning  Adaptive networks: Gradient descent and backpropagation [TMB 82]  [NSLJ] Backprop: a How to run the model; b How to write the model [NSLbook]  NeuroBench and the NeuroHomology Database

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 22 Syllabus Overview 4  The MNS1 Model 1: Hand Recognition  Systems concepts; Feedback and the spinal cord [TMB 31, 32]  The MNS 1 Model 2: Simulating the kinematics and biomechanics of reach and grasp  The MNS1 Model 3: Modeling the Core Mirror Neuron Circuit

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 23 5 Specific Aims for the Mirror Neuron Proposal 5 Specific Group Projects for CS 564 Development of the Mirror System: 1. Development of Grasp Specificity in F5 Motor and Canonical Neurons 2. Visual Feedback for Grasping: A Possible Precursor of the Mirror Property Recognition of Novel and Compound Actions and their Context: 3. The Pliers Experiment: Extending the Visual Vocabulary 4. Recognition of Compounds of Known Movements 5. From Action Recognition to Understanding: Context and Expectation

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 24 Rizzolatti, Fadiga, Matelli, Bettinardi, Perani, and Fazio: Broca's region is activated by observation of hand gestures: a PET study. PET study of human brain with 3 experimental conditions:  Object observation (control condition)  Grasping observation  Object prehension. The most striking result was highly significant activation in the rostral part of Broca's area. A Human Mirror System A key language area!!!

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 25 A New Approach to the Evolution of Human Language Rizzolatti, G, and Arbib, M.A., 1998, Language Within Our Grasp, Trends in Neuroscience, 21(5): : The Mirror System Hypothesis: Human Broca’s area contains a mirror system for grasping which is homologous to the F5 mirror system of monkey, and this provides the evolutionary basis for language parity - i.e., an utterance means roughly the same for both speaker and hearer.  This adds a neural “missing link” to the tradition that roots speech in a prior system for communication based on manual gesture. Beyond the Mirror: Seeing F5 as part of a larger mirror system, then extending our understanding via imitation to language-readiness.  This topic will be the jumping off point for the Spring 2002 version of CS 664, which will be taught by Professors Arbib and Itti.

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 26 Syllabus Overview 5  Control of eye movements [TMB 6.2]  Basal Ganglia and Control of eye movements - Dominey [NSLbook]  Dopamine and Sequence Learning  Abstract models of Sequence Learning  Extending the FARS model to mirror neurons and language  Project Reports 1, 2,3  Project Reports 4,5; Concluding Discussion

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 27 A Conceptual Restructuring of the Syllabus 1 Overview + Basic Concepts of Neurons and Schemas 1. Modeling the Mirror System: Setting Goals for the Course [Proposal] {Background: TMB Chapter 1} 2. Charting the Brain 1 [TMB 2.4] 3. The Brain as a Network of Neurons [TMB Section 2.3] 9. Schemas for Reaching and Grasping; Affordances [TMB 2.2, 5.3] {Background TMB 2.1, 5.2} 10. Charting the Brain 2

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 28 A Conceptual Restructuring of the Syllabus 2 Vision  Preprocessing 4. Visual Preprocessing [TMB 3.3] 6. Hebbian Learning and Visual plasticity; Self-organizing feature maps; [NSLJ] Kohonen maps  Low-Level Vision Supplementary reading not covered in lectures: Perceptual and motor schemas: Discussion of visual segmentation based on, e.g., edge and texture cues; Stereoscopic vision [TMB 7.1]; Motion perception and optic flow [TMB 7.2].  High-Level Vision 7. Higher level vision: object recognition {Background TMB 5.2} 18. The MNS1 Model 1: Hand Recognition

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 29 A Conceptual Restructuring of the Syllabus 3 Motor Control 19. Systems concepts; Feedback and the spinal cord [TMB 3.1, 3.2] 20. The MNS 1 Model 2: Simulating the kinematics and biomechanics of reach and grasp Visuo-Motor Integration 11. The FARS model of control of grasping 1: Population coding 12. The FARS model of control of grasping 2: Sequence learning and the basal ganglia 13. The FARS model of control of grasping 3: Working Memory 21. The MNS1 Model 3: Modeling the Core Mirror Neuron Circuit 22. Control of eye movements [TMB 6.2] 23. Basal Ganglia and Control of eye movements - Dominey [NSLbook]

Arbib: CS564 - Brain Theory and Artificial Intelligence, USC, Fall Lecture 1. Introduction and Overview 30 A Conceptual Restructuring of the Syllabus 4 Adaptive Networks 5. Adaptive networks: Hebbian learning, Perceptrons; Landmark learning [TMB 3.4] [NSLbook] 14. Reinforcement learning and motor control; [NSLJ] Conditional motor learning 15. Adaptive networks: Gradient descent & backpropagation [TMB 8.2] 24. Dopamine and Sequence Learning 25. Abstract models of Sequence Learning Higher Cognitive Functions 26. Extending the FARS model to mirror neurons and language Supplementary reading: Memory and Consciousness [TMB 8.3] Simulation and Neuroinformatics 8. Introduction to NSL: modules; SCS schematic capture system; Maxselector model [NSLbook] 16. [NSLJ] Backprop: a. How to run and write the model [NSLbook] 17. NeuroBench and the NeuroHomology Database