Lecture Overview Regier System: Limitations Image Schemas: Recap Force Dynamic Schemas: Recap Sensory-Motor Schemas –Evidence in Primates –Evidence in Humans Do motor schemas play a role in language? A Computational Model of Motor Schemas Learning Hand Action terms (Bailey) Cultural Schemas and frames

Limitations Scale Uniqueness/Plausibility Grammar Abstract Concepts Inference Representation

on construction: Grammar and Inference One sense of on SCHEMA TR-LM ROLES tr: trajector lm: landmark SCHEMA Contact ROLES r1: bounded-region r2: region SCHEMA Support ROLES supporter: bounded-region supportee: bounded-region CONSTRUCTION On FORM: Word self.f.orth  “on” MEANING EVOKES TR-LM as trlm EVOKES Contact as c EVOKES Support as s trlm.tr  c.r1 trlm.lm  c.r2 trlm.tr  s.supportee trlm.lm  s.supporter

Image Schemas in Language You wake out of a deep sleep and peer out from beneath the covers into your room. You gradually emerge out of your stupor, pull yourself out from under the covers, climb into your robe, stretch out your limbs, and walk in a daze out of the bedroom and into the bathroom. You look in the mirror and see your face staring out at you. You reach into the medicine cabinet, take out the toothpaste, squeeze out some toothpaste, put the toothbrush into your mouth, brush your teeth in a hurry, and rinse out your mouth. (Johnson 1987)

Image Schemas in Language You wake out of a deep sleep and peer out from beneath the covers into your room. You gradually emerge out of your stupor, pull yourself out from under the covers, climb into your robe, stretch out your limbs, and walk in a daze out of the bedroom and into the bathroom. You look in the mirror and see your face staring out at you. You reach into the medicine cabinet, take out the toothpaste, squeeze out some toothpaste, put the toothbrush into your mouth, brush your teeth in a hurry, and rinse out your mouth. (Johnson 1987)

Fictive Motion and Image Schemas (Talmy, Teenie Matlock) This fence goes from the plateau to the valley I looked out past the steeple The vacuum cleaner is down around behind the clothes-hamper The scenery rushed past us as we drove along

Lecture Overview Regier System: Limitations Image Schemas: Recap Force Dynamic Schemas: Recap Sensory-Motor Schemas –Evidence in Primates –Evidence in Humans Do motor schemas play a role in language? A Computational Model of Motor Schemas Learning Hand Action terms (Bailey) Cultural Schemas and frames

Force Dynamics, modals and causatives A gust of wind made the pages of my book turn. The appearance of the headmaster made the pupils calm down. The breaking of the dam let the water flow from the storage lake. The abating of the wind let the sailboat slow down.

Talmy’s force dynamic schemas Two entities in a forceful interaction Agonist (foregrounded) Antagonist (has an effect on the Agonist) Each entity exerts a force on the other An entity has an tendency towards rest (inaction) toward motion Opposed forces have different relative strength Depending on the relative strength the result of the interaction could make the Agonist move or be at rest.

Schematic Representation (Talmy)

FD Patterns A gust of wind made the pages of my book turn. The appearance of the headmaster made the pupils calm down. The breaking of the dam let the water flow from the storage lake. The abating of the wind let the sailboat slow down.

Semantic fieldForce-dynamics representation PhysicalThe ball kept rolling along the green Physical/psychological John can't go out of the house Intra-psychologicalHe refrained from closing the door Intra-psychological (lexicalized) She's civil to him Socio-psychological She gets to go to the park

Force Dynamics of Cause Philip Wolff (2003)

An Experiment with Force Dynamics

Language Specific Bias

Simulation-based language understanding Analysis Process Semantic Specification “Harry walked into the cafe.” Utterance CAFE Simulation Belief State General Knowledge Constructions construction W ALKED form self f.phon  [wakt] meaning : Walk-Action constraints self m.time before Context.speech-time self m..aspect  encapsulated

Simulation specification A simulation specification consists of: - schemas evoked by constructions - bindings between schemas

Closed Class vs. Open Class terms Image Schematic and Force Dynamic Patterns are expressed by closed class terms in language –Prepositions (in, on, into, out) –Modals and causatives (make, let, might, prevent) How about open class terms? –Verbs and Event descriptions – Motor Schemas - Embodied –Is there evidence for motor schemas and if so are they used in language? Frames – Composed from Image and motor schemas - Cultural

Verb + Image Schemas in Language You wake out of a deep sleep and peer out from beneath the covers into your room. You gradually emerge out of your stupor, pull yourself out from under the covers, climb into your robe, stretch out your limbs, and walk in a daze out of the bedroom and into the bathroom. You look in the mirror and see your face staring out at you. You reach into the medicine cabinet, take out the toothpaste, squeeze out some toothpaste, put the toothbrush into your mouth, brush your teeth in a hurry, and rinse out your mouth. (Johnson 1987)

Lecture Overview Regier System: Limitations Image Schemas: Recap Force Dynamic Schemas: Recap Sensory-Motor Schemas –Evidence in Primates –Evidence in Humans Do motor schemas play a role in language? A Computational Model of Motor Schemas Learning Hand Action terms (Bailey) Cultural Schemas and frames

Coordination of Pattern Generators

Coordination PATTERN GENERATORS, separate neural networks that control each limb, can interact in different ways to produce various gaits. –In ambling (top) the animal must move the fore and hind leg of one flank in parallel. –Trotting (middle) requires movement of diagonal limbs (front right and back left, or front left and back right) in unison. –Galloping (bottom) involves the forelegs, and then the hind legs, acting together

Sensory-Motor Schemas A sensory (perceptual) schema determines whether a given situation is present in the environment. – Object Detection – Spatial relation recognition Execution of current plans is made up of motor schemas which are akin to control systems but distinguished by the fact that they can be combined to form coordinated control programs Sensory and Motor Schemas are closely coupled circuits sensory- motor schemas.

Preshaping While Reaching to Grasp

Lecture Overview Regier System: Limitations Image Schemas: Recap Force Dynamic Schemas: Recap Sensory-Motor Schemas –Evidence in Primates –Evidence in Humans Do motor schemas play a role in language? A Computational Model of Motor Schemas Learning Hand Action terms (Bailey) Cultural Schemas and frames

The neural theory Human concepts are embodied. Many concepts make direct use of the sensory-motor capacities of our body-brain system. Many of these capacities are also present in non-human primates. Let us look at concepts that make use of our sensory-motor capacities, ex. Grasp.

Rizzolatti et al A New Picture

The fronto-parietal networks Rizzolatti et al F5-AIP F4-VIP F5-PF 3 circuits for grasp Neural Tracers

Area F5 General Purpose Neurons: General Grasping General Holding General Manipulating

A A Grasping with the mouth B B Grasping with the cl. hand C C Grasping with the ipsil. hand General Purpose Neurons in Area F5 (Rizzolatti et al. 1988)

General Purpose Neurons Achieve Partial Universality: Their firing correlates with a goal-oriented action of a general type, regardless of effector or manner.

Area F5c Convexity region of F5: Mirror neurons

F5c-PF Rizzolatti et al. 1998

F5 Mirror Neurons Gallese and Goldman, TICS 1998 tools

(Rizzolatti et al. Cog Brain Res 1996) Observed Action Executed Action Strictly congruent mirror neurons (~30%)

Category Loosening in Mirror Neurons (~60%) (Gallese et al. Brain 1996) A [C] is Observe (Execute) Precision Grip (Prototype) B [D] is Observe (Execute) Whole Hand Pre-hension

The F5c-PF circuit Links premotor area F5c and parietal area PF (or 7b). Contains mirror neurons. Mirror neurons discharge when: Subject (a monkey) performs various types of goal- related hand actions and when: Subject observes another individual performing similar kinds of actions

Phases Area F5 contains clusters of neurons that control distinct phases of grasping: opening fingers, closing fingers. Jeannerod, et al., 1995; Rizzolatti, et al., 2001.

Mirror Neurons Achieve Partial Universality, since they code an action regardless of agent, patient, modality (action/observation/hearing), manner, location. Partial Role Structure, since they code an agent role and a purpose role. The Agent Role: In acting, the Subject is an agent of that action. In observing, the Subject identifies the agent of the action as having the same role as he has when he is acting – namely, the agent role. The Purpose Role: Mirror neurons fire only for purposeful actions.

Mirror Neurons Achieve Category tightening and loosening

The F4-VIP circuit

The F4-VIP Circuit Links premotor area F4 and parietal area VIP. Transforms the spatial position of objects in peri-personal space into motor programs for interacting with those objects. Examples: Reaching for the objects, or moving away from them with various parts of your body such as the arm or head.

Area F4 Arm reaching Head turning

Somato-Centered Bimodal RFs in area F4 (Fogassi et al. 1996)

(Graziano et al. 1999)

Somato-Centered Bimodal RFs in area VIP (Colby and Goldberg 1999)

Somato-Centered Bimodal RFs in area F4 (Fogassi et al. J Neurophysiol 1996)

AIP and F5 (Grasping) in Monkey F5 - grasp commands in premotor cortex Giacomo Rizzolatti AIP - grasp affordances in parietal cortex Hideo Sakata

Size Specificity in a Single AIP Cell This cell is selective toward small objects, somewhat independent of object type ( Hideo Sakata) Note: Some cells show size specificity; others do not.

Summary of Fronto-Parietal Circuits Motor-Premotor/Parietal Circuits PMv (F5ab) – AIP Circuit “grasp” neurons – fire in relation to movements of hand prehension necessary to grasp object F4 (PMC) (behind arcuate) – VIP Circuit transforming peri-personal space coordinates so can move toward objects PMv (F5c) – PF Circuit F5c different mirror circuits for grasping, placing or manipulating object Together suggest cognitive representation of the grasp, active in action imitation and action recognition

Evidence in Humans for Mirror, General Purpose, and Action-Location Neurons Mirror: Fadiga et al. 1995; Grafton et al. 1996; Rizzolatti et al. 1996; Cochin et al. 1998; Decety et al. 1997; Decety and Grèzes 1999; Hari et al. 1999; Iacoboni et al. 1999; Buccino et al. 2001, Graziano 2005 General Purpose: Perani et al. 1995; Martin et al. 1996; Grafton et al. 1996; Chao and Martin Action-Location: Bremmer, et al., 2001.

Somatotopy of Action Observation Foot Action Hand Action Mouth Action Buccino et al. Eur J Neurosci 2001

The Mirror System in Humans BA6

MULTI-MODAL INTEGRATION The premotor and parietal areas, rather than having separate and independent functions, are neurally integrated not only to control action, but also to serve the function of constructing an integrated representation of: (a)Actions, together with (b)objects acted on, and (c)locations toward which actions are directed. In these circuits sensory inputs are transformed in order to accomplish not only motor but also cognitive tasks, such as space perception and action understanding.

Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model FARS (Fagg-Arbib-Rizzolatti-Sakata) Model Task Constraints (F6) Working Memory (46?) Instruction Stimuli (F2) AIP Dorsal Stream: Affordances IT Ventral Stream: Recognition Ways to grab this “thing” “It’s a mug” PFC AIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it. F5

Hypothetical coordinated control program for reaching and grasping Dashed lines — activation signals; solid lines — transfer of data. (Adapted from Arbib 2004) Perceptual Schemas Motor Schemas

Modeling Motor Schemas Relevant requirements (Stromberg, Latash, Kandel, Arbib, Jeannerod, Rizzolatti) –Should model coordinated, distributed, parameterized control programs required for motor action and perception. –Should be an active structure. –Should be able to model concurrent actions and interrupts. –Should model hierarchical control (higher level motor centers to muscle extensor/flexors. Computational model called x-schemas (