Conceptual Processing and Semantics (also more on the Mirror Neuron/Action Perception System) A.P. Saygin Cogs 172 UCSD

The Nature of Conceptual Representations How is semantics organized? Sensory/Functional Visual properties in a visual subsystem, auditory properties in an auditory subsystem, and so on… Domain-specific Organized by domains such as animals, plants, faces… Independent of sensory and motor representations.

Semantic Categories Specific deficit for living things JBR, SBR, KB, ING (Warrington & McCarthy, 1983; Warrington & Shallice, 1984); TOB - auditory (McCarthy & Warrington, 1988) Tasks used Producing definitions Matching names and pictures Picture naming Elizabeth Warrington

Semantic Categories Specific deficit for artifacts/non-living things YOT (Warrington & McCarthy, 1987) Tasks Matching a spoken name to one of several pictures (among an array of choices)

Semantic Categories and Deficits Specific deficit for artifacts/non-living things YOT (Warrington & McCarthy, 1987) Task: Matching a spoken name to one of several pictures (among an array of choices) Can be more specific (e.g., musical instruments, food) Tranel, Damasio & Damasio (1997): Meta-analysis of category specific deficits

Semantic Categories and Deficits Patient EW (Caramazza & Shelton) Naming test 100% Fruits & vegetables 92% Tools 34% Animals Other tasks as well PS: Caramazza overheard saying “The lesion can be in the toe!” – interested in the theory, not the neural bases

Living Nonliving Cases Animals Fruits/ Vegetables Artifacts X Warrington & Shallice (1984), Silveri & Gainotti (1988) Warrington & McCarthy (1983), Sheridan & Humphreys (1993) Hart & Gordon (1992), Caramazza & Shelton (1998) Hillis & Caramazza (1991) Hart, Berndt & Caramazza (1985), Farah & Wallace (1992) Not observed (so far) X: Disproportionate impairment See readings Caramazza and Martin et al. for more examples

How are living things and artifacts different? Note: Warrington et al. did not necessarily view the patient deficits as evidence for domain-specificity Instead discussed modality-specificity of semantic system Visual vs. Functional We may distinguish living things more by visual properties - How they look, how they move, colour, etc. Artifact knowledge more functional - What is it for? How is it used? Important to make sure the task does not confound the studies

However, Caramazza argues, there are patients whose deficits reveal category effects even when the tasks require functional judgments. (Visual judgments: Do ducks have long ears? Functional judgment: Are roses given on Valentine’s day?) Does this argument convince you? Even if the judgment is functional, could the results be consistent with the concept requiring sensory (here, visual) representations to be used? Could this explain the patient data? Living NonLiving Patient Visual Functional EW 67 74 96 99 GR 55 58 91 84 FM 73 69 JEN 42 51 75 83 DB 100 RC 70 79 90

Emergent Category Structure Categories can “emerge” from a system that isn not domain-specific. Semantic space is “lumpy” – i.e., things that move in a certain way tend to be made of similar materials. Items that share many properties ~ highly correlated properties will cluster in the brain What is the difference? - Domain-specific vs. emergent categories

Alex Martin et al. Sensorimotor/Functional model Distributed networks Embodiment Animate objects –visual properties Tools – functional properties

Sensory/Functional Account Evidence for modality effects Patients who cannot recognize objects in one modality, but can in another (Shallice, 1987; McCarthy & Warrington, 1988; Saffran & Schwartz, 1994)

Neuroimaging Studies in the Healthy Brain Silently naming animal drawings: occipital Verbalizing the action: Motor and superior temporal areas Verbalizing the color: Ventral areas near regions involved in color perception Silently naming tool drawings: motor See Martin, Ungerleider & Haxby chapter – This is pretty old for neuroimaging and methods have advanced since then but these studies illustrate the general points (& the debate is ongoing)

Viewing and naming manipulable objects activates left premotor and parietal cortex (Chao & Martin, 2000)

Why? We talked about mirror neuron theory (and biological motion). A brief overview again

Understanding Others Theory Neuroscience 3rd person (“other”) actions: Often visually perceived 1st person (“self”) actions: Rarely visually perceived Yet we are able to know what we perceive (Barresi & Moore, 1996, BBS) Neuroscience Visual hypothesis – Based on a visual analysis of the elements (body parts, objects, motion, etc). No sensorimotor involvement is required Simulation hypothesis – Analysis-by-synthesis. We map the visual representation onto our own sensorimotor representations

Frontal and posterior cortical regions are heavily interconnected Perception and Action Frontal and posterior cortical regions are heavily interconnected “Networks”

Mirror Neurons Frontal area F5 in the macaque (Rizzolatti lab, ca 1996) Later also found in parietal cortex Neuroimaging in humans: The mirror neuron system Inferior frontal cortex Inferior parietal cortex Superior temporal sulcus (STS)

Theory An individual can understand others’ actions by mapping the visual representation of the observed action onto his/her sensorimotor representation of the same action, thus using his/her own embodied experience of the world. “An action is understood when its observation causes the motor system of the observer to ‘resonate’ ” (Rizzolatti, Fogassi, & Gallese, 2001).

Mirror Neuron System Seen as a possible neural resource for linking between self and other, and for empathy, and social cognition. The specific properties of the system and the possible involvement of the mirror neuron system in brain disorders (from autism to psychiatric disorders) is currently of interest in cognitive neuroscience.

Mirror Neuron System Seen as a possible neural resource for linking between self and other, empathy, and social cognition. That’s important stuff. So mirror neurons have been popular! The involvement of the mirror neuron system in associated disorders (from autism to psychiatric disorders) is currently of intense interest. The mirror neuron system is: - Not a silver bullet Have sometimes been overhyped Functional properties still need to be specified

Embodiment We can be more agnostic about “mirror neurons” per se From mirror neurons to theories of perceptual and conceptual processing…

Mirror-like responses: Empathy for pain Pain observation increases activity in affective pain processing areas Singer et al., 2004, Science

Mirror-like responses: Disgust or olfactory stimulation Wicker et al., 2003, Neuron

Mirror-like responses: Listening to and producing speech UCLA/UCSD Study Listen to monosyllables /pa/ /gi/ Produce monosyllables /pa/ /gi/ Overlap found in premotor cortex. Note: This is a more superior area than Broca’s area. Wilson et al., 2004, Nature Neurosci.

Link to Semantics Activations for tools Chao & Martin could reflect automatic engagement of action/tool-use related systems Frontal ~ canonical neurons (manipulable objects)

Sensorimotor/Functional Theory of Semantics Semantic representations are related to sensorimotor properties Automatic activation of semantic representations Imaging, patient, TMS and behavioral studies Color - ventral temporal anterior to color perception areas Motion - lateral temporal lobe Superior: biological motion Middle: artifact motion Actions: Motor, premotor, parietal areas Modulations in sensory or motor regions can be evoked using linguistic stimuli Action words (Hauk, Johnsrude, & Pulvermuller, 2004; Tettamanti et al., 2005); odour-related words (Gonzalez et al., 2006)

Somatotopy in action observation Buccino et al 2001, Eur. J Neurosci

Somatotopy in Language Semantics Foot action words Hand action words Mouth action words Hauk, et al., 2004, Neuron

Motion Semantics and Vision FMRI study – we localized motion-sensitive area V5/MT+ and face-sensitive area FFA Motion sentences: “I drove from Modesto to Fresno” Static sentences: “Modesto and Fresno are in California” Fictive motion sentences: Use of a motion verb when there is no physical occurrence of motion (Talmy 1996, 2000) “The highway runs from Modesto to Fresno” “The mountain range goes from Mexico to Canada”

Saygin, McCullough, Alac & Emmorey, 2010, J. Cognitive Neuroscience

Readings: Caramazza, 1999 and Martin, Ungerleider & Haxby, 1999 What do you think?