Electrophysiology as a Brain Measure of Perceptual Sensitivity and Abstraction

/kæt/ what? when? where? Electrophysiology How it works Mapping tool Discrimination Categorization Prediction Timing

How electrophysiology works

Electrical Activity

Probing Electrical Activity Interfere with it Record it indirectly Record it directly

Zap! Transcranial Magnetic Stimulation K. L. Sakai et al. 2002, Neuron

Verb generation Verb generation after 15 min practice Raichle & Posner, Images of Mind cover image Bang! Functional Magnetic Resonance Imaging (fMRI)

MRI studies brain anatomy. Functional MRI (fMRI) studies brain function. MRI vs. fMRI Source: Jody Culham, fMRI for Newbies

Electroencephalography (EEG)

Event-Related Potentials (ERPs) s1s2s3 John islaughing.

Phillips, Kazanina, & Abada (2005) Cog. Br. Res. The producers knew that the actress wished that … The producers knew which jokes the actress wished that..

ERP Studies Cost ($$) –Relatively cheap equipment and maintenance Time Investment –Materials: e.g., 20 syllables; 128 x 4 target sentences; 256 fillers –Acquisition: 2-3 hours x participants –Analysis: 1-2 hours/person preprocessing; rest automatizable Strengths –Unbeatable temporal resolution –Easy to combine across participants –Movement possible; longer studies more tolerable –Interpretable results Disadvantages –Scalp topography ≠ localization –… and limited sex appeal

Brain Magnetic Fields (MEG) Brain magnetic fields recorded fully non-invasively by arrays of SQUID* detectors [*Superconducting QUantum Interference Device]

V skull CSF tissue MEG EEG B - noninvasive measurement - direct measurement. scalp recording surface current flow orientation of magnetic field Origin of the signal

How small is the signal? Earth field Urban noise Contamination at lung Heart QRS Muscle Fetal heart Spontaneous signal (  -wave) Signal from retina Intrinsic noise of SQUID Intensity of magnetic signal(T) Evoked signal Biomagnetism EYE (retina) Steady activity Evoked activity LUNGS Magnetic contaminants LIVER Iron stores FETUS Cardiogram LIMBS Steady ionic current BRAIN (neurons) Spontaneous activity Evoked by sensory stimulation SPINAL COLUMN (neurons) Evoked by sensory stimulation HEART Cardiogram (muscle) Timing signals (His Purkinje system) GI TRACK Stimulus response Magnetic contaminations MUSCLE Under tension

160 SQUID whole-head array pickup coil & SQUID assembly

Sensor layout: recording from 160 channels Response peak at 98ms after onset of an auditory stimulus, in the left and right temporal lobes.

Magnetic source imaging (MSI): MEG + MRI Dipole fit at response peak, 98ms after onset of stimulus

(Halgren et al. 2002)

MEG Studies Cost ($$) –Expensive equipment; little maintenance; liquid helium supply Time Investment –Materials: e.g., 20 syllables; 600 x 3 target sentences; 400 fillers –Acquisition: 1-2 x hours x 12 participants (+ structural MRI) –Analysis: similar to EEG, more individual-specific analyses Strengths –Unbeatable temporal resolution; easy set-up –Possibility of localization Disadvantages –Combining across individuals more difficult than in ERP studies –Longer studies, movement limited –Inverse problem for localization is very hard –Sensitivity depends on orientation and depth of source –Horrendously difficult with children, challenging w/ small heads

Electrophysiology in Syntax/Semantics 1. Classification Tool 2. Sensitive Timing Measure Sue takes her coffee with cream and socks. N400 - semantic anomaly The plane took we to paradise.P600 - syntactic anomaly The hungry guests helped himself to food.P600 Prerequisite: response components sensitive to distinct information types Mary praised Max’s of proof the theorem.‘ELAN’, ms No bills that the senators supported ever became law. control The bills that the senators supported ever became law. ~400ms response The bills that no senators supported ever became law. ~400ms response Prerequisite: explicit process models of syntax

Electrophysiology as a Mapping Tool

Sensory Maps Internal representations of the outside world. Cellular neuroscience has discovered a great deal in this area. Sensory Maps Internal representations of the outside world. Cellular neuroscience has discovered a great deal in this area.

Notions of sensory maps may be applicable to some aspects of human phonetic representations… …but there has been little success in that regard, and we shouldn’t expect this to yield much. Vowel Space

Obleser Lahiri J. Cogn. Neurosci., 16, (2004)

M100 Elicited by any well-defined onset Varies with tone frequency Varies with F1 of vowels May vary non-linearly with VOT variation Functional value of time-code unclear No evidence of higher-level representations (Poeppel & Roberts 1996) (Poeppel, Phillips et al. 1997) (Phillips et al. 1995; Sharma & Dorman 1999)

Electrophysiological measures of discrimination

Mismatch Response Latency: msec. Many-to-one ratio between standards and deviants X X X X X Y X X X X Y X X X X X X Y X X X Y X X X...

Localization of Mismatch Response (Phillips, Pellathy, Marantz et al., 2000) [Radiological view - left is right]

Basic MMN elicitation © Risto Näätänen

Mismatch Negativity (MMN) Sams et al. 1985

Tiitinen et al How does MMN latency, amplitude vary with frequency difference? 1000Hz tone std.

Different Dimensions of Sounds Length Amplitude Pitch …you name it … Amplitude of mismatch response can be used as a measure of perceptual distance

Impetus for Language Studies If MMN amplitude is a measure of perceptual distance, then perhaps it can be informative in domains where acoustic and perceptual distance diverge…

Place of Articulation Acoustic variation: F2 & F3 transitions

Place of Articulation Acoustic variation: F2 & F3 transitions [bæ][dæ]

Place of Articulation Acoustic variation: F2 & F3 transitions [bæ][dæ] within category between category

Place of Articulation Acoustic variation: F2 & F3 transitions [bæ][dæ] within category between category

Categories in Infancy High Amplitude Sucking - 2 month olds Eimas et al vs. 40 ms. VOT - yes 40 vs. 60 ms. VOT - no Infants show contrast, but this doesn’t entail phonological knowledge

Place of Articulation No effect of category boundary on MMN amplitude (Sharma et al. 1993) Similar findings in Sams et al. (1991), Maiste et al. (1995)

but…

Näätänen et al. (1997) e e/ö ö õ o

Relevance to Learning Models Place of articulation continuum (Lalonde & Werker, 1988) b -- d -- D 3 contrasts Nativeb -- d Non-natived -- D Non-phoneticb 1 -- b 5 Conflicting results –Dehaene-Lambertz 1997 native contrast only –Rivera-Gaxiola et al native + non-native contrasts Dehaene-Lambertz Rivera-Gaxiola Phillips (2001, Cognitive Science)

J. Cogn. Neurosci. 16: (2004) Contrast and Underspecification

Phonetic Category Effects Measures of uneven discrimination profiles Findings are mixed (…and techniques vary) Relies on assumption that effects of contrasts at multiple levels are additive, …plus the requirement that the additivity effect be strong enough to yield a statistical interaction Logic of next set of studies: –Eliminate contribution of lower levels by isolating the many-to-one ratio at a more abstract level of representations –Do this by introducing non-orthogonal variation among standards

Electrophysiological measures of abstraction

J. Cogn. Neuro., 12, (2000) /dæ//tæ/

Design 20ms40ms60ms Fixed Design - Discrimination

Design 0ms 8ms16ms24ms40ms48ms56ms64ms 20ms40ms60ms Fixed Design - Discrimination Grouped Design - Categorization Non-orthogonal within-category variation: precludes grouping via acoustic streaming.

Design 0ms 8ms16ms24ms40ms48ms56ms64ms 20ms40ms60ms Fixed Design - Discrimination Grouped Design - Categorization 20ms 28ms36ms44ms60ms68ms76ms84ms Grouped Design - Acoustic Control

/dæ/ standard vs. /dæ/ deviant

So what … Auditory cortex generator of MMF accesses representations that treat members of the same category as identical No indication of what might be the form of these representations, or where they might be stored MMF generator accesses multiple levels of representation

Sound Groupings (Phillips, Pellathy, & Marantz, 2000) pæ, tæ, tæ, kæ, dæ, pæ, kæ, tæ, pæ, kæ, bæ, tæ...

(Phillips, Pellathy, & Marantz, 2000)

More on features … Alternative account of the findings –No feature-based grouping –Independent MMF elicited by 3 low-frequency phonemes /bæ/ /dæ/ /gæ//pæ/ /tæ/ /kæ/ 29% 4% 87.5% 12.5% (Yeung & Phillips, 2004)

More on features … Next study distinguishes –Phoneme-level frequency –Feature-level status /gæ//bæ//dæ//tæ/ 37.5% 12.5% (Yeung & Phillips, 2004)

More on features … Next study distinguishes –Phoneme-level frequency –Feature-level status /gæ//bæ//dæ//tæ/ 37.5% 12.5% Phoneme-based classification (Yeung & Phillips, 2004)

More on features … Next study distinguishes –Phoneme-level frequency –Feature-level status /gæ//bæ//dæ//tæ/ 37.5% 12.5% Feature-based grouping (Yeung & Phillips, 2004)

More on features … Design –N = 10 –Multiple exemplars, individually selected boundaries –2 versions recorded for all participants, reversing [±voice] value –Acoustic control, with all VOT values in [-voice] range /gæ//bæ//dæ//tæ/ 37.5% 12.5% Feature-based grouping (Yeung & Phillips, 2004)

More on features … Left-anterior channels (Yeung & Phillips, 2004)

Kazanina et al., 2006 Proceedings of the National Academy of Sciences, 103, Nina Kazanina, U. of Bristol

Russian vs. Korean Three series of stops in Korean: –plain (lenis)pa ta ka –glottalized (tense, long)p’at’ak’a –aspiratedp h t h ak h a Intervocalic Plain Stop Voicing: /papo/[pabo]‘fool’ /ku papo/[k  babo]‘the fool’ Plain stops: –Bimodal distribution of +VOT and –VOT tokens –Word-initially: always a positive VOT –Word-medially intervocalically: a voicing lead (negative VOT)

Identification/ Rating Discrimination

Black: p <.05 White: n.s.

Russian vs. Korean MEG responses indicate that Russian speakers immediately map sounds from [d-t] continuum onto categories Korean speakers do not… … despite the fact that the sounds show bimodal distribution in their language Perceptual space reflects the functional status of sounds in encoding word meanings

Dupoux et al. (1999, Percep. Psychophys.) J. Cogn. Neurosci. 12: (2000) EXECTIVE SUITE French: CVC egma  egma Japanese: *CVC egma  eguma

Early (~200ms) Late (~600ms) Dehaene-Lambertz et al. (2000, J. Cogn. Neurosci.) Egma, egma, egma, egma, eguma, …

Mismatch Studies Perceptual experience of sound categories is mapped onto multiple levels of representation, at different degrees of abstraction Abstract category representations can be probed using non-orthogonal variation among sounds in an auditory mismatch paradigm  more-or-less immediate access during speech perception Beware of hasty inferences to normal time course of processing –Recurring standard sound establishes ‘model’ for parsing upcoming sounds –Detecting deviance from model, even at abstract level, may be faster than normal analysis processes –cf. related findings about ELAN (‘Early Left Anterior Negativity’) in syntactic ERP literature (Lau et al., 2006)

Electrophysiology and Prediction

McGurk Effect: auditory [pa] + visual [ka] = perceptual [ta]

Neurosci. Lett. 397: (2006) [aba] [ãba]

Electrophysiology and Timing

syntax > phonology phonology > syntax Science, 280: Lateralized Readiness Potential (LRP)

Prospects Many possibilities for more sophisticated phonetic/phonological studies –Mapping: what are the questions? –Early transforms of acoustic space: already an active area –Discrimination: needs a theory of variable access to levels of detail –Categorization: can be used to investigate learning and abstraction –Prediction: phonotactics is barely touched –Timing: possible LRP studies of computing derived properties