2. Electrophysiology as a Brain Measure of Perceptual Sensitivity and Abstraction

3. /kæt/ what? when? where? Electrophysiology How it works Mapping tool Discrimination Categorization Prediction Timing

4. How electrophysiology works

5. Electrical Activity

6. Probing Electrical Activity Interfere with it Record it indirectly Record it directly

7. Zap! Transcranial Magnetic Stimulation K. L. Sakai et al. 2002, Neuron

8. Verb generation Verb generation after 15 min practice Raichle & Posner, Images of Mind cover image Bang! Functional Magnetic Resonance Imaging (fMRI)

9. MRI studies brain anatomy. Functional MRI (fMRI) studies brain function. MRI vs. fMRI Source: Jody Culham, fMRI for Newbies

10. Electroencephalography (EEG)

14. Event-Related Potentials (ERPs) s1s2s3 John islaughing.

16. Phillips, Kazanina, & Abada (2005) Cog. Br. Res. The producers knew that the actress wished that … The producers knew which jokes the actress wished that..

17. ERP Studies Cost ($$) –Relatively cheap equipment and maintenance Time Investment –Materials: e.g., 20 syllables; 128 x 4 target sentences; 256 fillers –Acquisition: 2-3 hours x 12-24 participants –Analysis: 1-2 hours/person preprocessing; rest automatizable Strengths –Unbeatable temporal resolution –Easy to combine across participants –Movement possible; longer studies more tolerable –Interpretable results Disadvantages –Scalp topography ≠ localization –… and limited sex appeal

19. Brain Magnetic Fields (MEG) Brain magnetic fields recorded fully non-invasively by arrays of SQUID* detectors [*Superconducting QUantum Interference Device]

20. V skull CSF tissue MEG EEG B - noninvasive measurement - direct measurement. scalp recording surface current flow orientation of magnetic field Origin of the signal

21. How small is the signal? Earth field Urban noise Contamination at lung Heart QRS Muscle Fetal heart Spontaneous signal (  -wave) Signal from retina Intrinsic noise of SQUID Intensity of magnetic signal(T) Evoked signal Biomagnetism EYE (retina) Steady activity Evoked activity LUNGS Magnetic contaminants LIVER Iron stores FETUS Cardiogram LIMBS Steady ionic current BRAIN (neurons) Spontaneous activity Evoked by sensory stimulation SPINAL COLUMN (neurons) Evoked by sensory stimulation HEART Cardiogram (muscle) Timing signals (His Purkinje system) GI TRACK Stimulus response Magnetic contaminations MUSCLE Under tension

22. 160 SQUID whole-head array pickup coil & SQUID assembly

23. Sensor layout: recording from 160 channels Response peak at 98ms after onset of an auditory stimulus, in the left and right temporal lobes.

24. Magnetic source imaging (MSI): MEG + MRI Dipole fit at response peak, 98ms after onset of stimulus

25. (Halgren et al. 2002)

27. MEG Studies Cost ($$) –Expensive equipment; little maintenance; liquid helium supply Time Investment –Materials: e.g., 20 syllables; 600 x 3 target sentences; 400 fillers –Acquisition: 1-2 x 1-2.5 hours x 12 participants (+ structural MRI) –Analysis: similar to EEG, more individual-specific analyses Strengths –Unbeatable temporal resolution; easy set-up –Possibility of localization Disadvantages –Combining across individuals more difficult than in ERP studies –Longer studies, movement limited –Inverse problem for localization is very hard –Sensitivity depends on orientation and depth of source –Horrendously difficult with children, challenging w/ small heads

28. Electrophysiology in Syntax/Semantics 1. Classification Tool 2. Sensitive Timing Measure Sue takes her coffee with cream and socks. N400 - semantic anomaly The plane took we to paradise.P600 - syntactic anomaly The hungry guests helped himself to food.P600 Prerequisite: response components sensitive to distinct information types Mary praised Max’s of proof the theorem.‘ELAN’, 100-200ms No bills that the senators supported ever became law. control The bills that the senators supported ever became law. ~400ms response The bills that no senators supported ever became law. ~400ms response Prerequisite: explicit process models of syntax

29. Electrophysiology as a Mapping Tool

30. Sensory Maps Internal representations of the outside world. Cellular neuroscience has discovered a great deal in this area. Sensory Maps Internal representations of the outside world. Cellular neuroscience has discovered a great deal in this area.

31. Notions of sensory maps may be applicable to some aspects of human phonetic representations… …but there has been little success in that regard, and we shouldn’t expect this to yield much. Vowel Space

32. Obleser Lahiri J. Cogn. Neurosci., 16, 31-39 (2004)

33. M100 Elicited by any well-defined onset Varies with tone frequency Varies with F1 of vowels May vary non-linearly with VOT variation Functional value of time-code unclear No evidence of higher-level representations (Poeppel & Roberts 1996) (Poeppel, Phillips et al. 1997) (Phillips et al. 1995; Sharma & Dorman 1999)

34. Electrophysiological measures of discrimination

35. Mismatch Response Latency: 150-250 msec. Many-to-one ratio between standards and deviants X X X X X Y X X X X Y X X X X X X Y X X X Y X X X...

36. Localization of Mismatch Response (Phillips, Pellathy, Marantz et al., 2000) [Radiological view - left is right]

37. Basic MMN elicitation © Risto Näätänen

38. Mismatch Negativity (MMN) Sams et al. 1985

39. Tiitinen et al. 1994 How does MMN latency, amplitude vary with frequency difference? 1000Hz tone std.

40. Different Dimensions of Sounds Length Amplitude Pitch …you name it … Amplitude of mismatch response can be used as a measure of perceptual distance

41. Impetus for Language Studies If MMN amplitude is a measure of perceptual distance, then perhaps it can be informative in domains where acoustic and perceptual distance diverge…

42. Place of Articulation Acoustic variation: F2 & F3 transitions

43. Place of Articulation Acoustic variation: F2 & F3 transitions [bæ][dæ]

44. Place of Articulation Acoustic variation: F2 & F3 transitions [bæ][dæ] within category between category

45. Place of Articulation Acoustic variation: F2 & F3 transitions [bæ][dæ] within category between category

46. Categories in Infancy High Amplitude Sucking - 2 month olds Eimas et al. 1971 20 vs. 40 ms. VOT - yes 40 vs. 60 ms. VOT - no Infants show contrast, but this doesn’t entail phonological knowledge

47. Place of Articulation No effect of category boundary on MMN amplitude (Sharma et al. 1993) Similar findings in Sams et al. (1991), Maiste et al. (1995)

48. but…

49. Näätänen et al. (1997) e e/ö ö õ o

50. Relevance to Learning Models Place of articulation continuum (Lalonde & Werker, 1988) b -- d -- D 3 contrasts Nativeb -- d Non-natived -- D Non-phoneticb 1 -- b 5 Conflicting results –Dehaene-Lambertz 1997 native contrast only –Rivera-Gaxiola et al. 2000 native + non-native contrasts Dehaene-Lambertz Rivera-Gaxiola Phillips (2001, Cognitive Science)

51. J. Cogn. Neurosci. 16:577-583 (2004) Contrast and Underspecification

52. Phonetic Category Effects Measures of uneven discrimination profiles Findings are mixed (…and techniques vary) Relies on assumption that effects of contrasts at multiple levels are additive, …plus the requirement that the additivity effect be strong enough to yield a statistical interaction Logic of next set of studies: –Eliminate contribution of lower levels by isolating the many-to-one ratio at a more abstract level of representations –Do this by introducing non-orthogonal variation among standards

53. Electrophysiological measures of abstraction

54. J. Cogn. Neuro., 12, 1038-1055 (2000) /dæ//tæ/

55. Design 20ms40ms60ms Fixed Design - Discrimination

56. Design 0ms 8ms16ms24ms40ms48ms56ms64ms 20ms40ms60ms Fixed Design - Discrimination Grouped Design - Categorization Non-orthogonal within-category variation: precludes grouping via acoustic streaming.

57. Design 0ms 8ms16ms24ms40ms48ms56ms64ms 20ms40ms60ms Fixed Design - Discrimination Grouped Design - Categorization 20ms 28ms36ms44ms60ms68ms76ms84ms Grouped Design - Acoustic Control

59. /dæ/ standard vs. /dæ/ deviant

60. So what … Auditory cortex generator of MMF accesses representations that treat members of the same category as identical No indication of what might be the form of these representations, or where they might be stored MMF generator accesses multiple levels of representation

61. Sound Groupings (Phillips, Pellathy, & Marantz, 2000) pæ, tæ, tæ, kæ, dæ, pæ, kæ, tæ, pæ, kæ, bæ, tæ...

62. (Phillips, Pellathy, & Marantz, 2000)

63. More on features … Alternative account of the findings –No feature-based grouping –Independent MMF elicited by 3 low-frequency phonemes /bæ/ /dæ/ /gæ//pæ/ /tæ/ /kæ/ 29% 4% 87.5% 12.5% (Yeung & Phillips, 2004)

64. More on features … Next study distinguishes –Phoneme-level frequency –Feature-level status /gæ//bæ//dæ//tæ/ 37.5% 12.5% (Yeung & Phillips, 2004)

65. More on features … Next study distinguishes –Phoneme-level frequency –Feature-level status /gæ//bæ//dæ//tæ/ 37.5% 12.5% Phoneme-based classification (Yeung & Phillips, 2004)

66. More on features … Next study distinguishes –Phoneme-level frequency –Feature-level status /gæ//bæ//dæ//tæ/ 37.5% 12.5% Feature-based grouping (Yeung & Phillips, 2004)

67. More on features … Design –N = 10 –Multiple exemplars, individually selected boundaries –2 versions recorded for all participants, reversing [±voice] value –Acoustic control, with all VOT values in [-voice] range /gæ//bæ//dæ//tæ/ 37.5% 12.5% Feature-based grouping (Yeung & Phillips, 2004)

68. More on features … Left-anterior channels (Yeung & Phillips, 2004)

69. Kazanina et al., 2006 Proceedings of the National Academy of Sciences, 103, 11381-6 Nina Kazanina, U. of Bristol

70. Russian vs. Korean Three series of stops in Korean: –plain (lenis)pa ta ka –glottalized (tense, long)p’at’ak’a –aspiratedp h t h ak h a Intervocalic Plain Stop Voicing: /papo/[pabo]‘fool’ /ku papo/[k  babo]‘the fool’ Plain stops: –Bimodal distribution of +VOT and –VOT tokens –Word-initially: always a positive VOT –Word-medially intervocalically: a voicing lead (negative VOT)

71. Identification/ Rating Discrimination

72. Black: p <.05 White: n.s.

73. Russian vs. Korean MEG responses indicate that Russian speakers immediately map sounds from [d-t] continuum onto categories Korean speakers do not… … despite the fact that the sounds show bimodal distribution in their language Perceptual space reflects the functional status of sounds in encoding word meanings

74. Dupoux et al. (1999, Percep. Psychophys.) J. Cogn. Neurosci. 12:635-647 (2000) EXECTIVE SUITE French: CVC egma  egma Japanese: *CVC egma  eguma

75. Early (~200ms) Late (~600ms) Dehaene-Lambertz et al. (2000, J. Cogn. Neurosci.) Egma, egma, egma, egma, eguma, …

76. Mismatch Studies Perceptual experience of sound categories is mapped onto multiple levels of representation, at different degrees of abstraction Abstract category representations can be probed using non-orthogonal variation among sounds in an auditory mismatch paradigm  more-or-less immediate access during speech perception Beware of hasty inferences to normal time course of processing –Recurring standard sound establishes ‘model’ for parsing upcoming sounds –Detecting deviance from model, even at abstract level, may be faster than normal analysis processes –cf. related findings about ELAN (‘Early Left Anterior Negativity’) in syntactic ERP literature (Lau et al., 2006)

77. Electrophysiology and Prediction

79. McGurk Effect: auditory [pa] + visual [ka] = perceptual [ta]

82. Neurosci. Lett. 397:263-268 (2006) [aba] [ãba]

83. Electrophysiology and Timing

84. syntax > phonology phonology > syntax Science, 280: 572-574 Lateralized Readiness Potential (LRP)

86. Prospects Many possibilities for more sophisticated phonetic/phonological studies –Mapping: what are the questions? –Early transforms of acoustic space: already an active area –Discrimination: needs a theory of variable access to levels of detail –Categorization: can be used to investigate learning and abstraction –Prediction: phonotactics is barely touched –Timing: possible LRP studies of computing derived properties