From Sound to Sense and back again: The integration of lexical and speech processes From Sound to Sense and back again: The integration of lexical and speech processes David Gow Massachusetts General Hospital David Gow Massachusetts General Hospital Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester

Complex computations from sound to sense must be broken up for study. The Speech Chain Sound Sense Assume intermediate representations: Phonemes… Words… Syntactic Phrases…

The Standard Paradigm Sense Phonology Words Phonemes Sound

The Standard Paradigm Phonology Words Phonemes Delimited fields of study. Sound Speech Perception Spoken Word Recognition Phonology Phonemes * essential * or other sublexical category Sense

Why? Categorical Perception (CP) Sharp identification of tokens on a continuum. VOT PB % /p/ ID (%/pa/) Discrimination Discrimination poor within a phonetic category. Continuous Acoustic Detail => Discrete Categories Does CAD affect speech categorization?

Categorical Perception (CP) Defined fundamental computational problems. CP is output of Speech perception Input to Phonology Word recognition. Phonology Words Phonemes Sense Sound

But… Not all speech contrasts are categorical. Lots of tasks show non-categorical perception. Fry, Abramson, Eimas & Liberman (1962) Pisoni & Tash (1974) Pisoni & Lazarus (1974) Carney, Widden & Viemeister (1977) Hary & Massaro (1982) Pisoni, Aslin, Perey & Hennessy (1982) Healy & Repp (1982) Massaro & Cohen (1983) Miller (1997) Samuel (1997)… CP

Categorical Perception is about phonetic classification. Why has the Standard Paradigm persisted? Sound Sense The minimal computational problem: compute meaning from sound. CP tasks don’t necessarily tap a stage of this problem. ? CP Words Lexical activation… seems a good bet.

Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition. Why has the Standard Paradigm persisted?

Example: Word Segmentation Vowel Length Stress/Meter Coarticulation Words Phonemes CAD Segmentation Cue extra-segmental process. Word Recognition Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition. Why has the Standard Paradigm persisted?

 No.Standard Paradigm is fine… Yes.Hmm… Does continuous acoustic detail affect interpretation via core word-recognition processes? Need to use stimuli with: Precise control over CAD Need to use tasks that: reflect only minimal computational problem: meaning. are sensitive to acoustic detail. Sublexical Filter (phonemes)

Visual World Paradigm Subjects hear spoken language and manipulate objects in a visual world. Visual world includes set of objects with interesting linguistic properties (names) Eye-movements to each object are monitored throughout the task. Tanenhaus, Spivey-Knowlton, Eberhart & Sedivy (1995) Allopenna, Magnuson & Tanenhaus (1998)

Meaning based, natural task: Subjects must interpret speech to perform task. Eye-movements fast and time- locked to speech. Fixation probability maps onto dynamics of lexical activation. Context is controlled: meaning  lexical activation.

? Does continuous acoustic detail affect interpretation? Is lexical activation sensitive to continuous acoustic detail?

Combine tools of speech perception: 9-step VOT continuum. spoken word recognition: visual world paradigm McMurray, Tanenhaus & Aslin (2003)

A moment to view the items Methods

500 ms later

Bear Repeat 1080 times…

Target = Bear Competitor = Pear Unrelated = Lamp, Ship Time 200 ms Trials

Time (ms) VOT=0 Response= Fixation proportion

Systematic effect on competitor dynamics. Fixations to the competitor. Predictions Categorical Results Gradient Effect target competitor time Fixation proportion target competitor time Fixation proportion target What would lexical sensitivity to CAD look like?

Results ms 5 ms 10 ms 15 ms VOT ms 25 ms 30 ms 35 ms 40 ms VOT Competitor Fixations Time since word onset (ms) Response=

Task? P BSh L Phoneme ID Not part of minimal computational problem. Same stimuli in metalinguistic task… …more categorical pattern of fixations Continuous acoustic detail is not helpful in metalinguistic tasks…

Summary Word recognition shows gradient sensitivity to continuous acoustic detail. Not extra-segmental: VOT CAD affects higher-level processes. Consistent with other studies: Andruski, Blumstein & Burton (1994) Marslen-Wilson & Warren (1994) Utman, Blumstein & Burton (2000) Dahan, Magnuson, Tanenhaus & Hogan (2001) McMurray, Clayards, Aslin & Tanenhaus (2004) McMurray, Aslin, Tanenhaus, Spivey & Subik (in prep)

The Standard Paradigm? Sense Phonology Words Phonemes Continuous Acoustic Detail CAD affects higher-level processes. From other work: Lexical activation influences sublexical representations. Samuel & Pitt (2003) Magnuson, McMurray, Tanehaus & Aslin (2003) Samuel (1997) Elman & McClelland (1988)

The Standard Paradigm? Sense Phonology Words Phonemes Continuous Acoustic Detail CAD affects higher-level processes. From other work: Lexical activation influences sublexical representations. Phonological regularity affects signal interpretation. Massaro & Cohen (1983) Halle, Segui, Frauenfelder & Meunier (1998) Pitt (1998) Dupoux,Kakehi, Hirose, Pallier & Mehler, (1999)

? Sense Phonology Words Phonemes Continuous Acoustic Detail Perhaps interaction and integration make sense. Do they help solve sticky problems? YES

The Emerging Paradigm Integration of work in: spoken word recognition speech perception phonology New computations simplify old problems and solve new ones. Cognitive processes: Lexical activation & competition. Perceptual processes: sensitivity to CAD & perceptual grouping.

CAD is helpful in language comprehension. Word segmentation Coping with lawful variability due to assimilation Combination of approaches helps solve both problems.

Some lexical processes can’t work in the Standard Paradigm Lexical Segmentation

[  k t I v d I p A  t m I n t ] The SWR Solution

active [  k t I v d I p A  t m I n t ]

active department [  k t I v d I p A  t m I n t ]

active department act of dip art mint a part depart in are par Standard Paradigm: Template matching overgenerates [  k t I v d I p A  t m I n t ]

Overgeneration resolved through competition in TRACE (McClelland & Elman 1986) Problem: What if the speaker is trying to say “suck seeds”? ‘ k s I d - succeed suck seed activation Cycle Frauenfelder & Peeters (1990)

Cues shown to affect segmentation: Initial strong syllable Initial lengthening Increased aspiration Increased glottalization Lehiste, 1960; Garding,1967; Lehiste, 1972; Umeda, 1975; Nakatani & Dukes, 1977; Nakatani & Schaffer,1978; Cutler & Norris, 1988….. Implied processing model requires separate segmentation process Words Segmentation Phonemes CAD Recognition The Speech Solution

Problem: cues are subtle and varied, extra-segmental processes are inelegant ? Is there a better mechanism? Words Segmentation Phonemes CAD Recognition

The proposal had a strange syntax that nobody liked. ^ The proposal had a strange sin tax that nobody liked. ^ CAD affects interpretation. does not trigger segmentation. Gow & Gordon (1995) GRAMMAR primed Syntax TaxINCOME inhibited  GRAMMAR primed Syntax TaxINCOME primed

Observation: All segmentation cues happen to enhance word-initial features Strengthened cues facilitate activation, making intended words stronger competitors Incorporating CAD: Solves overgeneration problem. No extra-segmental segmentation process. Good Start Model Gow & Gordon (1995)

When continuous acoustic detail affects lexical activation, speech and SWR models can be integrated and simplified Summary

The emerging paradigm reframes computational problems Assimilation

English coronal place assimilation /coronal # labial/  [labial # labial] /coronal #velar/  [velar # velar] Standard Paradigm: Change is discrete phonemically neutralizing Redefining Computational Problems [ G  I m ]# berries nonword? right berries? ripe berries? [  a I p ]# berries

Standard Paradigm solution: Phonological inference (Gaskell & Marslen-Wilson, 1996; 1998; 2001) Knowledge driven inference : If [labial # labial] infer /coronal # labial/ greem beans  green (Gaskell & Marslen-Wilson, 1996; Gow, 2001) ripe berries  right (Gaskell & Marslen-Wilson, 2001; Gow, 2002) Moreover: Assimilation effects dissociated from linguistic knowledge (Gow & Im, in press)  ripe

Assimilatory modification is acoustically continuous This is not discrete feature change! Assimilation Produces CAD F2 Transitions in /æC/ Contexts Pitch Period Frequency (Hz) coronal assimilated labial F3 Transitions in /æC/ Contexts Pitch Period Frequency (Hz) coronal assimilated labial

Sma Select the ca t p box Regressive Context Effects

Subject Hears: Assim_Non-Coronal (cat/p box) Time (ms) Fixation Proportion Coronal (cat) Non-Coronal (cap)

Subject Hears: Assim Non-Coronal (cat/p drawing) Time (ms) Fixation Proportion Coronal (cat) Non-Coronal (cap)

Progressive Context Effects Progressive effect in the same experiment

Assimilation is resolved through phonological context. Fully assimilated items show neither* (Gaskell & Marslen-Wilson, 2001; Gow, 2002;2003) Assimilation: Use of CAD Partially-assimilated items show regressive context effects (Gow, 2002; 2003) progressive context effects (Gow, 2001; 2003)

assimilation # context Infinite regress (eternal ambiguity)…. or something more interesting?

Continuous acoustic detail is subject to basic perceptual processes

Feature cue parsing (Gow, 2003) [ k  t p b l E d ] A Perceptual Account

Feature cue parsing (Gow, 2003) Features encoded by multiple cues that are integrated

Feature cue parsing (Gow, 2003)

Assimilation creates cues consistent with multiple places

Feature cue parsing (Gow, 2003) Extract feature cues

Feature cue parsing (Gow, 2003) Group feature cues by similarity and resolve ambiguity

Feature cue parsing (Gow, 2003) example: eight…. ca t p # box ca t p # drawing ca t p #  | | | | [cor] [cor] [COR] [cor] [lab] [LAB] [lab] [lab]

example: eight…. ca t p # B ox ca t p # D rawing ca t p #  | | [cor] [cor] [COR] [cor] [lab] [LAB] [lab] [lab] Feature cue parsing (Gow, 2003) Progressive and regressive effects fall out of grouping

SWR problem (eternal ambiguity) replaced by simpler perceptual problem CAD important in solution: processing obstacle facilitates perception. Integration of continuous perceptual features facilitates higher-level processes. Facilitation via core-word recognition mechanisms—no extra-segmental routines required. Summary

Standard paradigm Created artificial boundaries that misframed issues. Continous acoustic detail is variability to be conquered.. The Standard Paradigm The basis of the standard paradigm is undercut. Meaning-based processes are affected by CAD. CAD is an essential component of word recognition.

The emerging paradigm Emphasis on methodologies that tap the minimal computational problem: meaning. Stresses integration of speech and spoken word recognition, questions methods and theory. Continuous acoustic detail is useful signal, not noise. The Emerging Paradigm

From Sound to Sense and back again: The integration of lexical and speech processes From Sound to Sense and back again: The integration of lexical and speech processes David Gow Massachusetts General Hospital David Gow Massachusetts General Hospital Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester