1. From Sound to Sense and back again: The integration of lexical and speech processes From Sound to Sense and back again: The integration of lexical and speech processes David Gow Massachusetts General Hospital David Gow Massachusetts General Hospital Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester

2. Complex computations from sound to sense must be broken up for study. The Speech Chain Sound Sense Assume intermediate representations: Phonemes… Words… Syntactic Phrases…

3. The Standard Paradigm Sense Phonology Words Phonemes Sound

4. The Standard Paradigm Phonology Words Phonemes Delimited fields of study. Sound Speech Perception Spoken Word Recognition Phonology Phonemes * essential * or other sublexical category Sense

5. Why? Categorical Perception (CP) Sharp identification of tokens on a continuum. VOT 0 100 PB % /p/ ID (%/pa/) 0 100 Discrimination Discrimination poor within a phonetic category. Continuous Acoustic Detail => Discrete Categories Does CAD affect speech categorization?

6. Categorical Perception (CP) Defined fundamental computational problems. CP is output of Speech perception Input to Phonology Word recognition. Phonology Words Phonemes Sense Sound

7. But… Not all speech contrasts are categorical. Lots of tasks show non-categorical perception. Fry, Abramson, Eimas & Liberman (1962) Pisoni & Tash (1974) Pisoni & Lazarus (1974) Carney, Widden & Viemeister (1977) Hary & Massaro (1982) Pisoni, Aslin, Perey & Hennessy (1982) Healy & Repp (1982) Massaro & Cohen (1983) Miller (1997) Samuel (1997)… CP

8. Categorical Perception is about phonetic classification. Why has the Standard Paradigm persisted? Sound Sense The minimal computational problem: compute meaning from sound. CP tasks don’t necessarily tap a stage of this problem. ? CP Words Lexical activation… seems a good bet.

9. Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition. Why has the Standard Paradigm persisted?

10. Example: Word Segmentation Vowel Length Stress/Meter Coarticulation Words Phonemes CAD Segmentation Cue extra-segmental process. Word Recognition Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition. Why has the Standard Paradigm persisted?

11.  No.Standard Paradigm is fine… Yes.Hmm… Does continuous acoustic detail affect interpretation via core word-recognition processes? Need to use stimuli with: Precise control over CAD Need to use tasks that: reflect only minimal computational problem: meaning. are sensitive to acoustic detail. Sublexical Filter (phonemes)

12. Visual World Paradigm Subjects hear spoken language and manipulate objects in a visual world. Visual world includes set of objects with interesting linguistic properties (names) Eye-movements to each object are monitored throughout the task. Tanenhaus, Spivey-Knowlton, Eberhart & Sedivy (1995) Allopenna, Magnuson & Tanenhaus (1998)

13. Meaning based, natural task: Subjects must interpret speech to perform task. Eye-movements fast and time- locked to speech. Fixation probability maps onto dynamics of lexical activation. Context is controlled: meaning  lexical activation.

14. ? Does continuous acoustic detail affect interpretation? Is lexical activation sensitive to continuous acoustic detail?

15. Combine tools of speech perception: 9-step VOT continuum. spoken word recognition: visual world paradigm McMurray, Tanenhaus & Aslin (2003)

16. A moment to view the items Methods

17. 500 ms later

18. Bear Repeat 1080 times…

19. Target = Bear Competitor = Pear Unrelated = Lamp, Ship Time 200 ms 1234512345 Trials

20. Time (ms) VOT=0 Response= 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 040080012001600 Fixation proportion

21. Systematic effect on competitor dynamics. Fixations to the competitor. Predictions Categorical Results Gradient Effect target competitor time Fixation proportion target competitor time Fixation proportion target What would lexical sensitivity to CAD look like?

22. Results 040080012001600 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0 ms 5 ms 10 ms 15 ms VOT 0400800120016002000 20 ms 25 ms 30 ms 35 ms 40 ms VOT Competitor Fixations Time since word onset (ms) Response=

23. Task? P BSh L Phoneme ID Not part of minimal computational problem. Same stimuli in metalinguistic task… …more categorical pattern of fixations Continuous acoustic detail is not helpful in metalinguistic tasks…

24. Summary Word recognition shows gradient sensitivity to continuous acoustic detail. Not extra-segmental: VOT CAD affects higher-level processes. Consistent with other studies: Andruski, Blumstein & Burton (1994) Marslen-Wilson & Warren (1994) Utman, Blumstein & Burton (2000) Dahan, Magnuson, Tanenhaus & Hogan (2001) McMurray, Clayards, Aslin & Tanenhaus (2004) McMurray, Aslin, Tanenhaus, Spivey & Subik (in prep)

25. The Standard Paradigm? Sense Phonology Words Phonemes Continuous Acoustic Detail CAD affects higher-level processes. From other work: Lexical activation influences sublexical representations. Samuel & Pitt (2003) Magnuson, McMurray, Tanehaus & Aslin (2003) Samuel (1997) Elman & McClelland (1988)

26. The Standard Paradigm? Sense Phonology Words Phonemes Continuous Acoustic Detail CAD affects higher-level processes. From other work: Lexical activation influences sublexical representations. Phonological regularity affects signal interpretation. Massaro & Cohen (1983) Halle, Segui, Frauenfelder & Meunier (1998) Pitt (1998) Dupoux,Kakehi, Hirose, Pallier & Mehler, (1999)

27. ? Sense Phonology Words Phonemes Continuous Acoustic Detail Perhaps interaction and integration make sense. Do they help solve sticky problems? YES

28. The Emerging Paradigm Integration of work in: spoken word recognition speech perception phonology New computations simplify old problems and solve new ones. Cognitive processes: Lexical activation & competition. Perceptual processes: sensitivity to CAD & perceptual grouping.

29. CAD is helpful in language comprehension. Word segmentation Coping with lawful variability due to assimilation Combination of approaches helps solve both problems.

30. Some lexical processes can’t work in the Standard Paradigm Lexical Segmentation

31. [  k t I v d I p A  t m I n t ] The SWR Solution

32. active [  k t I v d I p A  t m I n t ]

33. active department [  k t I v d I p A  t m I n t ]

34. active department act of dip art mint a part depart in are par Standard Paradigm: Template matching overgenerates [  k t I v d I p A  t m I n t ]

35. Overgeneration resolved through competition in TRACE (McClelland & Elman 1986) Problem: What if the speaker is trying to say “suck seeds”? ‘ k s I d - succeed suck seed activation Cycle Frauenfelder & Peeters (1990)

36. Cues shown to affect segmentation: Initial strong syllable Initial lengthening Increased aspiration Increased glottalization Lehiste, 1960; Garding,1967; Lehiste, 1972; Umeda, 1975; Nakatani & Dukes, 1977; Nakatani & Schaffer,1978; Cutler & Norris, 1988….. Implied processing model requires separate segmentation process Words Segmentation Phonemes CAD Recognition The Speech Solution

37. Problem: cues are subtle and varied, extra-segmental processes are inelegant ? Is there a better mechanism? Words Segmentation Phonemes CAD Recognition

38. The proposal had a strange syntax that nobody liked. ^ The proposal had a strange sin tax that nobody liked. ^ CAD affects interpretation. does not trigger segmentation. Gow & Gordon (1995) GRAMMAR primed Syntax TaxINCOME inhibited  GRAMMAR primed Syntax TaxINCOME primed

39. Observation: All segmentation cues happen to enhance word-initial features Strengthened cues facilitate activation, making intended words stronger competitors Incorporating CAD: Solves overgeneration problem. No extra-segmental segmentation process. Good Start Model Gow & Gordon (1995)

40. When continuous acoustic detail affects lexical activation, speech and SWR models can be integrated and simplified Summary

41. The emerging paradigm reframes computational problems Assimilation

42. English coronal place assimilation /coronal # labial/  [labial # labial] /coronal #velar/  [velar # velar] Standard Paradigm: Change is discrete phonemically neutralizing Redefining Computational Problems [ G  I m ]# berries nonword? right berries? ripe berries? [  a I p ]# berries

43. Standard Paradigm solution: Phonological inference (Gaskell & Marslen-Wilson, 1996; 1998; 2001) Knowledge driven inference : If [labial # labial] infer /coronal # labial/ greem beans  green (Gaskell & Marslen-Wilson, 1996; Gow, 2001) ripe berries  right (Gaskell & Marslen-Wilson, 2001; Gow, 2002) Moreover: Assimilation effects dissociated from linguistic knowledge (Gow & Im, in press)  ripe

44. Assimilatory modification is acoustically continuous This is not discrete feature change! Assimilation Produces CAD F2 Transitions in /æC/ Contexts 1550 1600 1650 1700 1750 1800 1850 Pitch Period Frequency (Hz) coronal assimilated labial F3 Transitions in /æC/ Contexts 2550 2600 2650 2700 2750 2800 Pitch Period Frequency (Hz) coronal assimilated labial

45. Sma Select the ca t p box Regressive Context Effects

46. Subject Hears: Assim_Non-Coronal (cat/p box) 0 0.1 0.2 0.3 0.4 0.5 0.6 040080012001600 Time (ms) Fixation Proportion Coronal (cat) Non-Coronal (cap)

47. Subject Hears: Assim Non-Coronal (cat/p drawing) 0 0.1 0.2 0.3 0.4 0.5 0.6 040080012001600 Time (ms) Fixation Proportion Coronal (cat) Non-Coronal (cap)

48. Progressive Context Effects Progressive effect in the same experiment

49. Assimilation is resolved through phonological context. Fully assimilated items show neither* (Gaskell & Marslen-Wilson, 2001; Gow, 2002;2003) Assimilation: Use of CAD Partially-assimilated items show regressive context effects (Gow, 2002; 2003) progressive context effects (Gow, 2001; 2003)

50. assimilation # context Infinite regress (eternal ambiguity)…. or something more interesting?

51. Continuous acoustic detail is subject to basic perceptual processes

52. Feature cue parsing (Gow, 2003) [ k  t p b l E d ] A Perceptual Account

53. Feature cue parsing (Gow, 2003) Features encoded by multiple cues that are integrated

54. Feature cue parsing (Gow, 2003)

55. Assimilation creates cues consistent with multiple places

56. Feature cue parsing (Gow, 2003) Extract feature cues

57. Feature cue parsing (Gow, 2003) Group feature cues by similarity and resolve ambiguity

58. Feature cue parsing (Gow, 2003) example: eight…. ca t p # box ca t p # drawing ca t p #  | | | | [cor] [cor] [COR] [cor] [lab] [LAB] [lab] [lab]

59. example: eight…. ca t p # B ox ca t p # D rawing ca t p #  | | [cor] [cor] [COR] [cor] [lab] [LAB] [lab] [lab] Feature cue parsing (Gow, 2003) Progressive and regressive effects fall out of grouping

60. SWR problem (eternal ambiguity) replaced by simpler perceptual problem CAD important in solution: processing obstacle facilitates perception. Integration of continuous perceptual features facilitates higher-level processes. Facilitation via core-word recognition mechanisms—no extra-segmental routines required. Summary

61. Standard paradigm Created artificial boundaries that misframed issues. Continous acoustic detail is variability to be conquered.. The Standard Paradigm The basis of the standard paradigm is undercut. Meaning-based processes are affected by CAD. CAD is an essential component of word recognition.

62. The emerging paradigm Emphasis on methodologies that tap the minimal computational problem: meaning. Stresses integration of speech and spoken word recognition, questions methods and theory. Continuous acoustic detail is useful signal, not noise. The Emerging Paradigm

63. From Sound to Sense and back again: The integration of lexical and speech processes From Sound to Sense and back again: The integration of lexical and speech processes David Gow Massachusetts General Hospital David Gow Massachusetts General Hospital Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester