The Effect of Incongruent Visual Cues on the Heard Quality of Front Vowels Hartmut Traunmüller Niklas Öhrström Dept. of Linguistics, University of Stockholm

Background We have earlier carried out an AV perception experiment in which congruent and incongruent AV stimuli were presented to subjects. The AV stimuli consisted of different front vowels presented within a [g_g] frame. They were incongruent with respect to openness (height) or roundedness or both. The subjects had to report which vowel they had heard. The response alternatives consisted of the nine letters that represent the long vowel phonemes of Swedish.

Background We have earlier carried out an AV perception experiment in which congruent and incongruent AV stimuli were presented to subjects. The AV stimuli consisted of different front vowels presented within a [g_g] frame. They were incongruent with respect to openness (height) or roundedness or both. The subjects had to report which vowel they had heard. The response alternatives consisted of the nine letters that represent the long vowel phonemes of Swedish.

Background We have earlier carried out an AV perception experiment in which congruent and incongruent AV stimuli were presented to subjects. The AV stimuli consisted of different front vowels presented within a [g_g] frame. The vowels were incongruent with respect to openness (height) or roundedness or both. The subjects had to report which vowel they had heard. The response alternatives consisted of the nine letters that represent the long vowel phonemes of Swedish.

Background We have earlier carried out an AV perception experiment in which congruent and incongruent AV stimuli were presented to subjects. The AV stimuli consisted of different front vowels presented within a [g_g] frame. The vowels were incongruent with respect to openness (height) or roundedness or both. The subjects had to report which vowel they had heard. The response alternatives consisted of the nine letters that represent the long vowel phonemes of Swedish.

Background Typical results: Visual roundedness combined with auditory openness. AVPercept ɡyɡɡeɡɡeɡ → ɡiɡ ɡeɡɡyɡ → ɡøɡɡøɡ ɡiɡɡyɡ → ɡyɡɡyɡ ɡeɡɡeɡɡiɡ → ɡeɡ

Background Explanation: Acoustic cues to openness (F 1 etc.) are prominent and reliable. Acoustic cues to roundedness (higher formants) are less reliable. Optic cues to roundedness are prominent and reliable; rounded lips are easy to distinguish from unrounded. Optic cues to openness are less reliable because of variation due to individual habits, attitude and emotion.

Background Explanation: Acoustic cues to openness (F 1 etc.) are prominent and reliable. Acoustic cues to roundedness (higher formants) are less reliable. Optic cues to roundedness are prominent and reliable; rounded lips are easy to distinguish from unrounded. Optic cues to openness are less reliable because of variation due to individual habits, attitude and emotion.

Background The mentioned experiment was designed with the objective of investigating categorical phonemic perception. However, subjects informally reported having heard vowels whose quality differed from that of ordinary Swedish vowels. Auditory rounding together with visual unrounding appeared to affect the heard backness quality of the vowel.

Background The mentioned experiment was designed with the objective of investigating categorical phonemic perception. However, subjects informally reported having heard vowels whose quality differed from that of ordinary Swedish vowels. Auditory rounding together with visual unrounding appeared to affect the heard backness quality of the vowel.

The present study The present experiment aims at exploring the effect of the optic signal on the finer phonetic, sub- categorical auditory perception of vowels.

The present study We reused a subset of the stimuli from the previous experiment. AV ɡyɡɡiɡ ɡyɡɡeɡ ɡyɡ-- ɡyɡ AV ɡeɡɡiɡ ɡeɡɡyɡ ɡeɡ-- ɡeɡ AV ɡiɡɡyɡ ɡiɡɡeɡ ɡiɡ-- ɡiɡ

The present study There were 4 speakers: 2 male, 2 female.

The present study There were 8 perceivers: They were selected from a previous experiment where they had shown sensitivity to the optic signal in incongruent audiovisual stimuli. The 8 subjects were all phonetically skilled and familiar with the IPA-chart for vowels.

The present study The subjects perceived the stimuli by way of headphones and a computer screen. The stimuli were presented in quasi-random order. Responses were given on electronic response sheets.

The present study The subjects were instructed to rate the dimensions of the heard vowel (or of those seen in purely optical stimuli). Lip rounding (6 degrees), 1st: unrounded; 5th: rounded Lip spreading (3 degrees) Openness (18 degrees), 2nd: close vowels, 6th: close-mid vowels Backness (11 degrees auditorily; 7 degrees visually), 2nd: front vowels, 6th (auditorily): central vowels

Results Openness opn vs. roundedness rnd; acoustic stimuli (listening only):

Results Openness opn vs. roundedness rnd; optic stimuli (lipreading only):

Results Openness opn of incongruent AV-stimuli vs. opn of A-stimuli: opn = opn A (r 2 = 0.97)

Results Roundedness rnd of incongruent AV-stimuli vs. rnd of A-stimuli: (no significant correlation)

Results Backness bac of incongruent AV-stimuli vs. rnd of A-stimuli: bac = rnd A (r 2 = 0.66) bac = rnd A – 0.20 rnd AV (r 2 = 0.74)

Results Openness opn of incongruent AV-stimuli vs. opn of V-stimuli: (no significant correlation)

Results Roundedness rnd of incongruent AV-stimuli vs. rnd of V-stimuli: rnd = rnd V (r 2 = 0.92) rnd = rnd V bac V (r 2 = 0.95)

Results Backness bac of incongruent AV-stimuli vs. rnd of V-stimuli: (significant negative correlation)

Discussion Rated backness in incongruent stimuli is correlated with roundedness in the stimuli. There are two hypothetical explanations for this: 1.The distance from the lips to the dorso-palatal ’place of articulation’ is increased by lip rounding as well as by tongue retraction. This would provide an articulatory (gestural) explanation. 2.F 2 ’ is lowered by lip rounding as well as by tongue retraction. This would provide an auditory explanation. Both explanations would be consistent with the placement of the rounded vowels to the right of their unrounded counterparts in IPA-charts.

Discussion Rated backness in incongruent stimuli is correlated with roundedness in the stimuli. There are two hypothetical explanations for this: 1.The distance from the lips to the dorso-palatal ’place of articulation’ is increased by lip rounding as well as by tongue retraction. This would provide an articulatory (gestural) explanation. 2.F 2 ’ is lowered by lip rounding as well as by tongue retraction. This would provide an auditory explanation. Both explanations would be consistent with the placement of the rounded vowels to the right of their unrounded counterparts in IPA-charts.

Discussion Rated backness in incongruent stimuli is correlated with roundedness in the stimuli. There are two hypothetical explanations for this: 1.The distance from the lips to the dorso-palatal ’place of articulation’ is increased by lip rounding as well as by tongue retraction. This would provide an articulatory (gestural) explanation. 2.F 2 ’ is lowered by lip rounding as well as by tongue retraction. This would provide an auditory explanation. Both explanations would be consistent with the placement of the rounded vowels to the right of their unrounded counterparts in IPA-charts.

Discussion Rated backness in incongruent stimuli is correlated with roundedness in the stimuli. There are two hypothetical explanations for this: 1.The distance from the lips to the dorso-palatal ’place of articulation’ is increased by lip rounding as well as by tongue retraction. This would provide an articulatory (gestural) explanation. 2.F 2 ’ is lowered by lip rounding as well as by tongue retraction. This would provide an auditory explanation. Both explanations would be consistent with the placement of the rounded vowels to the right of their unrounded counterparts in IPA-charts.

Discussion Analysis of perceived backness StimulusPredictionObservation A (acoustic) V (optic) Expl. 1 (gestural) Expl. 2 (auditory) roundedunroundedfrontedretracted unroundedroundedretractedfronted

Discussion Analysis of perceived backness StimulusPredictionObservation A (acoustic) V (optic) Expl. 1 (gestural) Expl. 2 (auditory) roundedunroundedfrontedretracted unroundedroundedretractedfronted

Discussion Analysis of perceived backness StimulusPredictionObservation A (acoustic) V (optic) Expl. 1 (gestural) Expl. 2 (auditory) roundedunroundedfrontedretracted unroundedroundedretractedfronted Conclusion: The effect is due to auditory (F 2 ’) rather than articulatory (gestural) associations.

Discussion We have seen that the perceived retractedness of A rounded but V unrounded vowels can be understood as due to a continuous auditory variable (F 2 ’). The variation in perceived retractedness cannot be explained on the basis of a late-integration hypothesis, since Swedish lacks non-front unrounded vowel phonemes, whose existence would be required in order to apply such a hypothesis. This is clear and direct evidence for early, sub- categorical integration.

x Thank you for your attention!