Chapter 16 Audiovisual Speech Perception Perry C. Hanavan, Au.D.

Questions Lipreading and speechreading difined the same in the text book? True False

Seeing at the Speed of Sound Watch: What It’s Like to Read Lips

Definitions Lipreading Speechreading the person relies ONLY on the visual speech signal and other visual cues provided by the talker Speechreading the person attends to both the visual AND auditory cues provided by the talker

General Tendencies Normal hearing persons rely on speechreading Persons listening in background noise tend to use visual cues for speech recognition Persons with hearing loss tend to rely more on the visual signal for speech recognition The greater the hearing loss, the more the tendency for reliance on visual cues

Speechreading Lip cues Facial expression cues Gesture cues Body language cues Linguistic Situational cues Auditory cues*

Speechreading for Communication Normal hearing adults Infants

Afternoon Clinic Appointments You arrive at work following lunch and find two patients waiting for clinical services. Dr. White is a 50 year old physician Mr. Black is a 20 year old assembly line worker in a noisy factory Who do you predict will have the better speechreading skills?

Characteristics of Good Speechreaders Visual word decoding Working memory (ability to store and manipulate items in memory simultaneously) Lexical identification speed (determining whether letters are a word) Phonological processing speed (whether two words rhyme) Verbal inference (how well one can complete a sentence with missing words) Age (Honnel et al 1991) (younger better)

Characteristics of Good Speechreaders Ability to capitalize on contextual cues Willingness to guess Mental agility Willingness to revise interpretations of partially recognized messages Linguistic skills World knowledge Modal differences (Erber, 1974) Neurophysiological (Summerfield, 1992) Cognitive (Ronnberg et al, 1999)

Factors that Influence Speechreading Difficulty

Predictive Powers Visual Word Decoding Working Memory May Have Predictive Power May Not Have Predictive Power May or May Not have Predictive Power Visual Word Decoding Working Memory Lexical Identification Speed Sentence Completion with Missing Words Visual Working Memory and Processing Speed Age Gender IQ Academic Achievement Amount of Practice Age and Duration of Onset of HL Socioeconomic Status Neurophysiologic Measures Use of Contextual Cues Willingness to Guess Mental Agility Revision Willingness of Contextual Cues Linguistic and World Knowledge

Lipreading: Follow the Eyes Lip shapes vary based on phoneme(s) produced Eyebrows rise with questions Tend to gaze at eyes, nose and mouth Occasional looks at forehead, cheeks and chin Prosodic information—lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

Gaze Saccades/Patterns

Gaze-direction-based MEG averaging during audiovisual speech perception

Lipreading: Follow the Eyes Prosodic judgments: lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

Lipreading: Follow the Eyes Phonetic judgments: monitored lower face such as lip and jaw movement

Factors Influencing Speechreading

Difficulty Lipreading One third speech sounds visible mid and back consonants invisible Vowels not highly visible Rapidity of speech – 150 to 250 word/min Coarticulation Stress can change appearance of word Talker variability Visemes and homophenes

Question Define viseme: Lip gloss Seeing eye dog Groups of speech sounds that APPEAR identical on the lips Words that LOOK identical on the mouth E. C & D

Visemes and Homophenes Groups of speech sounds that APPEAR identical on the lips p b m – f v – s z – n t d s z – k g Homophenes 40-60 % of words Words that LOOK identical on the mouth pan ban man

Consonants Grouped as Visemes

Homophenous Word Pairs

Vowels and Lipreading Vowels not considered highly visible Vowels tend to be audible Intense, long duration Front vowels Lips flat or spread Back vowels Lips rounded

Production of Vowels Cardinal Vowel Chart (YouTube)

Consonants Place of production Manner of production Voicing Individuals with high frequency hearing loss may have difficulty hearing place cues F2 transition (1000-2400 Hz range) Place cues tend to be visible Bilabial, labiodental, linguadental sounds visible Manner of production Cues not visible, must be heard F1 transition (250-1000 Hz range) Voicing Cues not visible Low frequency range

Vowel Formants

Consonants and AV Production Auditory Cues Visual Cues AR Place of Production Where T2 Formant Above 1000 Hz Front sounds visible Speechreading Manner of Production How T1 Formant Below 2000 Hz Not visible Auditory Training

Difficult Speaking Behaviors Mumble Looks away when speaking Chews gum Unusual accent Speech impediment Smiles too much Moves around while talking No facial expression Shouts High pitch voice Talks to rapidly Uses long complicated sentences Wears a beard/mustache Wears dark glasses

Talkers Easier to lipread someone familiar Family members, teachers, etc. Females easier to lipread than males However, auditory plus vision may be more difficult as females are less audible to person with hearing loss

Question Clear speech…which is not a principle of? Talk slightly faster Talk slightly louder Talk slightly slower Talk with some pauses None of the above

Talker Clear Speech Principles Cessations or pauses after phrases and sentences Louder slightly Enunciate all phonemes precisely and accurately (use clear speech versus conversational speech) Accentuation (full range of voice intonation and stress on key word) Rate of speech slowed

Conversation Vs. Clear Speech Shum, http://www.audiologyonline.com/articles/making-speech-more-distinct-12469

Conversation Vs. Clear Speech Shum, http://www.audiologyonline.com/articles/making-speech-more-distinct-12469

Conversation Vs. Clear Speech Shum, http://www.audiologyonline.com/articles/making-speech-more-distinct-12469

Message Structure – complexity of message, frequency of use, linguistic context Frequency of usage – how often a word occurs in everyday conversations Neighborhoods – fewer lexical neighbors can be beneficial Context – words specified by context are easier

Lexical Neighborhoods

Environment Viewing angle – face to face Distance – favorable seating Room conditions – lighting, lighting angle, shining light, interfering objects, room noise

Speechreader Audibility Use of appropriate amplification system, ALDS, cochlear implant Use of eyeglasses if necessary Emotional and physical state

Tadoma Method: Speechreading Deafblind Users of Tadoma Speechreading Method

Heather Whitestone: 1994 I find lip reading very stressful and frustrating because I am often confused. For example, if you look at person’s lips saying dog and saw, they look the same. With my hearing aid alone, I do not hear "s" or "d" sounds. So usually I have to use my common sense. For example, if someone said, "The dog is running across the street." Then I knew it was not the saw who ran across the street – it was the dog. Most hearing people do not understand that people in my position have to think incredibly fast in order to keep up with conversations. One-on-one conversations are not that stressful, but group conversations when coupled with background noise are nearly impossible. Lip reading is a grueling and exhausting mental exercise and lip readers are constantly thinking and trying to discern what is actually being said. I get real mad at those who think that I am stupid simply because I cannot hear. The truth is I get exhausted after a while and simply cannot keep up. At that point, I begin to guess at what is being said and eventually give up and choose to be quiet.

Baldi Meet Baldi (iPhone app) Baldi (2 iPhone app)

Question An oral interpreter? Repeats message in view of person with hearing loss Reads famous speeches Speaks for person who uses ASL All the above None of the above

Oral Interpreters A trained professional who sits in clear view of a person with a hearing loss and silently repeats a talker’s message as it is spoken

Bisensory Perceptions What we see may influence what we hear What we hear may influence what we see McGurk Effect

Lip Movements “…it is important to realize that we move our articulators to produce acoustically distinct sounds and NOT visually contrastive movements.” Mark Ross

Speechreading Factoid Campbell et al, (1981) surveyed the literature and found 38-58% of individuals with HL have accompanying visual deficiencies Johnson et al (1981) found 65% entering NTID demonstrated defective vision Vision loss may be greater among individuals with HL Individuals with HL need vision evaluation***

Auditory plus Vision When BOTH auditory and visual information is available, individuals with hearing loss tend to do better on communication tasks Example (Auditory plus Vision): Speech Recognition Score = 50% Speechreading Score = 20% Combined Visual/Auditory Score = 90%

WIPI Word Intelligibility by Picture Identification (WIPI) closed-set picture-pointing (six pictures per plate) appropriate for children whose language age is between 5 and 10-11 comprised of four 25-monosyllabic word lists contains 26 color plates (one for practice), six pictures per page. (A, V, A-V)

Larsen Recorded Test Auditory word discrimination test Pairs of phonemes in words few vs. chew bill vs. mill nice vs. vice Lists represent an attempt to present the phone in the initial, medial and final position of a word Limited choice—select one of two words by drawing line through printed word heard

Test of Child Speechreading (ToCS) Child friendly Computer based Sentence and word recognition http://www.isca-speech.org/archive_open/avsp09/papers/av09_028.pdf

Auditory Plus Vision Tye-Murray Eber CAVET (children) Sent-Ident Audition plus vision Vision only Audition alone Eber Sent-Ident

CAVET Children’s Audiovisual Enhancement Test Assesses speechreading enhancement in children within the vocabulary level of 7-9 year olds with profound prelingual hearing loss Designed to minimize ceiling and floor effects, eliminate syntactic factors, and minimize semantic factors 3 lists of 20 words each with half of words easy to recognize in a vision-only condition and half are less likely to be recognized in each list but presented in random order Each list is designated for auditory alone, visual alone, or auditory-visual only mode Test available in CD-ROM and VHS format (Tye-Murray & Geers, 2002)

CAVET

SENT-IDENT

Ling-6 Phoneme level detection and recognition test m, s, sh, e, a, u Procedure Detection and identification of phonemes

Craig Lipreading Test

Word Recogntion

Utley Lipreading Test

Denver Quick Test

Speechreading Activity

AV and Speechreading Detection Discrimination Recognition   Detection Discrimination Recognition Comprehension Non speech sounds Phonemes Ling 6 Sounds Syllables Words  Larsen Recorded Test CAVET WIPI ToCS Craig Lipreading Inventory Phrases Sentences SENT-IDENT Denver Quick Test Utley, WIPI, ToCS Connected speech TOPICON

Speechreading Factoid Three of the lipreading methods introduced into the U.S. were implemented by individuals with normal hearing until adulthood, at which time they acquired a significant hearing loss, and sought assistance. They developed methods that bear their names: Bruhn, Kinzie, and Nitchie. Bunger later wrote about the Jena method.

Analytic & Synthetic Analytic approach (bottom-up) perceive each of the basic parts before the whole can be identified Syllable considered to be the basic unit Bruhn & Jena methods Synthetic approach (top-down) Perception of the whole is paramount to perception of the basic parts Sentence considered to be the basic unit Nitchie and Kinzie methods

Traditional Lipreading Methods Four lipreading methods were introduced into the U.S.: Mueller-Walle introduced by Bruhn Nitchie introduced by Nitchie Kinzie introduced by Kinzie Jena introduced by Bunger

Mueller-Walle Method Originated in Germany Martha Emma Bruhn studied lipreading with Julius Mueller-Walle in Hamburg, Germany and introduced method in US Hallmark: rapid syllable drill emphasis on quickly identifying position and movement of speech sounds produced rapid syllable rhythmic drills: she-ma-flea and she-may-free practiced recognizing homophenous words using sentence cues to distinguish meaning lessons based on sound movement or group movements

Nitchie Method Edward Nitchie became deaf during adolescence Eventually established his own school for the deaf in NY Initially developed an analytic approach, altering the approach to a synthetic approach (credited with developing synthetic approach to speechreading) Speechreader studies articulatory movements by viewing meaningful monosyllabic words to develop eye training Advocated use of sentences and stories to train mind to comprehend the general meaning connected discourse Mirror training

Kinzie Method Cora acquired HL while medical student in PA Studied with Bruhn and then Nitchie combining best methods from both approaches Unique feature: graded lessons for children and adults with sentences as basis of instruction They recommended all sentences be “definite, natural, interesting, pleasing, rhythmical, and dignified”

Questions What is meant by imitating movements of speaker Kinesthetic Mimetic Auditory Visual All the above

Jena Method Developed by Karl Brauckman in Jena, Germany and promoted by Anna Bunger from Michigan Emphasizes syllable drills, rhythm practice, and kinesthetic awareness Material presented in rhythmic manner reinforcing fact that speech is rhythmic Focused on mimetic (imitating movements of speaker) and kinesthetic (perception of movement, position, etc.) forms and sensations

Question More recent methods of teaching speechreading tend to be: Analytic Synthetic Eclectic None of the above

Recent Speechreading Trends Decline of speechreading as sole element of AR with advent of technology Technology (HA, CI, ALDs) is providing opportunities for individuals to make increasing use of audible speech and other audible sounds Newer lipreading approaches tend to be eclectic Modification or combinations of earlier synthetic and analytic approaches

Holistic Approach Increase the child’s knowledge of the speechreading process Increase the child’s ability to generate strategies to facilitate more successful communication Increase the child’s confidence in the efficacy of high probability success Increase the child’s tolerance for communicative situations that have a higher degree of frustration Increase the child’s ability to generate personal goals for improving speechreading Increase the child’s motivation to improve speechreading abilities

Efficacy of Speechreading Training Studies are mixed regarding demonstrating improvement following treatment for adults Little research regarding children Children may have greater potential for benefit from speechreading training than adults

Developing Speechreading Skills First step is usually instructional and includes consideration for the process Second step may require speechreaders to reflect on their on habits and skills Third step may require speechreaders to identify difficult listening situations and formulate solutions Fourth step is introduction of formal speechreading lessons

Analytic Speechreading Training Focus on vowel and consonant recognition Underlying logic this curricula is to gradually increase reliance on auditory signal for discriminating phonemic contrasts while they are speechreading Reliance on Vision Reliance on Audition

Analytic Vowel Training Initial training of highly contrastive features i u a Differ in BOTH formant structure and on the mouth Back vs front vowels I i e ae front vowels u U o back vowels

Acoustic vs Tongue Position

Analytic Training Objectives Examples: Will discriminate words with i and u Front vs. back vowels Will discriminate words with i and a High vs. low vowels Will discriminate words with u and a Will identify words with i u and a

Books, CD, DVDs, etc. Speechreading Seeing and Hearing Speech

Discriminate Vowels with i and u beet/boot see/soup she/shoe leap/loop peel/pool heat/hoot jeep/jewel sheet/shoot keep/coop knee/new geese/goose need/nude read/rude

Identify Vowels with u beet/boot/bat soup/seat/sap lass/lease/loose hat/heat/hoot team/tam/tomb gas/geese/goose pool/pal/peel jeep/jab/jude teen/tool/tan

Analytical Consonant Training Features – manner, place, voicing Place – visible AND less audible Manner and voicing – NOT visible, but audible

Consonant Place of Production k g ng velar palatal Sh zh ch jd alveolar bilabial labiodental dental p b m f v th t d n s z l glottal h

Analytic Consonant Objectives will discriminate consonant pairs that differ in place of production and share either voice or manner will discriminate consonant pairs that share similar place of production but differ in manner and voice will discriminate consonant pairs that share place and manner and/or voice will identify consonants that share manner of production, using a four-item response set will identify consonants from a six-item response set of voiced or voiceless consonants

Discriminate Consonant Pairs that Differ in Place and Share Either Voice or Manner meet/geese pill/chill top/chop moose/goose pot/hot boat/coat bit/knit dog/jog peal/heal make/lake tear/chair pin/chin

Synthetic Speechreading Training Objectives will follow simple directions using a closed set response will identify a sentence illustration from a set of four dissimilar pictures will identify a sentence illustration from a set of four similar pictures will listen plus lipread to two related sentences, and then draw a picture about them or paraphrase them will speechread a paragraph-long narrative and then answer questions about it

Example of Synthetic Approach Sentences concerning cooking: I added a cup of flour. The bread is in the oven. Will you hand me the measuring cup. I need the box of sugar. The mixer is in the cabinet. The oven is set to 300 degrees. Put the bowl in the sink, please The pan is filled with batter. I will beat the eggs. Please pour a cup of milk.

Resources Online resources Read My Lips Lipreading Laboratory Gallaudet Read My Lips Movie Lipreading Laboratory