1. Chapter 6 Audiovisual Speech Perception
Perry C. Hanavan, Au.D.

2. Questions
Lipreading and speechreading difined the same in the text book? True False

3. Definitions Lipreading Speechreading
the person relies ONLY on the visual speech signal and other visual cues provided by the talker
Speechreading
the person attends to both the visual AND auditory cues provided by the talker

4. General Tendencies Normal hearing persons rely on speechreading
Persons listening in background noise tend to use visual cues for speech recognition
Persons with hearing loss tend to rely more on the visual signal for speech recognition
The greater the hearing loss, the more the tendency for reliance on visual cues

5. Speechreading Lip cues Facial expression cues Gesture cues
Body language cues
Linguistic
Situational cues
Auditory cues*

6. Speechreading for Communication
Normal hearing adults
Infants

7. Afternoon Clinic Appointments
You arrive at work following lunch and find two patients waiting for clinical services.
Dr. White is a 50 year old physician
Mr. Black is a 20 year old assembly line worker in a noisy factory
Who do you predict will have the better speechreading skills?

8. Characteristics of Good Speechreaders
Visual word decoding
Working memory (ability to store and manipulate items in memory simultaneously)
Lexical identification speed (determining whether letters are a word)
Phonological processing speed (whether two words rhyme)
Verbal inference (how well one can complete a sentence with missing words)
Age (Honnel et al 1991) (younger better)

9. Characteristics of Good Speechreaders
Ability to capitalize on contextual cues
Willingness to guess
Mental agility
Willingness to revise interpretations of partially recognized messages
Linguistic skills
World knowledge
Modal differences (Erber, 1974)
Neurophysiological (Summerfield, 1992)
Cognitive (Ronnberg et al, 1999)

10. Factors that Influence Speechreading Difficulty

11. Predictive Powers Visual Word Decoding Working Memory
May Have Predictive Power
May Not Have Predictive Power
May or May Not have Predictive Power
Visual Word Decoding
Working Memory
Lexical Identification Speed
Sentence Completion with Missing Words
Visual Working Memory and Processing Speed
Age
Gender IQ
Academic Achievement
Amount of Practice
Age and Duration of Onset of HL
Socioeconomic Status
Neurophysiologic Measures
Use of Contextual Cues
Willingness to Guess
Mental Agility
Revision Willingness of Contextual Cues
Linguistic and World Knowledge

12. Lipreading: Follow the Eyes
Lip shapes vary based on phoneme(s) produced
Eyebrows rise with questions
Tend to gaze at eyes, nose and mouth
Occasional looks at forehead, cheeks and chin
Prosodic information—lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

13. Gaze Saccades/Patterns

14. Gaze-direction-based MEG averaging during audiovisual speech perception

15. Lipreading: Follow the Eyes
Prosodic judgments:
lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

16. Lipreading: Follow the Eyes
Phonetic judgments:
monitored lower face such as lip and jaw movement

17. METS Talker Message Environment Speechreader
Facial expression length viewing angle lipreading skill
Diction syntactic complex distance residual hearing
Body language freq of word use lighting amplification
Speech rate homophenes room acoustics stress profile
Speech intensity context distractions attentiveness
Familiarity fatigue
to lipreader motivation
Facial characteristics language skills
Speech prosody
Objects in or over
mouth

18. Difficulty Lipreading
One third speech sounds visible
mid and back consonants invisible
Vowels not highly visible
Rapidity of speech – 150 to 250 word/min
Coarticulation
Stress can change appearance of word
Talker variability
Visemes and homophenes

19. Question Define viseme: Lip gloss Seeing eye dog
Groups of speech sounds that APPEAR identical on the lips
Words that LOOK identical on the mouth
E. C & D

20. Visemes and Homophenes
Groups of speech sounds that APPEAR identical on the lips
p b m – f v – s z – n t d s z – k g
Homophenes % of words
Words that LOOK identical on the mouth
pan ban man

21. Consonants Grouped as Visemes

22. Homophenous Word Pairs

23. Vowels and Lipreading Vowels not considered highly visible
Vowels tend to be audible
Intense, long duration
Front vowels
Lips flat or spread
Back vowels
Lips rounded

24. Production of Vowels
Cardinal Vowel Chart (YouTube)

25. Consonants Place of production Manner of production Voicing
Individuals with high frequency hearing loss may have difficulty hearing place cues
F2 transition ( Hz range)
Place cues tend to be visible
Bilabial, labiodental, linguadental sounds visible
Manner of production
Cues not visible, must be heard
F1 transition ( Hz range)
Voicing
Cues not visible
Low frequency range

26. Difficult Speaking Behaviors
Mumble
Looks away when speaking
Chews gum
Unusual accent
Speech impediment
Smiles too much
Moves around while talking
No facial expression
Shouts
High pitch voice
Talks to rapidly
Uses long complicated sentences
Wears a beard/mustache
Wears dark glasses

27. Talkers Easier to lipread someone familiar
Family members, teachers, etc.
Females easier to lipread than males
However, auditory plus vision may be more difficult as females are less audible to person with hearing loss

28. Question Clear speech…which is not a principle of?
Talk slightly faster
Talk slightly louder
Talk slightly slower
Talk with some pauses
None of the above

29. Talker Clear Speech Principles
Use clear speech versus conversational speech
Slightly slower talker rate
Slightly more intensity
Slightly more pauses after key words and phrases

30. Message
Structure – complexity of message, frequency of use, linguistic context
Frequency of usage – how often a word occurs in everyday conversations
Neighborhoods – fewer lexical neighbors can be beneficial
Context – words specified by context are easier

31. Lexical Neighborhoods

32. Environment Viewing angle – face to face Distance – favorable seating
Room conditions – lighting, lighting angle, shining light, interfering objects, room noise

33. Speechreader Audibility
Use of appropriate amplification system, ALDS, cochlear implant
Use of eyeglasses if necessary
Emotional and physical state

34. Tadoma Method: Speechreading
Deafblind Users of Tadoma Speechreading Method

36. Heather Whitestone: 1994
I find lip reading very stressful and frustrating because I am often confused. For example, if you look at person’s lips saying dog and saw, they look the same. With my hearing aid alone, I do not hear "s" or "d" sounds. So usually I have to use my common sense. For example, if someone said, "The dog is running across the street." Then I knew it was not the saw who ran across the street – it was the dog. Most hearing people do not understand that people in my position have to think incredibly fast in order to keep up with conversations. One-on-one conversations are not that stressful, but group conversations when coupled with background noise are nearly impossible. Lip reading is a grueling and exhausting mental exercise and lip readers are constantly thinking and trying to discern what is actually being said. I get real mad at those who think that I am stupid simply because I cannot hear. The truth is I get exhausted after a while and simply cannot keep up. At that point, I begin to guess at what is being said and eventually give up and choose to be quiet.

37. Baldi
Meet Baldi (iPhone app)
Baldi (2 iPhone app)

38. Question An oral interpreter?
Repeats message in view of person with hearing loss
Reads famous speeches
Speaks for person who uses ASL
All the above
None of the above

39. Oral Interpreters
A trained professional who sits in clear view of a person with a hearing loss and silently repeats a talker’s message as it is spoken

40. Bisensory Perceptions
What we see may influence what we hear
What we hear may influence what we see
McGurk Effect