Chapter 6 Audiovisual Speech Perception

Slides:



Advertisements
Similar presentations
Improving audibility as a foundation for better speech understanding Pamela Souza, PhD Northwestern University Evanston, IL.
Advertisements

Human Speech Recognition Julia Hirschberg CS4706 (thanks to John-Paul Hosum for some slides)
Language and Cognition Colombo, June 2011 Day 8 Aphasia: disorders of comprehension.
Chapter 8 Auditory Training Perry C. Hanavan, Au.D.
The nature of sound Types of losses Possible causes of hearing loss Educational implications Preparing students for hearing assessment.
Speech Perception Overview of Questions Can computers perceive speech as well as humans? Does each word that we hear have a unique pattern associated.
Audiovisual benefit for stream segregation in elderly listeners Esther Janse 1,2 & Alexandra Jesse 2 1 Utrecht institute of Linguistics OTS 2 Max Planck.
CSD 5400 REHABILITATION PROCEDURES FOR THE HARD OF HEARING
CSD 5400 REHABILITATION PROCEDURES FOR THE HARD OF HEARING Auditory Training.
CSD 3000 DEAFNESS IN SOCIETY Topic 6 The Deaf Adult.
Learning Styles.
Los Angeles Unified School District Division of Special Education Schools for All Children Deaf and Hard of Hearing Donnalyn Jaque-Antón Associate Superintendent.
© 2014 wheresjenny.com Lip reading LIP READING. © 2014 wheresjenny.com Lip reading Vocabulary Decipher : Succeed in understanding, interpreting, or identifying.
Hearing Impairments By: Nayeli Padilla. What is it? Hearing impairment: problem/damage to one or more parts of the ear.
1 Speech Perception 3/30/00. 2 Speech Perception How do we perceive speech? –Multifaceted process –Not fully understood –Models & theories attempt to.
Assessing Listening.
Lipreading: how it works. Learning objectives Recognise the different processes and skills involved in lipreading Revise factors that help or hinder lipreading.
Soundfield Amplification Soundfield Amplification Kirsten Marconi-Hutkay, Au.D., CCC-A Educational Audiologist Stark County Educational Service Center.
Speech Perception 4/4/00.
Deaf Culture and American Sign Language Hello lderpictures/.pond/clipartASL.jpg.w180h123.jpg.
Decoding Dyslexia Parent Support Group October,
Chapter 9 Audio-Visual Technologies Perry C. Hanavan.
COMMUNICATING EFFECTIVELY Chapter 4. Communication to me is,………
What can we expect of cochlear implants for listening to speech in noisy environments? Andrew Faulkner: UCL Speech Hearing and Phonetic Sciences.
Effective Communication Skills for the Deaf and Hard of Hearing Employee & Organizational Development Diversity Training Module.
Speechreading Based on Tye-Murray (1998) pp
Listening. Communication Implies at least 2 people: One to create a meaningful message and one to recreate the message.
CLIENT COMMUNICATIONS. Definition of Communication  Webster’s dictionary defines communication as “to give, or give and receive, information, signals,
Deaf Awareness Training Beth Elkins Personal Support Manager Cambridge Regional College.
LISTENING: QUESTIONS OF LEVEL FRANCISCO FUENTES NICOLAS VALENZUELA.
American Sign Language (ASL)
Teaching pronunciation
School-Age Children with HL Chapter 14
1.02 Factors that affect communication
1.02 Factors that affect communication
Ups and Downs Southwest Conference 2007
Prosody and Non- Verbal Communication
Chapter 16 Audiovisual Speech Perception
Communication Skills COMM 101 Lecture#2
Chapter 5 Connecting through Nonverbal Communication.
Why can’t Deaf people just speech read?
is what ?? It is a process of exchanging – Information Ideas Thoughts
English Phonetics and Phonology
Mrs. Jameson ASL A Story About the Ear Mrs. Jameson.
1.02 Factors that affect communication
Mrs. Jameson ASL A Story About the Ear Mrs. Jameson.
Communication Influences/Barriers 1.02 PP2
Communication with the Elderly
Dr. Susan Easterbrooks Professor, GSU Dr. Nanci Scheetz Professor, VSU
Communication.
Professional Communications
Listening Skill Pertemuan ke-3.
1.02 Factors that affect communication
1.02 Factors that affect communication
An Introduction to Speechreading
1.02 Factors that affect communication
Communication is an integral part of quality healthcare
Hearing Loss 101 Billie Wortham Wyoming Department of Education.
1.02 Factors that affect communication
OTHERS: COMMUNICATION
Factors that affect communication
Communication Skills for the Healthcare Professional
Communications Haven, Yovannca.
Speechreading Perry C. Hanavan.
Grade 8 Life Orientation
Tinnitus activities THERAPY
Follow-up: hearing and communication
Aging, Hearing Loss and Amplification: Beyond the Audiogram
Sensorineural Loss & Dementia
Presentation transcript:

Chapter 6 Audiovisual Speech Perception Perry C. Hanavan, Au.D.

Questions Lipreading and speechreading difined the same in the text book? True False

Definitions Lipreading Speechreading the person relies ONLY on the visual speech signal and other visual cues provided by the talker Speechreading the person attends to both the visual AND auditory cues provided by the talker

General Tendencies Normal hearing persons rely on speechreading Persons listening in background noise tend to use visual cues for speech recognition Persons with hearing loss tend to rely more on the visual signal for speech recognition The greater the hearing loss, the more the tendency for reliance on visual cues

Speechreading Lip cues Facial expression cues Gesture cues Body language cues Linguistic Situational cues Auditory cues*

Speechreading for Communication Normal hearing adults Infants

Afternoon Clinic Appointments You arrive at work following lunch and find two patients waiting for clinical services. Dr. White is a 50 year old physician Mr. Black is a 20 year old assembly line worker in a noisy factory Who do you predict will have the better speechreading skills?

Characteristics of Good Speechreaders Visual word decoding Working memory (ability to store and manipulate items in memory simultaneously) Lexical identification speed (determining whether letters are a word) Phonological processing speed (whether two words rhyme) Verbal inference (how well one can complete a sentence with missing words) Age (Honnel et al 1991) (younger better)

Characteristics of Good Speechreaders Ability to capitalize on contextual cues Willingness to guess Mental agility Willingness to revise interpretations of partially recognized messages Linguistic skills World knowledge Modal differences (Erber, 1974) Neurophysiological (Summerfield, 1992) Cognitive (Ronnberg et al, 1999)

Factors that Influence Speechreading Difficulty

Predictive Powers Visual Word Decoding Working Memory May Have Predictive Power May Not Have Predictive Power May or May Not have Predictive Power Visual Word Decoding Working Memory Lexical Identification Speed Sentence Completion with Missing Words Visual Working Memory and Processing Speed Age Gender IQ Academic Achievement Amount of Practice Age and Duration of Onset of HL Socioeconomic Status Neurophysiologic Measures Use of Contextual Cues Willingness to Guess Mental Agility Revision Willingness of Contextual Cues Linguistic and World Knowledge

Lipreading: Follow the Eyes Lip shapes vary based on phoneme(s) produced Eyebrows rise with questions Tend to gaze at eyes, nose and mouth Occasional looks at forehead, cheeks and chin Prosodic information—lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

Gaze Saccades/Patterns

Gaze-direction-based MEG averaging during audiovisual speech perception

Lipreading: Follow the Eyes Prosodic judgments: lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

Lipreading: Follow the Eyes Phonetic judgments: monitored lower face such as lip and jaw movement

METS Talker Message Environment Speechreader Facial expression length viewing angle lipreading skill Diction syntactic complex distance residual hearing Body language freq of word use lighting amplification Speech rate homophenes room acoustics stress profile Speech intensity context distractions attentiveness Familiarity fatigue to lipreader motivation Facial characteristics language skills Speech prosody Objects in or over mouth

Difficulty Lipreading One third speech sounds visible mid and back consonants invisible Vowels not highly visible Rapidity of speech – 150 to 250 word/min Coarticulation Stress can change appearance of word Talker variability Visemes and homophenes

Question Define viseme: Lip gloss Seeing eye dog Groups of speech sounds that APPEAR identical on the lips Words that LOOK identical on the mouth E. C & D

Visemes and Homophenes Groups of speech sounds that APPEAR identical on the lips p b m – f v – s z – n t d s z – k g Homophenes 40-60 % of words Words that LOOK identical on the mouth pan ban man

Consonants Grouped as Visemes

Homophenous Word Pairs

Vowels and Lipreading Vowels not considered highly visible Vowels tend to be audible Intense, long duration Front vowels Lips flat or spread Back vowels Lips rounded

Production of Vowels Cardinal Vowel Chart (YouTube)

Consonants Place of production Manner of production Voicing Individuals with high frequency hearing loss may have difficulty hearing place cues F2 transition (1000-2400 Hz range) Place cues tend to be visible Bilabial, labiodental, linguadental sounds visible Manner of production Cues not visible, must be heard F1 transition (250-1000 Hz range) Voicing Cues not visible Low frequency range

Difficult Speaking Behaviors Mumble Looks away when speaking Chews gum Unusual accent Speech impediment Smiles too much Moves around while talking No facial expression Shouts High pitch voice Talks to rapidly Uses long complicated sentences Wears a beard/mustache Wears dark glasses

Talkers Easier to lipread someone familiar Family members, teachers, etc. Females easier to lipread than males However, auditory plus vision may be more difficult as females are less audible to person with hearing loss

Question Clear speech…which is not a principle of? Talk slightly faster Talk slightly louder Talk slightly slower Talk with some pauses None of the above

Talker Clear Speech Principles Use clear speech versus conversational speech Slightly slower talker rate Slightly more intensity Slightly more pauses after key words and phrases

Message Structure – complexity of message, frequency of use, linguistic context Frequency of usage – how often a word occurs in everyday conversations Neighborhoods – fewer lexical neighbors can be beneficial Context – words specified by context are easier

Lexical Neighborhoods

Environment Viewing angle – face to face Distance – favorable seating Room conditions – lighting, lighting angle, shining light, interfering objects, room noise

Speechreader Audibility Use of appropriate amplification system, ALDS, cochlear implant Use of eyeglasses if necessary Emotional and physical state

Tadoma Method: Speechreading Deafblind Users of Tadoma Speechreading Method

Heather Whitestone: 1994 I find lip reading very stressful and frustrating because I am often confused. For example, if you look at person’s lips saying dog and saw, they look the same. With my hearing aid alone, I do not hear "s" or "d" sounds. So usually I have to use my common sense. For example, if someone said, "The dog is running across the street." Then I knew it was not the saw who ran across the street – it was the dog. Most hearing people do not understand that people in my position have to think incredibly fast in order to keep up with conversations. One-on-one conversations are not that stressful, but group conversations when coupled with background noise are nearly impossible. Lip reading is a grueling and exhausting mental exercise and lip readers are constantly thinking and trying to discern what is actually being said. I get real mad at those who think that I am stupid simply because I cannot hear. The truth is I get exhausted after a while and simply cannot keep up. At that point, I begin to guess at what is being said and eventually give up and choose to be quiet.

Baldi Meet Baldi (iPhone app) Baldi (2 iPhone app)

Question An oral interpreter? Repeats message in view of person with hearing loss Reads famous speeches Speaks for person who uses ASL All the above None of the above

Oral Interpreters A trained professional who sits in clear view of a person with a hearing loss and silently repeats a talker’s message as it is spoken

Bisensory Perceptions What we see may influence what we hear What we hear may influence what we see McGurk Effect