Uses for Automatic Speech Recognition with Diverse English Speakers 2002 American Speech-Language-Hearing Association Annual Convention Atlanta, Georgia World Congress Center, Room: A314, Saturday, Nov :30PM – 5:30PM Presenters/Authors: Kathleen Eilers Crandall, Ph.D., Paula M. Brown, Ph.D., Donna E. Gustina, and Stephen S. Campbell National Technical Institute for the Deaf Rochester Institute of Technology

Seminar – Presenters Kathleen Eilers crandall, Ph.D. Department of English, National Technical Institute for the Deaf, Rochester Institute of Technology Paula M. Brown, Ph.D., CCC-SLP Department of Speech and Language, National Technical Institute for the Deaf, Rochester Institute of Technology

The Glossograph Fay wrote about an experimental mechanical device used to transcribe human speech, and said, “… it is not unreasonable to hope that some instrument will yet be contrived …“ Fay, E.A. (1883). The glossograph. American Annals of the Deaf, 28,

Sci-Fi or Reality? "The pen was an archaic instrument, seldom used even for signatures...Apart from very short notes, it was usual to dictate everything into the speak-write…” (Nineteen eighty-four. Orwell, 1949)

Two Projects Teacher use of ASR: –English Classroom/Lab Project Student use of ASR: –Speech Project Funded by a grant from the Parsons Foundation of California

English Classroom/Lab Project

English Classroom/Lab Project Purpose Investigate direct use of ASR by classroom teacher to learn: Is acceptable recognition level attained? Under what conditions? –Style of speaking –Communication mode –Language complexity

Related Work Use of ASR by an intermediary Intermediary, a ‘captionist,’ re-speaks professor’s words into a computer Intermediary summarizes professor’s words into a computer (‘interpreted speech’) Intermediary may use C-print (a shorthand typing system) in combination with ASR

Related Work Use of ASR by the primary speaker iCommunicator™ _info.html _info.html Liberated Learning Environment (St. Mary’s University, Halifax, Nova Scotia)

Speech Project

Speech Project Intent Can ASR become better than a naïve listener? Can ASR serve as an effective and motivating feedback system?

Speech Project How ASR Is Used Educationally Visual displays provide feedback regarding speech production Natural way of learning Expect feedback to reflect accuracy –Assume if don’t get right picture, you were wrong

English Classroom/Lab Project

English Classroom/Lab Project Teacher -- Students Teacher -- Speaker –Native speaker of American English –User of ASL as a second language –Trained the ASR equipment Students -- Readers –Young adult college students who are deaf or hard-of- hearing –Reading and writing skills at the lowest quartile of entering students –Enrolled in basic level English language reading and writing courses

English Classroom/Lab Project Evaluation Procedures ASR Software: –Dragon Naturally Speaking –IBM ViaVoice –Microsoft Office Speaking styles: –Spontaneous conversation –Dictation-like speech Communication modes: –Speaking –Simultaneously speaking and signing

English Classroom/Lab Teacher station Control system Smart Board & LCD Projector Smart Board Student Stations

English Classroom/Lab Project Accuracy Needs Vary by population and message predictability –New vs. Known information –Fluent readers vs. Language learners –Reading for pleasure vs. Reading to master new information CLOZE research and prediction of missing information

English Classroom/Lab Project Results: ASR Software

English Classroom/Lab Project Results: Communication Mode

English Classroom/Lab Project Results: Language Complexity

English Classroom/Lab Project Correcting Text Error correction –What to correct –When to correct –How to correct

Multitasking Demands Normal tasks for speaker/teacher –Formulating ideas relevant to topic –Attending to learning needs of students –Meeting lipreading and sign language needs Added tasks for speaker/teacher –Speaking to produce readable ASR text –Monitoring text –Making corrections

Speech Project

Speech Project Training Sequence Read a paragraph Correct and train recognition errors Reread paragraph Correct and train recognition errors Create transfer paragraph or spontaneous speech Correct and train recognition errors

Recognition Accuracy

Improvement Across Sessions

Improvement Within Session

Speech Project Improvement Evaluated Improvement across sessions Improvement within a session –Improvement with speaker training –Improvement with ASR training

Recommendations Discussion Questions

Grammatical Correctness Is ASR accuracy affected by the grammatical correctness of the user’s speech? Student written responses spoken as written: Accuracy – 93.8% Student written responses spoken after corrected: Accuracy %

Style of Speaking 1.Style of speaking that more closely resembles dictation approaches a usable accuracy rate. 2.Lowering the complexity does not improve accuracy.

Conditions of Use Direct use of ASR by a language teacher -- Useful only under very controlled conditions. Illustrating the generation of written language Demonstrating the use of notes and outlines to produce written text Translating selected sign language utterances into English text during discussions

ASR: Classroom Use Prepared Outline Student’s Screen Teacher’s Screen

Considerations Training –Critical to reach over 90% accuracy –Training with conversation Corrections –Familiarity with strategies –Dictate, Spell, Right click Equipment –Microphone headsets - design, comfort, and size –Demand on computer processor –Effect of optional settings

Language Processing Teaching/Learning Issues: Does ASR promote the learning of reading and writing for Deaf and Hard-of-Hearing students? How do students process this information? Do students attend to multiple inputs? Can teachers attend to this many tasks effectively?

More Questions Who is at fault? –Speaker or ASR receiver? Acceptability of input –Various voices –Nontypical speakers User friendliness –Want immediate use

Presenters Kathleen Eilers Crandall, Ph.D. Department of English National Technical Institute for the Deaf Rochester Institute of Technology Lyndon Baines Johnson Building Phone: (585) Fax: (585) Web: Paula M. Brown, Ph. D., CCC-SLP Department of Speech and Language National Technical Institute for the Deaf Rochester Institute of Technology Lyndon Baines Johnson Building Phone: (585) V/TDD Fax: (585) Web: