1. Uses for Automatic Speech Recognition with Diverse English Speakers 2002 American Speech-Language-Hearing Association Annual Convention Atlanta, Georgia World Congress Center, Room: A314, Saturday, Nov 23 2002 4:30PM – 5:30PM Presenters/Authors: Kathleen Eilers Crandall, Ph.D., Paula M. Brown, Ph.D., Donna E. Gustina, and Stephen S. Campbell National Technical Institute for the Deaf Rochester Institute of Technology

2. Seminar – Presenters Kathleen Eilers crandall, Ph.D. Department of English, National Technical Institute for the Deaf, Rochester Institute of Technology Paula M. Brown, Ph.D., CCC-SLP Department of Speech and Language, National Technical Institute for the Deaf, Rochester Institute of Technology

3. The Glossograph Fay wrote about an experimental mechanical device used to transcribe human speech, and said, “… it is not unreasonable to hope that some instrument will yet be contrived …“ Fay, E.A. (1883). The glossograph. American Annals of the Deaf, 28, 67-69.

4. Sci-Fi or Reality? "The pen was an archaic instrument, seldom used even for signatures...Apart from very short notes, it was usual to dictate everything into the speak-write…” (Nineteen eighty-four. Orwell, 1949)

5. Two Projects Teacher use of ASR: –English Classroom/Lab Project Student use of ASR: –Speech Project Funded by a grant from the Parsons Foundation of California

6. English Classroom/Lab Project

7. English Classroom/Lab Project Purpose Investigate direct use of ASR by classroom teacher to learn: Is acceptable recognition level attained? Under what conditions? –Style of speaking –Communication mode –Language complexity

8. Related Work Use of ASR by an intermediary Intermediary, a ‘captionist,’ re-speaks professor’s words into a computer Intermediary summarizes professor’s words into a computer (‘interpreted speech’) Intermediary may use C-print (a shorthand typing system) in combination with ASR http://cprint.rit.edu/ http://cprint.rit.edu/

9. Related Work Use of ASR by the primary speaker iCommunicator™ http://www.myicommunicator.com/product _info.html http://www.myicommunicator.com/product _info.html Liberated Learning Environment http://www.liberatedlearning.com (St. Mary’s University, Halifax, Nova Scotia) http://www.liberatedlearning.com

10. Speech Project

11. Speech Project Intent Can ASR become better than a naïve listener? Can ASR serve as an effective and motivating feedback system?

12. Speech Project How ASR Is Used Educationally Visual displays provide feedback regarding speech production Natural way of learning Expect feedback to reflect accuracy –Assume if don’t get right picture, you were wrong

13. English Classroom/Lab Project

14. English Classroom/Lab Project Teacher -- Students Teacher -- Speaker –Native speaker of American English –User of ASL as a second language –Trained the ASR equipment Students -- Readers –Young adult college students who are deaf or hard-of- hearing –Reading and writing skills at the lowest quartile of entering students –Enrolled in basic level English language reading and writing courses

15. English Classroom/Lab Project Evaluation Procedures ASR Software: –Dragon Naturally Speaking –IBM ViaVoice –Microsoft Office Speaking styles: –Spontaneous conversation –Dictation-like speech Communication modes: –Speaking –Simultaneously speaking and signing

16. English Classroom/Lab Teacher station Control system Smart Board & LCD Projector Smart Board Student Stations

17. English Classroom/Lab Project Accuracy Needs Vary by population and message predictability –New vs. Known information –Fluent readers vs. Language learners –Reading for pleasure vs. Reading to master new information CLOZE research and prediction of missing information

18. English Classroom/Lab Project Results: ASR Software

19. English Classroom/Lab Project Results: Communication Mode

20. English Classroom/Lab Project Results: Language Complexity

21. English Classroom/Lab Project Correcting Text Error correction –What to correct –When to correct –How to correct

22. Multitasking Demands Normal tasks for speaker/teacher –Formulating ideas relevant to topic –Attending to learning needs of students –Meeting lipreading and sign language needs Added tasks for speaker/teacher –Speaking to produce readable ASR text –Monitoring text –Making corrections

23. Speech Project

24. Speech Project Training Sequence Read a paragraph Correct and train recognition errors Reread paragraph Correct and train recognition errors Create transfer paragraph or spontaneous speech Correct and train recognition errors

25. Recognition Accuracy

26. Improvement Across Sessions

27. Improvement Within Session

28. Speech Project Improvement Evaluated Improvement across sessions Improvement within a session –Improvement with speaker training –Improvement with ASR training

29. Recommendations Discussion Questions

30. Grammatical Correctness Is ASR accuracy affected by the grammatical correctness of the user’s speech? Student written responses spoken as written: Accuracy – 93.8% Student written responses spoken after corrected: Accuracy - 94.3%

31. Style of Speaking 1.Style of speaking that more closely resembles dictation approaches a usable accuracy rate. 2.Lowering the complexity does not improve accuracy.

32. Conditions of Use Direct use of ASR by a language teacher -- Useful only under very controlled conditions. Illustrating the generation of written language Demonstrating the use of notes and outlines to produce written text Translating selected sign language utterances into English text during discussions

33. ASR: Classroom Use Prepared Outline Student’s Screen Teacher’s Screen

34. Considerations Training –Critical to reach over 90% accuracy –Training with conversation Corrections –Familiarity with strategies –Dictate, Spell, Right click Equipment –Microphone headsets - design, comfort, and size –Demand on computer processor –Effect of optional settings

35. Language Processing Teaching/Learning Issues: Does ASR promote the learning of reading and writing for Deaf and Hard-of-Hearing students? How do students process this information? Do students attend to multiple inputs? Can teachers attend to this many tasks effectively?

36. More Questions Who is at fault? –Speaker or ASR receiver? Acceptability of input –Various voices –Nontypical speakers User friendliness –Want immediate use

37. Presenters Kathleen Eilers Crandall, Ph.D. Department of English National Technical Institute for the Deaf Rochester Institute of Technology Lyndon Baines Johnson Building - 2264 Phone: (585) 475-5111 Fax: (585) 475-6500 Email: kecncp@rit.edukecncp@rit.edu Web: http://www.rit.edu/~kecncphttp://www.rit.edu/~kecncp Paula M. Brown, Ph. D., CCC-SLP Department of Speech and Language National Technical Institute for the Deaf Rochester Institute of Technology Lyndon Baines Johnson Building - 3851 Phone: (585) 475-6593 V/TDD Fax: (585) 475-6500 Email: pmbnci@rit.edupmbnci@rit.edu Web: http://www.rit.edu/~462www/http://www.rit.edu/~462www/