1. Applying New Voice Recognition Technology to Formative Assessment Margaret Heritage UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing (CRESST) Markus Iseli Henry Samueli School of Engineering, UCLA CRESST Conference, Los Angeles, CA September 8th, 2005,

2. Overview Project Aims and Components Features of the Program Automatic Speech Recognition Technology Interface Demo Assessment Framework Looking to the Future

3. Project Components Develop speech recognition technology for children Apply technology to create an on-demand, easy-to-use system of assessment in reading for students in grades K-2 Develop system capacity to present auditory, text, graphical stimuli, and to score, analyze and adapt to responses Develop query-based data mining to monitor students’ progress Develop easy-to-understand displays of data analysis

4. Specific Aims Develop assessment system that : Is helpful for teachers (i.e. has instructional utility and saves time) Reduces variability (e.g., consistent instructions, consistent delays, consistent scoring) Automatically scores and analyzes children’s performance on reading assessment tasks

5. Distinguishing Features Strong interdisciplinary interactions among electrical engineering, computer science, education, psychology and linguistics Collaboration with expert teachers Focus on bilingual (Mexican-Spanish accented English) students Validation of the system

6. Lens of Project Information that teachers can use next day in their instruction? Sensitivity to English Language Learners(ELLs) Sensitivity to Language Knowledge

7. Instructional Utility Effective classroom is assessment-centered (NRC, 2000) ongoing assessment of students’ learning that provides the day-to-day fuel for instruction Formative assessment ‘ used to adapt the teaching work to meet the learning needs’ (Black, Harrison, Lee, Marshall, & Wiliam, 2003, p.2).

8. Reading error or pronunciation difference? In Spanish, compared to English Phonetics p t k closer to Eng b d g than to p t k s z n t d: tongue on teeth, not behind them Sounds missing: th, oy, etc. Phonology s+ptkbdg only across syllables Distinctions like ‘bit-beat’ not made Literacy Words spelled ‘y’ pronounced ‘j’, (by some) Words spelled ‘i’ pronounced ‘ee’, etc.

9. Academic Language Among the factors contributing to non-comprehension of text: inadequate knowledge of the words used, lack of familiarity with the syntactic structures ( Lyon, 1998)

10. Automatic Speech Recognition (ASR) Teach the computer to understand human speech OR Teach the human being how to talk to be understood by a computer Three main challenges: Speaker: gender, pronunciation, health, dialect, language Environment: noise, other speech System: devices, program

11. Child vs. Adult Speech Children don’t talk just to be understood by the computer, they just talk! Children cannot yet control their articulators as well as adults Children have different anatomical features (shorter vocal tract), and these features change fast Children have very high pitch frequency

12. Designing an ASR System Questions Who is going to use the ASR system? In what environment will it be used? Implementation Collect “a lot” of appropriate data Train the system Test the system and make changes

13. Our Assessment System The system will: Assess each student in the same manner (no dependence on teacher) Produce visual stimuli (letters, words, phrases) and record the child’s vocal response Measure response times very accurately Analyze the recorded audio and other data to generate reports which are useful to teachers (ASR) Be easily accessible (internet) Handle multiple users at the same time

14. System Architecture Interface Client SideServer Side

15. More detailed…

16. Flash Interface: Live Demo http://kittychan.icsl.ucla.edu/tball Flash interface design by Larry Casey

17. System Design Benefits Accessible Software: common web browser Hardware: standard microphone Flexible Command structure is open-ended Allows for any audio-visual testing set-up Stable Constant audio stream: everything is captured Stimulus/response data is recorded in real time Scalable Content, display, navigation are independent

18. Assessment Framework Recall the lens Guiding questions: Are the assessments embedded in an instructional framework? What is the instructional value of the information? How much assessment is too much?

19. Assessment Framework Guiding questions privilege a hierarchical rather than a uniform approach to assessment. All students take benchmark assessments as a check on progress Some students take 'drill down' assessments related to specific skills on an as-needed based for diagnosis Teachers have guidance on what to assess

20. Assessment Framework Skills Assessed: Phonemic Awareness Word recognition Oral reading (accuracy and rate) Comprehension Syntax

21. Assessment Framework Links to: English Language Development Standards English Language Arts Standards

22. Narrative Oral Reading Narrative Reading Comp Narrative Oral Reading Narrative Reading Comp Basic Monitoring Spine of Reading Assessment Framework Narrative Oral Reading Narrative Reading Comp

23. The Dam (decodable word list) K/1 High Frequency Word List Rapid Naming Letter Sound Comp. BPST Narrative Oral Reading Narrative Reading Comp Narrative Listening Comp. After student demonstrates mastery of listening comprehension and word reading tasks, begin assessing in connected text skills. After student demonstrates mastery of letter sound and naming tasks, begin assessing regular and irregular word reading. 1. 2. 3. 4. Basic Reading Assessment Framework - Kindergarten Narrative Oral Reading Narrative Reading Comp Begin the framework with screening assessments in listening comprehension, letter sound, and naming tasks. Repeat assessing connected text skills throughout year as needed. If problem arises, recheck word reading development

24. I.I. Reading Assessment Framework with Interventions - Kindergarten Phonemic Awareness I.I. The Dam (decodable word list) I.I. Irregular Word list Narrative Listening Comp. The Dam (decodable word list) K/1 High Frequency Word List Rapid Naming Letter Sound Comp. BPST If student demonstrates skill level below mastery, provide instructional intervention and reassess. Narrative Oral Reading Narrative Reading Comp Continue assessing connected text skills throughout year as needed. Vocab and Topic Knowledge I.I. Written Lang. Comp. (Syntax) Narrative Oral Reading Narrative Reading Comp Oral Lang. Comp. I.I. Vocab and Topic Knowledge

25. Oral Language Comp. San Diego High Frequency Word List BPST Narrative Oral Reading Narrative Reading Comp Repeat assessing connected text skills throughout year as needed. If problem with progress arises, recheck word reading development. After student demonstrates mastery of listening comprehension, letter sound, and word reading tasks, begin assessing in connected text skills. Begin the framework with screening assessments in listening comprehension, letter sound, and word reading tasks. 1. 2.3. 1st and 2nd Grades Narrative Oral Reading Narrative Reading Comp

26. San Diego High Frequency Word List BPST Narrative Listening Comp. Reading Assessment Framework with Interventions- First and Second Grades The Dam (decodable word list) Irregular Word list I.I. Vocab and Topic Knowledge Written Lang. Comp. (Syntax) I.I. Vocab and Topic Knowledge Oral Lang. Comp. (Syntax) I.I. The Dam (decodable word list) K/1 High Frequency Word List I.I. Phonemic Awareness I.I. Rapid Naming I.I. Letter Sound Comp. I.I. Continue assessing connected text skills throughout year as needed. Narrative Oral Reading Narrative Reading Comp

27. Example Kindergarten/1 st grade BPST Blending Words with: Shortmapripmetrub mop Final-efineroperakecutekite Longsoapleakpainfeedray r-controlfursortsirtarserve OVDcoinmoon round lawn foot 2 syllablesilent ladder napkin locate cactus

28. Reporting

29. Looking to the Future Validation of assessment system Development of ASR to include discourse level performances Leverage other CRESST technology( QSP) Applications to other domains ( e.g., math and science) Applications to other grade levels