Automating Early Assessment of Academic Standards for Very Young Native and Non-Native Speakers of American English better known as The TBALL Project (Technology Based Assessment of Language and Literacy) NSF-IERI award REC 0326214 Abeer Alwan (UCLA),Shrikanth Narayanan (USC), and David Pearson (UCB) IV. Reporting: What teachers see Speech Recognition Approach Recorded 300 children from 6 elementary schools; more than 80% of them are bilingual. Data used to train the ASR system. System has a 93% verification accuracy and correlates well with teacher scoring. ASR Approaches: Pronunciation modeling Speaker adaptation techniques Noise robust (front end and/or back end) Background Report of National Reading Panel (2000) advocates use of classroom-based assessments Many classroom assessments are not psychometrically sound Investment of time limits widespread use Distinguishing Features - Study the speech of native English and of Bilingual (Hispanic origin) K-2 students - Longitudinal and cross-sectional studies - Balance pedagogy and technology: strong interdisciplinary interactions between EE, CS, Education, Psychology and Linguistics - Correlation of measures with later performance - Validation of system, children’s performance, and teachers’ practices Spanish-accented English Field Testing Four schools in Los Angeles and one in Oakland with ELL populations; Two schools in Los Angeles, two in Oakland with native English speakers (total of 240 K-2 students) Very positive feedback from teachers in terms of students’ enthusiasm, ease of use of, and efficiency in conducting the tests I. Assessment Framework Many different aspects of reading skills Phonemic Awareness; Alphabet Identification; Letter-sound knowledge, Blending, Spelling, Segmenting. Word Recognition (Rate and Accuracy). Picture naming. Syntax, Comprehension Framework is hierarchical rather than a uniform approach to assessment All students take benchmark assessments Some students take 'drill down' assessments Teachers have guidance on what to assess TBALL Specific Aims Report of National Reading Panel (2000) advocates use of classroom-based assessments Many classroom assessments are not psychometrically sound Investment of time limits widespread use Acoustic phonetic knowledge transfer /v/ /f/ (very) /dh/ /d/ English Phoneme Acoustically similar Spanish Phonemes Map Think Listen Produce Jose /h ow z ey/ Joint /jh oy n t/ /h oy n t/ Orthographical knowledge transfer /j/ /h/ (word initially) III. HCI and Database Design Reading in Context Several children found it easier to read words in context than to identify them in a word list. Typical mispronunciations: Him/heem, as/us will/weell, read/reet, with/whit, by/be However, when the words were embedded in a sentence, the children had correct pronunciations. For example, the child that said ‘reet’ for ‘read’ had the proper pronunciation when he read : ‘I can read my book’ Syntax also plays a role. One child didn’t distinguish between ‘Came’ and ‘Come’. However, when he was presented the sentence: ‘I came to play but my friend said “ I will not play with you”, he read ‘came’ correctly. TBALL Team Vocab and Topic Knowledge Oral Lang. Comp. (Syntax) I.I. The Dam (decodable word list) Irregular Word list I.I. Displaying data for different groups and needs (including age-appropriate HCI design) Narrative Listening Comp. Schools: 5 LAUSD schools UES 2 Bay Area schools Narrative Oral Reading I.I. The Dam (decodable word list) K/1 High Frequency Word List NarrativeReading Comp San Diego High Frequency Word List Vocab and Topic Knowledge Written Lang. Comp. (Syntax) I.I. EE GSEIS CS EE Linguistics Psychology Education BPST Phonemic Awareness I.I. Rapid Naming Letter Sound Comp. 11 graduates & 7 undergraduate students, 6 teachers Query-based Datamining Database design allocates a place to put the collected data and its context, e.g., Demographic info from parent, date, TIME, test Content material from test Later the data can used for computations, e.g., Words in isolation correct: 21/51 = 41% Words in connected text: 20/36 = 55% Teacher or administer can easily access the data through queries, e.g., How did non-native speakers do relative to native speakers on task x? Did a child’s performance change from K to G1? Components II. Automatic Speech Recognition Challenges: Children have shorter vocal tract lengths (hence, higher resonances) and higher pitch Significant intra- and inter-speaker variability Significant variability in pronunciations due to different linguistic backgrounds, misarticulations, and signal to noise ratio of the recording environment How to distinguish reading errors from pronunciation differences Looking to the Future Beta version of the system, fall 2006; Alpha version, fall 2007 Assess other skills (mathematical and scientific reasoning) Extend the current system to other grade levels and language pairs Refine assessment tasks, materials, and automated techniques based on feedback from teachers and children Address validity, utility, and impact for native and non-native speakers Train teachers to use the system, deploy in more classrooms Please visit us at: http://diana.icsl.ucla.edu/TBALL/ Present auditory, text, graphical stimuli Measure decoding, comprehension skills Score, analyze, and adapt to responses Query-based datamining: monitor progress, compare, experiment Displays and summary screens for teachers to combine data to help make decisions Instructional guidance for teacher development