Micro-analysis of Fluency Gains in a Reading Tutor that Listens: Jack Mostow and Joseph Beck Project LISTEN (www.cs.cmu.edu/~listen) Carnegie Mellon University Society for the Scientific Study of Reading 12th Annual Meeting, June, 2005 Funding: National Science Foundation, Heinz Endowments 1

Wide vs. repeated guided oral reading Guided oral reading builds fluency [NRP 00]. Typically repeated oral reading Is repeated reading better than wide reading? Unclear! [Kuhn & Stahl JEP 03] Past work analyzed reading rates on passages and word lists. E.g. scramble word order to study context effects [Levy, …] This talk: use finer-grained Reading Tutor data. 2

Project LISTEN’s Reading Tutor: Rich source of guided oral reading data 2002-2003 database: 8 diverse schools 600 students (K-6) 26,000 sessions 600,000 sentences read 4 million words heard Get hours from Joe 3

Reading speeds up with practice: example Initial encounter of muttered: I’ll have to mop up all this (5630 ms) muttered Dennis to himself but how 5 weeks later (different word pair in different sentence): Dennis (110 ms) muttered oh I forgot to ask him for the money Word reading time = latency + production time  1/fluency Beck et al. [TICL 04] used latency to assess proficiency (R2>.8). What predicts word reading time? 4

Proficient readers are faster. 5

Long words are slower. 6

Reading time speeds up over successive encounters, but by less and less. 7

Linear model of word reading speedup Predicted variable is speedup on the same word Reduction in word reading time From one (“practice”) encounter of a word To the next (“test”) encounter N = 243,172 opportunities for speedup By 352 students (gr 1-6) with WRMT pretest scores Include rereading as practice but not as test Exclude encounters after the first 8 Exclude 36 stop words (the, a, …) 8

Predictors of speedup What is the student’s reading level? How many letters long is the word? How often has the student seen the word before? 9

Contextual predictors Has the student seen the practice sentence before? (Students have not seen test sentences before.) Has the student seen the practice word pair before? Has the student seen the test word pair before? 10

Results Higher readers speed up less: 3 ms less per grade level. Longer words speed up more: 2.4 ms more per letter. Speedup averages 18 ms per encounter (for the first seven). Speedup averages 30 ms after the first encounter, then 3 ms less after each subsequent encounter. 11

Context effects A new practice sentence helps 27 ms more than an old one. Wide reading beats rereading! A new practice word pair helps 21 ms more. A test word pair seen before is 13 ms faster. 12

Are these effects statistically reliable? Standard errors are 0~3 ms. But: Linear model ignores dependencies Within student (other than proficiency) Within word (other than length) So: Compute p by counting N students, not encounters. Restrict to students with 50+ encounters in each condition Compare how many students do better in one condition vs. the other. Use sign test to compute significance. New practice sentences beat old ones for 100 of 135 students (p = .000). No marked differences between groups New practice word pairs beat old ones for 151 of 222 students (p = .000). 13

Does word speedup relate to test score gains? In contrast, # encounters doesn’t predict gains (R2<.001). 14

What affects fluency growth? Practice on specific words Especially in new sentences Practice on specific word pairs Helps if tested, but new practice pairs are better Practice in decoding new words Not analyzed in this talk 15

Conclusions Wide guided oral reading beats repeated reading! Correlational, not experimental; what uncontrolled sample bias? Excludes rereading but not recency effects Relevant to human-guided oral reading? Is rereading more motivating for poor readers? Wide reading requires more text and more guidance! Micro-analysis of tutor data finds subtle (ms) effects Large, fine-grained, longitudinal sample Speech recognition is imperfect But unlikely to be biased wrt our variables Thank you! Questions? See papers & videos at www.cs.cmu.edu/~listen. 16

Thanks to fellow LISTENers & friends Tutoring: Dr. Joseph Beck, mining tutorial data Prof. Albert Corbett, cognitive tutors Becky Kennedy, linguist Joe Valeri, activities and interventions Listening: Dr. Evandro Gouvea, acoustic training John Helman, transcriber Dr. Mosur Ravishankar, speech recognizer Programmers: Andrew Cuneo, application Field staff: Kristin Bagwell Julie Sleasman Dr. Roy Taylor Grad students: Kai-min Chang, LTI Cecily Heiner, MCALL Ayorkor Mills-Tettey, RI Interns: Alisa Grishman Brooke Hensler James Leszczenski Rachel Minkoff Research partners: DePaul U. Toronto U. British Columbia Ghana Advisory board Kathryn Ayres, children’s stories Rollanda O’Connor, reading Charles Perfetti, reading … Schools www.cs.cmu.edu/~listen 17