Refined Micro-analysis of Fluency Gains in a Reading Tutor that Listens Jack Mostow* and Joseph Beck Project LISTEN (www.cs.cmu.edu/~listen) Carnegie Mellon University * Consultant and Scientific Advisory Board Chair, Soliloquy Learning Society for the Scientific Study of Reading 13th Annual Meeting, July, 2006 Funding: National Science Foundation, Heinz Endowments Add SL affiliation 1

Research questions and approach Guided oral reading builds fluency [NRP 00] Typically repeated oral reading … but how do its benefits vary? How good is repeated vs. wide reading? How good is massed vs. spaced practice? How do the answers vary with student proficiency? Approach: micro-analyze oral reading data Massive: hundreds of children Longitudinal: entire school year Fine-grained: word by word Massive: hundreds of children Fine-grained: individual words Longitudinal: entire school year 2

Project LISTEN’s Reading Tutor: Rich source of guided oral reading data Massive 650 students age 5-14 Mostly grades 1-4 Longitudinal 2003-2004 school year 55,000 sessions Fine-grained 6.9 million words “Heard” by recognizer Get hours from Joe; say less Video at www.cs.cmu.edu/~listen 3

Reading speeds up with practice: example Initial encounter of muttered: I’ll have to mop up all this (5630 ms) muttered Dennis to himself but how 5 weeks later (different word pair in different sentence): Dennis (110 ms) muttered oh I forgot to ask him for the money Word reading time = latency + production time  1/fluency How does word reading time change in general? 4

Learning curve for mean reading time of first 20 encounters, excluding top 50 words Do some types of encounters help more than others? 5

Four types of word encounters New context? First time today? 1. Read muttered in a new story. Wide Spaced 2. Read muttered in another sentence. Massed 3. On a later day, reread sentence 1. Reread 4. Then reread sentence 2. Predict reading time for 770,858 type 1 encounters from prior encounters of all 4 types. 6

Predictor variables Number of word encounters so far of each type Wide vs. reread Spaced vs. massed Word difficulty # of letters # of past help requests (controls for difficulty for that student) Student proficiency WRMT Word Identification grade-equivalent score, e.g. 2.3 Interpolated for each encounter from pre- and post-test scores 7

Exponential model of word reading time = L * # letters + (P * proficiency + constant A) * e - learning rate B *Exposure Define weights for each type of encounter r for rereading vs. 1 for wide reading m for massed vs. 1 for spaced h for help requests Exposure = weighted sum of # of word encounters so far 1 * # of wide, spaced encounters + r * # of reread, spaced encounters + m * # of wide, massed encounters + r * m * # of reread, massed encounters + h * # of help requests [Beck, J. Using learning decomposition to analyze student fluency development. ITS2006 Educational Data Mining Workshop, Taiwan.] 8

Analysis Use SPSS non-linear regression to fit parameters Caveat: 770,858 trials are not independent So be conservative: Split 650 students into 10 groups Fit r, m, … for each group From the 10 estimates of each parameter, compute: Mean ± standard error Differs significantly from 1? A, L, P, B, h, r, m 9

Overall results Wide reading beats rereading r < 1 (p = .007) 2 new stories ≈ 3 old stories Spaced beats massed practice m = .67 ± .13 m < 1 (p = .007) 2 spaced encounters ≈ 3 massed encounters Do these results vary by proficiency? 10

Effects of proficiency Bottom third Middle third Top third Word ID GE 1.8 (0-2.3) 2.7 (2.3-3.1) 4.5 (3.1-10.2) Reread (r) .93 ± .23 .99 ± .23 .79 ± .25 Massed (m) 1.73 ± .46 * .41 ± .08 ** .41 ± .21 ** When does wide reading beat rereading? Maybe only for high readers? Seeing a word again the same day May help low readers more than waiting (p = .058) Helps higher readers less than seeing it later (p < .01) Say significantly different from 1. The lowest group had a WRMT word ID score of 0 to 2.3 with a mean of 1.8 and SD of 0.47. Medium was 2.3 to 3.1 with a mean of 2.7 and SD of 0.21. The high proficiency group scores ranged from 3.1 to 10.2, with a mean of 4.5 and SD of 1.38. 11

Conclusion: type of practice matters! Wide reading beats rereading At least for higher readers Advantage of spaced practice varies with proficiency Low readers: seeing a word again the same day may help more Higher readers: better to wait Fluency growth is slow (learning curve is gradual) So differences in practice quality are hard to detect But possible by micro-analysis of massive, longitudinal, fine-grained data Future work Clarify interaction with proficiency Refine model of fluency practice Test correlational results experimentally Thank you! Questions? See papers & videos at www.cs.cmu.edu/~listen To do: 1. Vet revised conclusions. The results are more intuitive now, but how much should we believe them given the shift due to averaging differently? 2. The by-proficiency results contradict the overall result on rereading. What to say? 3. Thanks for "compare asr with transcription.xls". I was disappointed that the correlation was so low and error was so high. How long did the queries take, especially joining lex_view with aligned_word? I'm curious to compare to text-space false alarm rates. 4. lex_view for 2005_2006? Joe doesn't know, but will email when done. 12

Predictive models of word reading in text SSSR2005 SSSR2006 Predict Growth from word encounter i to i+1 Performance at encounter i+1 Outcome Reading time speedup Reading time, errors, help requests Predictor Encounter i of word Encounters 1..i Model Linear Exponential 13

Outcome variable Combine reading time, errors, help requests Cap reading time at 3 seconds (0.1% of data) Treat error as 3 seconds Treat help request as 3 seconds the following words were excluded: The 50 most frequent words in English Encounters after the first 20 with a word There were multiple reasons to exclude these data, including running out of memory (even being clever about which columns were really needed) and altering the shape of the learning curve due to a poor model of prior encounters. 3. The first word of a sentence. Since we define reading time = latency + production, and latency is undefined for the first word of the sentence, we do not model first word performance. 4, One character "words" as they might be spelling. 14