Carnegie Mellon Project LISTEN16/29/2004 Which Help Helps? Effects of Various Types of Help on Word Learning in an Automated Reading Tutor that Listens Jack Mostow, Joseph E. Beck, Cecily Heiner Project LISTEN ( Carnegie Mellon University Presented 6/29/2004 at 11 th Annual Meeting of the Society for the Scientific Study of Reading, Amsterdam. This work was supported in part by the National Science Foundation under ITR/IERI Grant No. REC Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation or the official policies, either expressed or implied, of the sponsors or of the United States Government.

Carnegie Mellon Project LISTEN26/29/2004 Previous findings on word help Spaai (1991):  Whole word feedback beat sounding out Wise (1992):  Whole word, syllables, sub-syllables beat sounding out  Onset + Rime (c+at) best for less severely disabled  Syllables best for more severely disabled But:  Limited samples of students  Limited data per student

Carnegie Mellon Project LISTEN36/29/2004 Project LISTEN’s Reading Tutor: A rich source of experimental data The Reading Tutor beats independent practice…  Effect sizes up to 1.3 (Mostow SSSR02, Poulsen 04)Mostow SSSR02Poulsen 04 … but how? Embed an experiment to find out!  9 schools ( )  200 computers  600 students, age 6-12  Average 9.2 hours each  Detailed interaction logs  460,000 words helped  5 million words read aloud  Scored using automatic speech recognition (ASR)  “Invisible experiments”  Randomized trials Frey, K. (October 17, 2003). CMU's Automated Reading Tutor Helps Children (2m16s). On Evening News. Pittsburgh, PA: WTAE. At

Carnegie Mellon Project LISTEN46/29/2004 Within-subject experiment design: 270 students, 180,909 randomized trials Outcome: success = ASR accepts word within first utterance (How) does the type of help affect the next encounter? Randomized choice among feasible types Student clicks “read.” “I love to read stories.” “People sit down and …” “… read a book.” Student is reading a story Student needs help on a word Tutor chooses what help to give Student continues reading Student sees word in a later sentence Time passes…

Carnegie Mellon Project LISTEN56/29/ ,909 word hints (average success rate 66.1%) Whole word:  56,791 Say WordSay Word  24,841 Say In ContextSay In Context Decomposition:  6,280 SyllabifySyllabify  14,223 Onset RimeOnset Rime  19,677 Sound OutSound Out  22,933 One GraphemeOne Grapheme Analogy:  13,165 Rhymes WithRhymes With  13,671 Starts LikeStarts Like Semantic:  14,685 RecueRecue  2,285 Show PictureShow Picture  488 Sound EffectSound Effect Which types stood out?  Best: Rhymes With 69.2% ± 0.4%Rhymes With  Worst: Recue 55.6% ± 0.4%Recue Example: People sit down and read a book.

Carnegie Mellon Project LISTEN66/29/2004 What help did students prefer? Students interrupted 30% of help to get other help. Interruption varied by help type.  Least interrupted (14%): Say WordSay Word  Most interrupted (67%): RecueRecue Success was lower when students interrupted.  If interrupted: 58.5% ± 0.1%  If not: 69.4% ± 0.4%  Difference presumably due to student effects

Carnegie Mellon Project LISTEN76/29/2004 What helped the same day vs. later? Success rates varied with time till next encounter.  67.4% ± 0.1% for 105,424 on same day  64.3% ± 0.2% for 75,485 on later day Time till next encounter varied by help type.  Soonest (72% same day): Sound Effect (frequent words)Sound Effect  Latest (48% same day): Syllabify (rarer words)Syllabify Best help types varied with time till next encounter.  Same day: Say In Context 71.5% ± 0.4%Say In Context  Later day: Rhymes With 67.3% ± 0.6%Rhymes With Caveats:  Same day outcomes include recency effects.  Later day outcomes skew toward harder words.

Carnegie Mellon Project LISTEN86/29/2004 What helped which words best? Same day:Later day: Grade 1 words:Say In ContextSay In Context, Onset Rime Grade 2 words:Rhymes WithRhymes With, Say In Context Say In Context Rhymes With Grade 3 words:Say In ContextOne GraphemeOne Grapheme, Rhymes With Rhymes With Compare within level to control for word difficulty.

Carnegie Mellon Project LISTEN96/29/2004 Conclusions Reading Tutor is a powerful research platform  Hundreds of thousands of randomized trials  Millions of read words  Detected subtle effects despite imperfect ASR Embedded experiments have tricky confounds  Student behavior, recency, word difficulty  So design and analyze with caution!  See Joe Beck’s poster and Type of word help mattered measurably  Recue took longest and helped least Recue  Say In Context helped in the short term Say In Context  Rhymes With had longer lasting benefits Rhymes With

Carnegie Mellon Project LISTEN106/29/2004 The Project LISTEN team  Director:  Jack Mostow  Tutoring:  Dr. Joseph Beck, mining tutorial data  Prof. Albert Corbett, cognitive tutors  Prof. Rollanda O’Connor, reading  Prof. Kathy Ayres, stories for children  Joe Valeri, activities and interventions  Becky Kennedy, linguist  Listening:  Dr. Mosur Ravishankar, recognizer  Dr. Evandro Gouvea, acoustic training  John Helman, transcriber  Programmers:  Andrew Cuneo, application  Karen Wong, Teacher Tool  Field staff:  Dr. Roy Taylor  Kristin Bagwell  Julie Sleasman  Grad students:  Hao Cen, HCI  Cecily Heiner, MCALL  Peter Kant, Education  Shanna Tellerman, ETC  Plus:  Research partners  DePaul  UBC  U. Toronto  Advisory board  Teachers  Children