1. Experimenter-defined measures in a Reading Tutor that Listens
Jack Mostow
Project LISTEN (
Carnegie Mellon University
Funding: National Science Foundation
IERI PI Meeting, Sept. 9-10, 2004 1

2. Thanks to fellow LISTENers
Tutoring:
Dr. Joseph Beck, mining tutorial data
Prof. Albert Corbett, cognitive tutors
Prof. Rollanda O’Connor, reading
Prof. Kathy Ayres, stories for children
Joe Valeri, activities and interventions
Becky Kennedy, linguist
Listening:
Dr. Mosur Ravishankar, recognizer
Dr. Evandro Gouvea, acoustic training
John Helman, transcriber
Programmers:
Andrew Cuneo, application
Karen Wong, Teacher Tool
Field staff:
Dr. Roy Taylor
Kristin Bagwell
Julie Sleasman
Grad students:
Hao Cen, HCI
Cecily Heiner, MCALL
Peter Kant, Education
Shanna Tellerman, ETC
Plus:
Advisory board
Research partners
DePaul
UBC
U. Toronto
Schools 2

3. Uses of experimenter-defined measures
Analyze usage
Assess students
Evaluate interventions 3

4. 1. Analyzing usage Participation Frequency Duration Hiatus rate
What % of enrolled students use the Reading Tutor?
Frequency
How often do they use it?
Duration
For how long at a time?
Hiatus rate
With how many timeouts?
Crash rate
With what % ending in crashes? 4

5. 2002-2003 usage: lab > classroom -- but top classrooms > average lab5

6. 2. Assessing students: Latency decreases over time
Initial encounter of muttered:
I’ll have to mop up all this (5630) muttered Dennis to himself but how
5 weeks later:
Dennis (110) muttered oh I forgot to ask him for the money
How does latency evolve in general? 6

7. Help and latency predict WI scores with correlation > 0.9.
Learning Curves for help and latency (for 2.4 million word encounters in ’00-01)
Help request rate
Word latency (in ms)
# of previous encounters of the word
Help and latency predict WI scores with correlation > 0.9. 7

8. Time as an indicator of motivation
% time picking stories predicts lower gains (R = –0.4)
Mostow, J., Aist, G., Beck, J., Chalasani, R., Cuneo, A., Jia, P., & Kadaru, K. (2002, June 5-7). A La Recherche du Temps Perdu, or As Time Goes By: Where does the time go in a Reading Tutor that listens? Proceedings of the Sixth International Conference on Intelligent Tutoring Systems (ITS'2002), Biarritz, France,
Hasty responses indicate guessing
Beck, J. E. (2004, August 31). Using response times to model student disengagement. Proceedings of the ITS2004 Workshop on Social and Emotional Intelligence in Learning Environments, Maceio, Brazil. 8

9. 3. Evaluating interventions: vocabulary [Aist PhD thesis]
Randomly explain some new words but not others.
Test each new word the next day.
Did kids do better on explained vs. unexplained words?
Overall: NO; 38%  36%, N = 3,171 trials
Rare, 1-sense words tested 1-2 days later: YES! 44% >> 26%, N = 189. 9

10. Evaluating interventions: word help [SSSR04]
Student is reading a story
Student needs help on a word
Tutor chooses what help to give
Student continues reading
Student sees word in a later sentence
Time passes…
‘People sit down and …’
Student clicks ‘read.’
Decision (randomized)
‘… read a book.’
Explain outcome = word read OK, then explain segmentation artifact; read ASR as “automatic speech recognizer”
‘I love to read stories.’
Outcome: read fluently?
How does outcome vary by help, word, and delay? 10

11. What helped which words best, when
What helped which words best, when? (for 270 students, 180,909 randomized trials)
Best overall: Rhymes With 69.2% ± 0.4%
Worst overall: Recue 55.6% ± 0.4%
Compare within level to control for word difficulty.
Supplying the word helped best in the short term…
But rhyming hints had longer lasting benefits.
Same day:
Later day:
Grade 1 words:
Say In Context,
Onset Rime
Grade 2 words:
Say In Context, Rhymes With
Rhymes With
Grade 3 words:
Say In Context
Rhymes With, One Grapheme 11

12. Project LISTEN’s Reading Tutor: A rich source of experimental data
database:
9 schools
> 200 computers
> 50,000 sessions
> 1.5M tutor responses
> 10M words recognized
Embedded experiments
Randomized trials
See videos, papers, etc. at
Thank you! Questions?
Get hours from Joe 12

13. Project LISTEN’s Reading Tutor (video)
John Rubin (2002). The Sounds of Speech (Show 3). On Reading Rockets (Public Television series commissioned by U.S. Department of Education). Washington, DC: WETA.
Available at 13

14. Map data stream to data set: trials
Context: Decision: Outcome: 14