1. Educational Data Mining Success Stories
Jack Mostow
Project LISTEN (
“Home run”: demonstrable increase in learning
“Base hit”: likely to improve learning by informing:
Educational researchers
Teachers
Students
Tutor developers
Automated tutors
IERI PI Meeting panel: Data Mining and Analysis
Funding: National Science Foundation 1

2. Predicted a student’s speedup on a word
1. Informing (Reading) Researchers: Microanalysis of repeated vs. wide reading
Mostow & Beck (SSSR2005)
Predicted a student’s speedup on a word
Reduction in word reading time
From one (“practice”) encounter of a word
To the next (“test”) encounter in a new context
By training a linear model based on:
What was the student’s reading level?
How many letters long was the word?
How often had the student seen the word before?
Had the student seen the practice sentence before?
(More predictor variables …) 2

3. Results based on Reading Tutor data
N = 243,172 speedup opportunities for 352 gr 1-6 students.
Speedup averaged 18 ms per encounter (for the first seven).
Higher readers sped up less: 3 ms less per grade level.
Longer words sped up more: 2 ms more per letter.
A new practice sentence helped 27 ms more than an old one.
Wide reading beat rereading! 3

4. 2. Informing Teachers: What influences student outcomes in an online course?
Scheines et al. (JECR 2005): TETRAD related variables logged in “Causal and Statistical Reasoning” (N = 47 students):
pre: pre-test %
quiz: average % on quizzes
final: % on final exam
print: % of modules printed
Instrument to estimate effects of voluntary questions so as to infer causality from observation
voluntary questions: % attempted
Positive effect on performance
Online-only, inhibited by printing
Telling next-year students helped!
(Standardized regression coefficients of variables regressed on parents) 4

5. 3. Informing students: Proactive help to prevent likely mistakes
Merceron & Yacef (AIED 2005):
Induced prediction rules “if missed X, likely to miss Y” for web-based logic tutor (N = 860 students)
Warn students before predicted mistakes occur
Warning phrased by teacher = tutor designer 5

6. Aist (2001 PhD): Explain some new words; later, test all.
4. Informing tutor designers: Does explaining new vocabulary help more than just reading in context?
Aist (2001 PhD): Explain some new words; later, test all.
Did kids do better on explained vs. unexplained words?
Overall: NO; 38%  36%, N = 3,171 trials
Rare, 1-sense words tested 1-2 days later: YES! 44% >> 26%, N = 189. 6

7. 5a. Informing tutors: Infer knowledge from behavior
Corbett et al. (1995): Knowledge tracing updates at each step the probability that the student has learned the relevant rule.
Helps tutor decide what to teach next
Predicts error rate with r = 0.85 (0.90 after refining the rules)
Average learning curve for 21 cognitive rules 7

8. 5b. Informing tutors: learn what to do
Beck et al. (AAAI 2000): learned a teaching policy for a given goal, e.g. “problems average 30 sec”
Learned policy cut time per problem by 30% (p<.001)
N = 58 students using middle school math tutor in classroom
Simulated student (predicts effects of tutor actions)
Tutorial agent
Data from prior users of tutor
Teaching policy
Tutor action
“try again”
Result “correct answer,
took 15 sec.”
PA uses reinforcement learning
“environment” is the PSM (I.e. the student)
reward function is based on the teaching goal 8

9. Summary Educational data mining can inform: Educational researchers
Wide reading apparently beat rereading (Mostow & Beck, SSSR2005)
Teachers
Infer causal effects of observed student choices (Scheines et al., JECR 2005)
Students
Warn of likely mistakes to prevent them (Merceron & Yacef, AIED 2005)
Tutor developers
Discover not just whether an intervention works, but when (Aist PhD 2001)
Automated tutors
Infer student knowledge from student behavior (Corbett et al., 1995)
Learn teaching policies that improve outcomes (Beck et al., AAAI 2000) 9