Educational Data Mining Success Stories Jack Mostow Project LISTEN (www.cs.cmu.edu/~listen) “Home run”: demonstrable increase in learning “Base hit”: likely to improve learning by informing: Educational researchers Teachers Students Tutor developers Automated tutors IERI PI Meeting panel: Data Mining and Analysis Funding: National Science Foundation 1

Predicted a student’s speedup on a word 1. Informing (Reading) Researchers: Microanalysis of repeated vs. wide reading Mostow & Beck (SSSR2005) Predicted a student’s speedup on a word Reduction in word reading time From one (“practice”) encounter of a word To the next (“test”) encounter in a new context By training a linear model based on: What was the student’s reading level? How many letters long was the word? How often had the student seen the word before? Had the student seen the practice sentence before? (More predictor variables …) 2

Results based on Reading Tutor data N = 243,172 speedup opportunities for 352 gr 1-6 students. Speedup averaged 18 ms per encounter (for the first seven). Higher readers sped up less: 3 ms less per grade level. Longer words sped up more: 2 ms more per letter. A new practice sentence helped 27 ms more than an old one. Wide reading beat rereading! 3

2. Informing Teachers: What influences student outcomes in an online course? Scheines et al. (JECR 2005): TETRAD related variables logged in “Causal and Statistical Reasoning” (N = 47 students): pre: pre-test % quiz: average % on quizzes final: % on final exam print: % of modules printed Instrument to estimate effects of voluntary questions so as to infer causality from observation voluntary questions: % attempted Positive effect on performance Online-only, inhibited by printing Telling next-year students helped! (Standardized regression coefficients of variables regressed on parents) 4

3. Informing students: Proactive help to prevent likely mistakes Merceron & Yacef (AIED 2005): Induced prediction rules “if missed X, likely to miss Y” for web-based logic tutor (N = 860 students) Warn students before predicted mistakes occur Warning phrased by teacher = tutor designer 5

Aist (2001 PhD): Explain some new words; later, test all. 4. Informing tutor designers: Does explaining new vocabulary help more than just reading in context? Aist (2001 PhD): Explain some new words; later, test all. Did kids do better on explained vs. unexplained words? Overall: NO; 38%  36%, N = 3,171 trials Rare, 1-sense words tested 1-2 days later: YES! 44% >> 26%, N = 189. 6

5a. Informing tutors: Infer knowledge from behavior Corbett et al. (1995): Knowledge tracing updates at each step the probability that the student has learned the relevant rule. Helps tutor decide what to teach next Predicts error rate with r = 0.85 (0.90 after refining the rules) Average learning curve for 21 cognitive rules 7

5b. Informing tutors: learn what to do Beck et al. (AAAI 2000): learned a teaching policy for a given goal, e.g. “problems average 30 sec” Learned policy cut time per problem by 30% (p<.001) N = 58 students using middle school math tutor in classroom Simulated student (predicts effects of tutor actions) Tutorial agent Data from prior users of tutor Teaching policy Tutor action “try again” Result “correct answer, took 15 sec.” PA uses reinforcement learning “environment” is the PSM (I.e. the student) reward function is based on the teaching goal 8

Summary Educational data mining can inform: Educational researchers Wide reading apparently beat rereading (Mostow & Beck, SSSR2005) Teachers Infer causal effects of observed student choices (Scheines et al., JECR 2005) Students Warn of likely mistakes to prevent them (Merceron & Yacef, AIED 2005) Tutor developers Discover not just whether an intervention works, but when (Aist PhD 2001) Automated tutors Infer student knowledge from student behavior (Corbett et al., 1995) Learn teaching policies that improve outcomes (Beck et al., AAAI 2000) 9