Data Mining in Education Ryan S. Baker University of Pennsylvania

Over the last decades The explosion in data has transformed field after field

Big Data in Climate Science In climate science, methods for dealing with big data have led to better climate models that give us a few days warning about major storms

Big Data in Physics In high-energy physics, it’s made it possible to analyze the data of 40 million proton collisions per second in looking for evidence of the once-elusive Higgs Boson.

Big Data in Biology And in biology, it’s enabled researchers to sequence at least one version of the more than 3 billion nucleotides in the human genome, and researchers are now conducting individualized gene research building off this work.

Big Data in E-Commerce

Big Data in Education The moment is arriving when we can obtain and utilize similar amounts of data in education

Interactive Learning Environments As more learning takes place within educational software and online learning environments of various types, it becomes much easier to gather very rich data on individual students’ learning and engagement within specific subjects. For example, a student might use a science simulation like the Inq-ITS, to learn science content and inquiry skills. Or they might learn scientific inquiry skill and content within a virtual environment like EcoMUVE. They might learn math skill in an action game like Zombie Division – the student has a set of weapons with numbers associated with them, a 2 for a sword, or a 5 for a gauntlet, and they can divide a skeleton if the weapon divides the number on the skeleton’s chest. Or they might learn math in a conceptual story-based learning environment like Reasoning Mind… or by doing math problems in a workbook-like environment like ASSISTments. All of these environments generate rich data streams that have been used in EDM analyses. And this kind of software in becoming more widespread every day. Systems like the Cognitive Tutor, or ASSISTments, or Reasoning Mind, are used by tens or hundreds of thousands of students, one or two days a week.

MOOCs and online courses

Student Log Data *000:22:297 READY . *000:25:875 APPLY-ACTION WINDOW; LISP-TRANSLATOR::AUTHORINGTOOL-TRANSLATOR, CONTEXT; 3FACTOR-CROSS-XPL-4, SELECTIONS; (GROUP3_CLASS_UNDER_XPL), ACTION; UPDATECOMBOBOX, INPUT; "Two crossover events are very rare.", *000:25:890 GOOD-PATH *000:25:890 HISTORY P-1; (COMBOBOX-XPL-TRACE SIMBIOSYS), *000:25:890 READY *000:29:281 APPLY-ACTION SELECTIONS; (GROUP4_CLASS_UNDER_XPL), INPUT; "The largest group is parental since crossovers are uncommon.", *000:29:281 GOOD-PATH *000:29:281 HISTORY *000:29:281 READY *001:20:733 APPLY-ACTION SELECTIONS; (ORDER_GENES_OBS_XPL), INPUT; "The Q and q alleles have interchanged between the parental and SCO genotypes.", *001:20:733 SWITCHED-TO-EDITOR *001:20:748 NO-CONFLICT-SET *001:20:748 READY *001:32:498 APPLY-ACTION INPUT; "The Q and q alleles have interchanged between the parental and DCO genotypes.", *001:32:498 GOOD-PATH *001:32:498 HISTORY *001:32:498 READY *001:37:857 APPLY-ACTION SELECTIONS; (ORDER_GENES_UNDER_XPL), INPUT; "In the DCO group BOTH outer genes cross over so the interchanged gene is the middle one.", *001:37:857 GOOD-PATH For example, as a student uses one of these interactive learning environments, the student will make hundreds of meaningful actions each hour – pausing and thinking before making an incorrect answer, asking for help, rapidly changing settings on a simulation, running away from a skeleton. When the data is logged, these behaviors provide us with incredibly rich detail about learning and engagement, that we can analyze.

Grade and Outcome Data

Student Engagement Data

Large-scale data Log data from systems used by hundreds of thousands of students per year ALEKS, Cognitive Tutor, Reasoning Mind Whole-university-system data on course-taking and outcomes

Data in Education Used to Be Dispersed Hard to Collect Small-Scale

Data Today

Data Today

PSLC DataShop (Koedinger et al, 2008, 2010) >250,000 hours of students using educational software within LearnLabs and other settings >30 million student actions, responses & annotations

“the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” To quote the Society for Learning Analytics Research… EDM and learning analytics methods have some similarities with traditional data mining methods, but as with the other areas where data mining methods have been common: bioinformatics, medical informatics, business analytics, data analysis methods in physics, and so on, the unique features of the domain of education leads to the development of unique methods. (www.solaresearch.org/mission/about)

Goals Joint goal of exploring the “big data” now available on learners and learning To promote New scientific discoveries & to advance science of learning Better assessment of learners along multiple dimensions Social, cognitive, emotional, meta-cognitive, etc. Individual, group, institutional, etc. Better real-time support for learners

Many types of EDM/LA Method (Baker & Siemens, 2014; building off of Baker & Yacef, 2009) Prediction Structure Discovery Relationship mining Distillation of data for human judgment Discovery with models

Prediction Develop a model which can infer a single aspect of the data (predicted variable) from some combination of other aspects of the data (predictor variables) Which students are bored? Which students will fail the class? Infer something that matters, so we can do something about it

Structure Discovery Find structure and patterns in the data that emerge “naturally” No specific target or predictor variable What problems map to the same skills? Are there groups of students who approach the same curriculum differently? Which students develop more social relationships in MOOCs?

Relationship Mining Discover relationships between variables in a data set with many variables Are there trajectories through a curriculum that are more or less effective? Which aspects of the design of educational software have implications for student engagement?

Many applications Dropout/success prediction Automated detection of learning, engagement, emotion, strategy, for better individualization Better reporting for teachers and other stakeholders Basic discovery in education

Individualization requires Determining something about the student Knowing what matters Doing the right thing about it

Determining something about the student Knowing what matters Doing the right thing about it

Stuff We Can Infer: Complex Learning Is the student learning to solve complex problems that require inquiry? (Sao Pedro et al., 2013; Baker & Clarke-Midura, 2013) Is the student developing rich conceptual understanding in domains such as physics? (Shute & Ventura, 2013; Rowe et al., 2015)

Stuff We Can Infer: Robust Learning Will the student remember what they learned? (Jastrzembski et al., 2006; Pavlik et al., 2008; Wang & Beck, 2012) Is the student prepared for future learning? (Baker et al., 2011; Hershkovitz et al., 2013)

Stuff We Can Infer: Meta-Cognition How confident is the student? (Litman et al., 2006; McQuiggan, Mott, & Lester, 2008; Arroyo et al., 2009) Is the student asking for help when they need it? (Aleven et al., 2004, 2006) Is the student persisting in the face of challenge? (Ventura et al., 2012)

Stuff We Can Infer: Disengaged Behaviors Gaming the System (Baker et al., 2004, 2008, 2010; Walonoski & Heffernan, 2006; Beal, Qu, & Lee, 2007) Carelessness (San Pedro et al., 2011; Hershkovitz et al., 2011) Inexplicable Behavior (Rowe et al., 2009; Wixon et al., 2012)

Stuff We Can Infer: Affect (Emotion in Context) Boredom Frustration Confusion Engaged Concentration/Flow Curiosity Excitement Situational Interest Joy/Delight (D’Mello et al., 2008; Mavrikis, 2008; Arroyo et al., 2009; Conati & Maclaren, 2009; Lee et al., 2011; Sabourin et al., 2011; Baker et al., 2012, 2014; Paquette et al., 2014, 2015; Pardos et al., 2014; Kai et al., 2015)

No physical sensors needed Now feasible to infer these constructs solely from student interaction with the learning system

Example Automated detectors of student engagement and affect in ASSISTments (Pardos et al., 2013; Ocumpaugh et al., 2014)

Process

Field Observations of Student Engagement and Affect Using BROMP observation protocol (Ocumpaugh et al., 2015) >150 coders certified in USA, Philippines, India, UK Synchronized to log files with Android app HART

Use data mining to find behaviors that co-occur with human observations Distill features of interaction hypothesized to correlate to desired construct

Use data mining to find behaviors that co-occur with human observations Distill features of interaction hypothesized to correlate to desired construct Best to use theoretical understanding and automated discovery together (Sao Pedro et al., 2012; Paquette et al., 2015)

Use data mining to find behaviors that co-occur with human observations Try a small set of data mining/prediction/classification algorithms that fit different kinds of patterns Decision Trees Decision Rules Step Regression Naïve Bayes K*

Test model generalizability on new students and new populations In this case, students in rural, urban, and suburban schools in Northeastern USA Diverse in terms of SES, race, ethnicity

Model Goodness (Pardos et al., 2013) Construct Algo A’ Kappa Boredom JRip 0.632 0.229 Frustration Naïve Bayes 0.681 0.301 Engaged Concentration K* 0.678 0.358 Confusion J48 0.736 0.274 Off-Task REPTree 0.819 0.506 Gaming 0.802 0.370 The affect detectors’ predictive performance were evaluated using A' [28] and Cohen’s Kappa [18]. An A' value (which is approximately the same as the area under the ROC curve [28]) of 0.5 for a model indicates chance-level performance for correctly determining the presence or absence of an affective state in a clip, and 1.0 performing perfectly. Cohen’s Kappa assesses the degree to which the model is better than chance at identifying the affective state in a clip. A Kappa of 0 indicates chance-level performance, while a Kappa of 1 indicates perfect performance. A Kappa of 0.45 is equivalent to a detector that is 45% better than chance at identifying affect. As discussed in [37], all of the affect and behavior detectors performed better than chance. Detector goodness was somewhat lower than had been previously seen for Cognitive Tutor Algebra [cf. 6], but better than had been seen in other published models inferring student affect in an intelligent tutoring system solely from log files (where average Kappa ranged from below zero to 0.19 when fully stringent validation was used) [19, 22, 44]. The best detector of engaged concentration involved the K* algorithm, achieving an A' of 0.678 and a Kappa of 0.358. The best boredom detector was found using the JRip algorithm, achieving an A' of 0.632 and a Kappa of 0.229. The best confusion detector used the J48 algorithm, having an A’ of 0.736, a Kappa of 0.274. The best detector of off-task behavior was found using the REP-Tree algorithm, with an A’ value of 0.819, a Kappa of 0.506. The best gaming detector involved the K* algorithm, having an A’ value of 0.802, a Kappa of 0.370. These levels of detector goodness indicate models that are clearly informative, though there is still considerable room for improvement. The detectors emerging from the data mining process had some systematic error in prediction due to the use of re-sampling in the training sets (models were validated on the original, non-resampled data), where the average confidence of the resultant models was systematically higher or lower than the proportion of the affective states in the original data set. This type of bias does not affect correlation to other variables since relative order of predictions is unaffected, but it can reduce model interpretability. To increase model interpretability, model confidences were rescaled to have the same mean as the original distribution, using linear interpolation. Rescaling the confidences this way does not impact model goodness, as it does not change the relative ordering of model assessments. Application of Affect and Behavior Models to Broader Data Set Once the detectors of student affect and behavior were developed, they were applied to the data set used in this paper. As mentioned, this data set was comprised of 2,107,108 actions in 494,150 problems completed by 3,747 students in three school districts. The result was a sequence of predictions of student affect and behavior across the history of each student’s use of the ASSISTment system.

Result Models can make inference in real-time (20 second delay) Models can be applied at scale to retrospective log files

Determining something about the student Knowing what matters Doing the right thing about it

Example Take automated detectors of engagement, affect, and learning in ASSISTments Applied to several years of entire-year student data Thousands of students Millions of actions within the software

Engagement and Standardized Exam Score (Pardos et al., 2013, 2014) Detectors applied to whole year of data for 1,393 students Gaming the system (r = -0.36) Boredom (r = -0.2) Engaged concentration (r = +0.36) We then evaluate the relationship of these measures to student outcomes, such as state exam scores. Applying applyint the assistments detectors to a whole year of data for aroun 1400 students, We found out end-of year exam scores to be negatively correlated to boredom and gaming the system, and positively correlated to engaged concentration

College Attendance (San Pedro, Baker, Bowers, & Heffernan, 2013) The detectors can predict Whether a student will go to college or not, ~6 years later 69% of the time for new students

College Attendance (San Pedro, Baker, Bowers, & Heffernan, 2013) And the model can indicate what aspects of a student’s behavior are predictive of college attendance Alex is less likely to go to college Top predictive factors: he is getting confused and gaming the system… Maria is less likely to go to college Top predictive factors: she is getting bored and careless…

College Major (San Pedro et al., 2014, 2015) And these same constructs can also predict what a student will major in when they get to college

Another Example Student interaction within a MOOC in data science can predict whether the student will eventually submit a scientific paper in the field (Wang et al., under review) Forum lurkers are more likely to submit a scientific paper than forum posters!

Determining something about the student Knowing what matters Doing the right thing about it

What do we do? When we know that a student is bored… or gaming the system… or has shallow learning… or etc. etc. etc.

Huge Space of Potential Interventions

Huge Space of Potential Interventions Automated interventions delivered by animated agents

Huge Space of Potential Interventions Stealth interventions that change learner experience in subtle ways

Huge Space of Potential Interventions Reports to instructors, guidance counselors, parents, students themselves…

Examples of Use (at scale) ALEKS – Models of prerequisite structure and knowledge used to select material for students High school and college math and science

Examples of Use (at scale) Reasoning Mind – automated detectors of engagement used to provide analytics to regional coordinators about teacher effectiveness Elementary school math

Examples of Use (not at scale) Project LISTEN – data mining used to determine which strategies work for which students, and to help select which stories to give students to read Elementary school reading

Examples of Use (at scale) ASSISTments – automated detectors of engagement and knowledge used to determine how to re-design learning experiences to increase effectiveness Middle school math

Examples of Use (not yet at scale) ASSISTments – automated detectors of engagement and knowledge used to make predictions about student outcomes and provide reports to school guidance counselors Middle school math

Examples of Use (at scale) Course Signals, Zogotech, Soomo – At-risk prediction models used to provide actionable information to instructors and academic advisors College

Huge Space of Potential Interventions Still an open area for the field And an area of considerable ongoing research for my lab

The Big Idea Thanks to the big data now becoming available on student learning

The Big Idea Thanks to the big data now becoming available on student learning And a combination of data mining and knowledge engineering

The Big Idea Thanks to the big data now becoming available on student learning And a combination of data mining and knowledge engineering We can make inferences about students in real-time

The Big Idea Thanks to the big data now becoming available on student learning And a combination of data mining and knowledge engineering We can make inferences about students in real-time That are predictive of long-term outcomes

Eventual Goal Track a student’s engagement now

Eventual Goal Track a student’s engagement now Predict the longer-term impact

Eventual Goal Track a student’s engagement now Predict the longer-term impact Intervene to help re-engage students and support their learning

Eventual Goal Track a student’s engagement now Predict the longer-term impact Intervene to help re-engage students and support their learning Helping to create an educational system more sensitive to individual learners’ needs

Get Involved Lots of opportunities to learn more about this emerging field, right here in the Boston area

Get Involved MHE’s data science group right here in Boston has some amazing leaders in EDM Excellent groups at MIT, HarvardX as well Learning Analytics Summer Institute Local Meetings often held in Boston, though none scheduled for this year ACM L@S 2017 will be held in Cambridge

Get Involved Edm-announce mailing list learninganalytics@googlegroups.com

See our free online MOOT “Big Data and Education” Learn More twitter.com/BakerEDMLab Baker EDM Lab weibo.com/u/5370802148 Baker EDM Lab See our free online MOOT “Big Data and Education” Offered as EdX MOOC, next iteration in a few months All lab publications available online – Google “Ryan Baker”