Educational Data Mining: A Methodological Review

Educational Data Mining: A Methodological Review
Ryan S.J.d. Baker Julius and Rosa Sachs Distinguished Lecturer Teachers College, Columbia University President, International Educational Data Mining Society

Educational Data Mining
A growing research area with important overlaps with Learning Analytics Though the two research communities have so far largely grown in parallel

A Definition (Baker, 2010) “The area of scientific inquiry centered around the development of methods for exploring the unique kinds of data that come from educational settings, and using those methods to better understand students and the settings which they learn in.” Drawing heavily from the methods of data mining (cf. Witten & Frank, 2005) But adapting those methods to the unique aspects of educational data

EDM is… “… escalating the speed of research on many problems in education.” “Not only can you look at unique learning trajectories of individuals, but the sophistication of the models of learning goes up enormously.” Arthur Graesser, Editor, Journal of Educational Psychology

EDM Institutions EDM Conference since 2008 Journal of EDM since 2009
Before that, EDM Workshops since 2005 Journal of EDM since 2009 International EDM Society since 2011

EDM and LAK With the emergence of the EDM Society and SoLAR, there is an opportunity to articulate two visions of the future of how educational data will be explored The two visions have considerable similarities, but differences that matter too I believe we should cooperate, collaborate, and share But should also attempt to articulate differences in values and methods, in a spirit of friendly competition

Key Similarities (drawn in part from Siemens & Baker, 2012)
Data-intensive approaches to studying learning and learners Goal of improving education and basic research in education Can drive planning, decision-making, interventions

Key Differences (drawn in part from Siemens & Baker, 2012)
LAK Leveraging human judgment with automated discovery EDM Automated discovery that leverages human judgment

LAK Stronger emphasis on understanding systems as wholes, in their full complexity EDM Stronger emphasis on reducing to components and analyzing individual components and relationships between them

LAK Greater focus on informing and empowering instructors and learners EDM Greater focus on automated adaptation (e.g. by the computer with no human in the loop)

LAK Greater focus on addressing needs of multiple stakeholders with information drawn from data EDM Greater focus on issues of model generalizability across settings and populations

Types of EDM method (Baker & Yacef, 2009; updated in Baker & Siemens, under review)
Prediction Classification Regression Latent Knowledge Estimation Structure Discovery Clustering Factor Analysis Domain Structure Discovery Network Analysis Relationship mining Association rule mining Correlation mining Sequential pattern mining Causal data mining Distillation of data for human judgment Discovery with models

Types of EDM method Many of these overlap with traditional data mining methods Some new or special cases that are particularly prominent in EDM Latent Knowledge Estimation Domain Structure Discovery Discovery with Models

One EDM vision In illustrating how these methods can be useful, I’d like to share some thoughts on my vision of how EDM can contribute to education It may be interesting afterwards to discuss how this vision is similar to and different from various perspectives on how learning analytics can contribute

Predict the Future Ryan Shaun Joazeiro de Baker In my view, one of the coolest things about EDM and LAK is that we can now – to some degree -- Predict the Future

Predict the Future Change the Future Ryan Shaun Joazeiro de Baker And as such, we can Change the Future

Educational Software As more learning takes place within educational software and online learning environments of various types, it becomes much easier to gather very rich data on individual students’ learning and engagement within specific subjects. For example, a student might use a science simulation like the Inq-ITS, to learn science content and inquiry skills. Or they might learn scientific inquiry skill and content within a virtual environment like EcoMUVE. They might learn math skill in an action game like Zombie Division – the student has a set of weapons with numbers associated with them, a 2 for a sword, or a 5 for a gauntlet, and they can divide a skeleton if the weapon divides the number on the skeleton’s chest. Or they might learn math in a conceptual story-based learning environment like Reasoning Mind… or by doing math problems in a workbook-like environment like ASSISTments. All of these environments generate rich data streams that have been used in EDM analyses. And this kind of software in becoming more widespread every day. Systems like the Cognitive Tutor, or ASSISTments, or Reasoning Mind, are used by tens or hundreds of thousands of students, one or two days a week.

Student Log Data *000:22:297 READY . *000:25:875 APPLY-ACTION WINDOW; LISP-TRANSLATOR::AUTHORINGTOOL-TRANSLATOR, CONTEXT; 3FACTOR-CROSS-XPL-4, SELECTIONS; (GROUP3_CLASS_UNDER_XPL), ACTION; UPDATECOMBOBOX, INPUT; "Two crossover events are very rare.", *000:25:890 GOOD-PATH *000:25:890 HISTORY P-1; (COMBOBOX-XPL-TRACE SIMBIOSYS), *000:25:890 READY *000:29:281 APPLY-ACTION SELECTIONS; (GROUP4_CLASS_UNDER_XPL), INPUT; "The largest group is parental since crossovers are uncommon.", *000:29:281 GOOD-PATH *000:29:281 HISTORY *000:29:281 READY *001:20:733 APPLY-ACTION SELECTIONS; (ORDER_GENES_OBS_XPL), INPUT; "The Q and q alleles have interchanged between the parental and SCO genotypes.", *001:20:733 SWITCHED-TO-EDITOR *001:20:748 NO-CONFLICT-SET *001:20:748 READY *001:32:498 APPLY-ACTION INPUT; "The Q and q alleles have interchanged between the parental and DCO genotypes.", *001:32:498 GOOD-PATH *001:32:498 HISTORY *001:32:498 READY *001:37:857 APPLY-ACTION SELECTIONS; (ORDER_GENES_UNDER_XPL), INPUT; "In the DCO group BOTH outer genes cross over so the interchanged gene is the middle one.", *001:37:857 GOOD-PATH For example, as a student uses one of these interactive learning environments, the student will make hundreds of meaningful actions each hour – pausing and thinking before making an incorrect answer, asking for help, rapidly changing settings on a simulation, running away from a skeleton. When the data is logged, these behaviors provide us with incredibly rich detail about learning and engagement, that we can analyze.

While much of this data is still locked within specific companies or research labs, an increasing amount is becoming openly available for research. For example, today, the Pittsburgh Science of Learning Center DataShop, led by Ken Koedinger and John Stamper, hosts data from hundreds of thousands of student-hours of learning that any researcher in the world can download and study. (Koedinger et al., 2008, 2010)

We can predict a lot of things from that data

Stuff We Can Infer: Student Knowledge
Two decades now of research to infer student knowledge as it changes during learning… (Corbett & Anderson, 1995; Martin & VanLehn, 1995; Shute, 1995; Pavlik et al., 2009; Pardos et al., 2012)

Stuff We Can Infer: Disengaged Behaviors
Gaming the System (Baker et al., 2004, 2008, 2010; Walonoski & Heffernan, 2006; Beal, Qu, & Lee, 2007) Off-Task Behavior (Baker, 2007; Cetintas et al., 2010) Carelessness (San Pedro et al., 2011; Hershkovitz et al., 2011) WTF Behavior (Rowe et al., 2009; Wixon et al., UMAP2012)

Stuff We Can Infer: Meta-Cognition
Self-Efficacy/Uncertainty/Confidence (Litman et al., 2006; McQuiggan, Mott, & Lester, 2008; Arroyo et al., 2009) Unscaffolded Self-Explanation (Shih et al., 2008; Baker, Gowda, & Corbett, 2011) Help Avoidance (Aleven et al., 2004, 2006)

Stuff We Can Infer: Affect
Boredom (D’Mello et al., 2008; Sabourin et al., 2011; Baker et al., 2012) Frustration (McQuiggan et al., 2007; D’Mello et al., 2008; Sabourin et al., 2011; Baker et al., 2012) Confusion (D’Mello et al., 2008; Lee et al., 2011; Sabourin et al., 2011; Baker et al., 2012) Engaged Concentration/Flow (D’Mello et al., 2008; Sabourin et al., 2011; Baker et al., 2012)

So these days we can assess a lot of things
So these days we can assess a lot of things. But this kind of work is just the tip of the iceberg, in comparison to what we’re going to be able to do in the near future. Because assessing where a student is right now, what they know, what they’re doing, what they’re feeling, that’s useful, but it’s not enough.

Not prepared for career
Fail course Not prepared for career Career success We also need to know where the student is going. We need to be able to answer questions like: Is this student going to pass or fail the course? Is the student going to be prepared for the next course? Is the student going to be prepared to learn new skills when he or she enters the workforce, and succeed in his or her career? And are motivational factors going to get in the way of the student’s success? Is the student going to think “career X isn’t for me”, or “I can’t succeed in career X”, and therefore make that perception a reality? To put it another way, when we talk about the pipeline of students becoming ready for 21st century careers, where is the student most likely to drop out of the pipeline? Not prepared for next course Motivational factors

Recent work Has been able to predict
EDM and learning analytics methods are proving to be useful for this. For example, recent work has been able to predict

Which students are prepared for future learning
Which students are prepared for future learning, more likely to do well when they get to the next topic in class. You can see here that the red and blue student are doing about equally well on the first topic, but then diverge on the second topic. We can now predict that this divergence is going occur, early within topic 1, for some educational software. The model that predicts this builds off of models of student behaviors within the software, including whether students gamed the system, and how they used the software’s help features. Topic 1 Topic 2 [Baker, Gowda, & Corbett, 2011; Hershkovitz et al., in preparation]

Which students are more likely to fail the class
B It can also predict which students are more likely to fail the class; both the blue and red students pass the class, but the green student is behind from the beginning and never catches up. Researchers have developed models that are able to predict failure in online classes from very early in the semester. F [Arnold, 2010; Ming & Ming, in press]

Which students are more likely to drop out of school or university
And EDM and Learning Analytics methods can even predict which students are more likely to drop out of school or university. The Purdue Signals Project has been successful at identifying this early in a student’s university sequence. [Dekker et al., 2009; Kovacic, 2011; Marquez-Vera et al., 2012]

So with educational data mining,

Predict the Future We can predict the future!

But is it enough? But is it enough to just predict the future?
What if we predict the future, and then we don’t do anything about it? The reason to predict the future is…

Predict the Future To change the future! Change the Future

Ways to Change the Future
Reports to students, instructors, parents Automated adaptation

NO EDM WITH EDM Misconception Misconception Self-Explain Self-Explain
YES NO YES No Learning Learning Intervention Learning To give one example, consider the graph on the left; take a student who has a misconception, and who gets some feedback on that misconception from the learning software they’re using. If they think through and self-explain that feedback, they’ll learn, there’s a lot of research showing that, but what if they don’t self-explain? Then they won’t learn. But if an EDM-based model can catch that they don’t self-explain, then we can intervene, so that they do learn. Learning NO EDM WITH EDM

NO EDM EDM + TEACHER Misconception Misconception Self-Explain
YES NO YES No Learning Learning Teacher Intervention Learning Sometimes it should be the software nudging the student. Other times it should be a teacher, or a parent, or even a friend. We don’t really know as a field when different types of support, and different supporters, are most effective. We’ll need to leverage everyone we can, working to include and empower everyone who can help the student learn. Learning NO EDM EDM + TEACHER

Automatically Responding to Gaming the System
I’d like to give an example of how we can use automated detectors to do this. It comes from my group’s research on gaming the system in middle school mathematics. As I mentioned a few minutes ago, students who game try to get the answer without learning the material, by systematically guessing or clicking through hints to get the answer, and then moving on without thinking about the answer. Across several studies, we’ve seen that students who game the system typically start out behind other students, and fall further behind by the post-test.

(Baker et al., 2006; Rodrigo et al., 2011)
We developed an automated detector that could identify when a student was gaming, and built it into a cartoon character, Scooter the Tutor. Scooter responded to gaming in two ways. (Baker et al., 2006; Rodrigo et al., 2011) Graphics adapted from Microsoft Office Assistant

First, emotional expressions. If the student never games, Scooter dances around and looks happy. (Baker et al., 2006; Rodrigo et al., 2011) Graphics adapted from Microsoft Office Assistant

But if the student appears to be gaming, Scooter looks increasingly upset and becomes redder and redder. <pause> By the way, the boys loved seeing Scooter get really mad. They’d all get Scooter happy, and then they’d start at the same time and sit there and compete to see who could get Scooter mad first, when the teacher wasn’t looking. Not quite what we planned. (Baker et al., 2006; Rodrigo et al., 2011) Graphics adapted from Microsoft Office Assistant

Scooter also gave supplementary exercises that covered the material the student had bypassed through gaming, giving students another way to learn the material they’d missed. (Baker et al., 2006; Rodrigo et al., 2011) Graphics adapted from Microsoft Office Assistant

The result was that about half as many students gamed the system, 17% when Scooter was present, as compared to 33% in the same tutor lesson without Scooter.

EXPERIMENTAL CONTROL Students who got Scooter’s interventions actually caught up to the rest of the class. You can see here that students who didn’t game, and didn’t receive exercises from Scooter – the pink line – started out way ahead of the students who gamed and got exercises from Scooter. But by the post-test, the gaming students had caught up to the rest of the class.

EXPERIMENTAL CONTROL Usually, students who game the system start behind the rest of the class and fall further behind by the end of the lesson, which is exactly what we saw in the control condition of that study, when we look at the students who gamed according to the detector, and would have gotten exercises from Scooter. If you compare those two blue lines, you can see what a difference an intervention based on an EDM detector can produce.

Just one example This is just one example of how predicting the future can change the future.

Just one example Predict the Future
Predict the future: we detect that a specific student is gaming the system, and they’ll learn less than other students

Just one example Predict the Future Change the Future
Change the future: we intervene right then, to give them a different way to learn the material, and they catch up to the rest of the class. Change the Future

Predict the Future With educational data mining, we’ll be able to go beyond assessing where a student is now, to predicting the future. We can determine where a student is disengaged, where the student is struggling. And then we can use that information to change the future, helping students to engage and to develop deep and robust learning. Models developed through educational data mining can help to support students better, and help teachers and school administrators to support students better. Change the Future

Fail course Not prepared for career Career success With EDM and Learning Analytics, we can cut off the arrows that lead the student away from career success, or at least weaken them. We can catch when a student is at risk of failing a class, provide them with better and more targeted learning support, and equip their teachers with useful information to help these students catch up. Not prepared for next course Motivational factors

Fail course Not prepared for career Career success <pause> We can catch when a student’s learning is shallow, and they’ll succeed now but not later. And then we can help them learn more deeply, so that they don’t just succeed in the current course but learn what’s necessary to prepare them for their future learning. Not prepared for next course Motivational factors

Fail course Not prepared for career Career success <pause> Eventually, we can infer what trajectories through a course sequence and what learning experiences indicate that a student is or isn’t prepared for their future career. And then we can alter the curriculum to improve the support for those students. Not prepared for next course Motivational factors

Fail course Not prepared for career Career success <pause> And finally, we can figure out when motivational and attitudinal factors are likely to hold a student back and we can provide that information to teachers and guidance counselors and parents – not to mention using that information to help select enrichment experiences for the student, to engage and excite them about their future opportunities. Not prepared for next course Motivational factors

Fail course Not prepared for career Career success <pause> And there you have it. At each stage of a student’s progression to eventual success, there may be something that EDM, Learning Analytics, and interventions based on those methods can do to help. There’s a lot of work that’ll be necessary to get there. We need to get the data, we need to use that data effectively, and then we need thoughtful design, in partnership with teachers and other relevant folks, to develop appropriate and effective interventions that leverage those detectors. There’s a lot to do, for EDM to be of full use to the education research and practice communities, and for EDM to achieve its potential for enabling new and more effective ways of supporting learners, teachers, school administrators, and parents. Not prepared for next course Motivational factors

Predict the Future EDM can help us predict the future. It’s up to all of us to change it. Change the Future

Resources I’ll open the floor to general questions and discussion in a minute But first, I want to recommend some resources to everyone

Resources on EDM Journal of Educational Data Mining
We welcome submissions from LAK community members! Handbook of Educational Data Mining Proceedings of EDM

Resources on EDM Baker, R.S.J.d., Yacef, K. (2009) The State of Educational Data Mining in 2009: A Review and Future Visions. Journal of Educational Data Mining, 1 (1), 3-17 Baker, R.S.J.d. (2010) Data Mining for Education. In McGaw, B., Peterson, P., Baker, E. (Eds.) International Encyclopedia of Education (3rd edition), vol. 7, pp Oxford, UK: Elsevier.

Resources on EDM Romero, C., Ventura, S. (2010) Educational Data Mining: A Review of the State-of-the-Art. IEEE Transaction on Systems, Man, and Cybernetics, Part C: Applications and Reviews. 6, Romero, C., Ventura, S. (2007) Educational Data Mining: A Survey from 1995 to Expert Systems with Applications, 33(1),

Resources on data mining in general
Statistical Data Mining Tutorials Andrew Moore, Google

MOOC on EDM Methods “Big Data in Education”
Available through Coursera platform on August 5, 2013 Not yet on the web But me if you want to be notified when the course is available for registration Complementary to what you’re taking now; focused on learning to apply algorithms and methods

The End Thanks! Questions? Comments?

Educational Data Mining: A Methodological Review

Similar presentations

Presentation on theme: "Educational Data Mining: A Methodological Review"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Educational Data Mining: A Methodological Review

Similar presentations

Presentation on theme: "Educational Data Mining: A Methodological Review"— Presentation transcript:

Similar presentations

About project

Feedback