Download presentation
Presentation is loading. Please wait.
1
IERI educational data mining panel
Joseph E. Beck Project LISTEN Center for Automated Learning and Discovery Carnegie Mellon University Funding: National Science Foundation
2
Discussion question “What do data mining tools and methods provide as output, what do they require in terms of input and expertise, which ones seem especially appropriate for educational data mining, and why?”
3
Discussion question “What do data mining tools and methods provide as output, what do they require in terms of input and expertise, which ones seem especially appropriate for educational data mining, and why?”
4
Overview Focusing on input since first step
Providing three big lessons learned Work done in context of Project LISTEN’s Reading Tutor
5
Project LISTEN’s Reading Tutor
6
Why isn’t educational data mining process smoother?
Problems with data collection Not collecting sufficiently detailed data Data are a mess Data are observational
7
Problem: not collecting sufficiently detailed data
Solution: Instrument your software! Record everything you can think of You’ll probably find a use for it later Common to think of research questions after the fact It’s nice to be able to answer them
8
Examples of data to collect
Start/end times for sessions, modules, help, etc. Every item the tutor displays/says Why the tutor did that output What else the tutor could have done Student typed input Student mouse clicks 7,600 hours of fine grained data required only 15 GB of disk space (320 GB disk is only $140)
9
Problem: Data are a mess
You’ve recorded everything you can think of, and wind up with something like 16466, Notice, "Tue Apr 10 12:30: ", , "CListener::FinalizeUtterance", "EndUtterance" 16467, Notice, "Tue Apr 10 12:30: ", , "CCapture::WriteWaveFile(int)", "Wrote File: d:\\listen\\cd\\Tue-Sep \\Capture\\fAT \\dec-fAT Apr wav"
10
Solution: use a database
Database enables tabular representation of data Greatly speeds analyses Removes problem of parsing logfiles
11
Problem: data are observational
Many possible questions Does an intervention work? (What is an intervention?) For whom? In what contexts? Difficult to answer such questions observationally Need experimental trials
12
Solution Design intervention carefully to answer questions about its effectiveness Two properties Assess its own effectiveness Enable causal conclusions “Embedded experiments”
13
Example Have intervention to teach student to pronounce a word
14
Student 3rd grade reading proficiency Intervention How to pronounce a word
15
Select good intervention words
Student 3rd grade reading proficiency Intervention How to pronounce a word Select good intervention words Travelers, Borrowed
16
3rd grade reading proficiency Intervention How to pronounce a word
Student 3rd grade reading proficiency Intervention How to pronounce a word Select good intervention words Travelers, Borrowed Flip coin to decide randomly which word to teach
17
3rd grade reading proficiency Intervention How to pronounce a word
Student 3rd grade reading proficiency Intervention How to pronounce a word Select good intervention words Travelers, Borrowed Flip coin to decide randomly which word to teach Don’t teach “tails” word travelers Teach “heads” word borrowed
18
3rd grade reading proficiency Intervention How to pronounce a word
Student 3rd grade reading proficiency Intervention How to pronounce a word Select good intervention words Travelers, Borrowed Flip coin to decide randomly which word to teach Don’t teach “tails” word travelers Teach “heads” word borrowed Assess both words “Please pronounce ‘travelers’” “Please pronounce ‘borrowed’”
19
3rd grade reading proficiency Intervention How to pronounce a word
Student 3rd grade reading proficiency Intervention How to pronounce a word Select good intervention words Travelers, Borrowed Flip coin to decide randomly which word to teach Don’t teach “tails” word travelers Teach “heads” word borrowed Assess both words “Please pronounce ‘travelers’” “Please pronounce ‘borrowed’” Record details of trial Student, words, success on assessment, type of intervention, etc.
20
3rd grade reading proficiency Intervention How to pronounce a word
Student 3rd grade reading proficiency Intervention How to pronounce a word Select good intervention words Travelers, Borrowed Flip coin to decide randomly which word to teach Don’t teach “tails” word travelers Teach “heads” word borrowed Assess both words “Please pronounce ‘travelers’” “Please pronounce ‘borrowed’” Record details of trial Student, words, success on assessment, type of intervention, etc.
21
3rd grade reading proficiency Intervention How to pronounce a word
Student 3rd grade reading proficiency Intervention How to pronounce a word Select good intervention words Travelers, Borrowed Flip coin to decide randomly which word to teach Don’t teach “tails” word travelers Teach “heads” word borrowed Assess both words “Please pronounce ‘travelers’” “Please pronounce ‘borrowed’” Record details of trial Student, words, success on assessment, type of intervention, etc.
22
(Trimmed) Example of recorded data
User Word Intervention Test performance User3 travelers Pronunciation Correct borrowed None User1 disbelief Spelling delegated Incorrect Can use preferred modeling approach (e.g. logistic regression, decision tree, etc.)
23
Lessons Instrument and record everything Use a database
You can't analyze what isn’t there Use a database Most analysis techniques require tabular format An intervention should be able to assess itself So embed randomized controlled trials in it to allow causal inference
Similar presentations
