1. 1 Cognitive Analysis of Student Learning Using LearnLab Brett van de Sande, Kurt VanLehn, & Tim Nokes University of Pittsburgh

2. 2 Agenda I.LearnLab methodology II.Demonstration of Andes, an intelligent homework tutor III.Log File Analysis

3. 3 Goal: To understand physics learning Policy level –e.g., Physics for high school freshman? Instructional level –e.g., How much assistance to give? –e.g., How much practice per topic? –e.g., How to handle errors? Neurocognitive level –e.g., Can neuroimaging distinguish deep from shallow studying of a text? Our focus

4. 4 Traditional methods for studying learning Design experiment –Modify text, classroom activities, tests… –e.g., Project Scale-up Lab experiment –Modify just one factor –Brief; money instead of grades, …

5. 5 PSLC methods Educational data mining –Logs from instrumented courses –Some analysis is automated In vivo experiments –Control of variables –Instrumented courses Next

6. 6 Instrumented courses (Called LearnLab courses) Existing class + data collection –Homework done on a tutoring system or photocopied and analyzed –Photocopies of quizzes, exams –FCI given before and after the course –Demographics, GPAs, Majors… –Handouts, slides, clicker data,… Instructor, student & IRB cooperation –Anonymity

7. 7 Existing Physics LearnLab Course(s) US Naval Academy –Course take by all 2 nd year students –LearnLab is in 4 of about 20 sections –Profs. Wintersgill, McClanahan Your course here

8. 8 Basic data mining question What features of students’ histories are statistically associated with learning gains? e.g., What are the differences between histories of Student A and Student F? Student A:25% on pretest Semester-long history85% on post-test Student F:25% on pretest Semester-long history20% on post-test

9. 9 Knowledge decomposition hypothesis Decompose knowledge to be learned into a set of knowledge components –e.g., Newton’s third law –e.g., Centripetal acceleration Assume each knowledge component is learned independently –An approximation/idealization

10. 10 Data mining with knowledge components (KCs) StudentKCPre-testHistoryPost-test A135%…85% A215%…10% A325%…20% B150%…20% B210%… B325%…80% For each KC, find statistical associations between histories and gains.

11. 11 History decomposition hypothesis Decompose the student’s history into events such that each event addresses only one (or a few) knowledge components. –Reading a paragraph about Newton’s 3 rd law –Drawing a reaction force vector –Seeing the instructor draw a reaction force –Drawing a centripetal acceleration vector Assume that a KC’s learning gain depends only on that KC’s events

12. 12 Events for 1 student on 1 KC (e.g., Newton’s 3 rd law) TimeContextBehavior 8/27/07 9:05FCI item 3Incorrect 8/27/07 9:12FCI item 10Incorrect 9/13/07 18:06Textbook, pg. 111Highlighted 9/13/07 21:11Problem 5-11, drawing FBD Omitted force on the hand due to block 9/14/07 9:12Lecture, slide 20Taking notes 9/15/07 22:05Problem 5-11, drawing FBD Draws force on the hand due to block etc.

13. 13 Some events are not currently available TimeContextBehavior 8/27/07 9:05FCI item 3Incorrect 8/27/07 9:12FCI item 10Incorrect 9/13/07 18:06Textbook, pg. 111Highlighted 9/13/07 21:11Problem 5-11, drawing FBD Omitted force on the hand due to block 9/14/07 9:12Lecture, slide 20Taking notes 9/15/07 22:05Problem 5-11, drawing FBD Draws force on the hand due to block etc.

14. 14 More feasible data mining Predict learning gains of a KC given the sequence of events relevant to that KC On an event that assesses mastery of a KC, predict the student’s performance during that event given the sequence of preceding events relevant to that KC

15. 15 Predicting correctness of events that assess mastery Context Event type P(Correct) FCI item 3Assessment0.10 FCI item 10Assessment0.12 Reading textbook, pg. 111, paragraph 3 InstructionNot applicable Problem 5-11, drawing force on hand due to block Assessment0.30 Lecture, slide 20InstructionNot applicable Problem 5-11, drawing force on hand … with remedial feedback if needed Assessment then instruction 0.55

16. 16 Learning curves Plot assessment events on x-axis –Ordered chronologically Plot measure of mastery on y-axis –Usually aggregated across subjects e.g., proportion of 100 subjects who performed correctly on this event

17. 17 An expected learning curve Frequency of correct 0 0.5 1.0 12345678 Assessment events

18. 18 Summary of PSLC educational data mining Given knowledge to be learned –Decompose into knowledge components Given students’ histories from an instrumented course –Divide into assessment/instruction events –such that one KC (or a few) per event For each KC, find a function on a sequence of events that predicts the KC’s –learning gain during the course –learning curve

19. 19 PSLC methods Educational data mining –Logs from instrumented courses –Some analysis is automated Andes produces logs with KCs DataShop draws learning curves, etc. Correlation ≠ Causation In vivo experiments –Control of variables –Instrumented courses Next

20. 20 Two major types of in vivo experiments Short & fat –During one lesson or one unit Long & skinny –During whole course –“invisibly”

21. 21 Example of a short, fat, in vivo experiment (Hausmann 07) During a 2-hour period (usually used for lab work) ~25 students in the room, each with a laptop and a headset mike Repeat 3 times: –Study a video while explaining it into the mike –Solve a problem 4 experimental conditions, varying the content of the video and the instructions for explaining it Random assignment of students to conditions Dependent measures include learning curves Result: Instructions to self-explain worked best regardless of content of the video

22. 22 Example of a long, skinny in vivo experiment (Katz 07) During 8 weeks of a 13-week course Random assignment to 2 conditions: –Experimental group: After solving certain homework problems, the student discussed the solution with a natural language tutoring system –Control group: Extra homework problems Result: Experiment > Control on some conceptual measures

23. 23 Robust Learning Immediate learning –During an immediate post-test –Similar content to training (near transfer) Robust learning –Far transfer –Retention –Acceleration of future learning Does manipulation of instruction on topic A affect rate of learning of a later topic, B?

24. 24 Summary of PSLC methodology Data mining –Instrumented (LearnLab) courses –Knowledge components –Instructional and assessment events –Learning curves In vivo experiments –Short & fat vs. long & skinny –Robust learning

25. 25 Agenda I.LearnLab methodology II.Demonstration of Andes, an intelligent homework tutor III.Log File Analysis Next

26. 26

27. 27 Define variables Draw free body diagram (3 vectors and body) Define coordinates (3 choices for this problem) Upon request, Andes gives hints for what to do next

28. 28 Principle-based help for incorrect entry Red/green gives immediate feedback for student actions

29. 29

30. 30 # Log of Andes session begun Tuesday, July 17, 2007 12:12:28 by [User] on [Computer]... 05:03 DDE (read-problem-info "S2E" 0 0)... 02:35 Axes Axes-671 64 335 143 296 02:35 Axes-dlg Axes-671 || … 02:38 C dir 40 02:42 BTN-CLICK 1 OK 02:42 DDE (assert-x-axis NIL 40 Axes-671 "x" "y" "z") 02:42 DDE-COMMAND assoc step (DRAW-AXES 40) 02:42 DDE-COMMAND assoc op DRAW-VECTOR-ALIGNED-AXES 02:42 DDE-COMMAND set-score 39 02:42 DDE-RESULT |T|... 10:02 E 0 F1_y+F2_y=0 10:02 EQ-SUBMIT 0 10:02 DDE (lookup-eqn-string "F1_y+F2_y=0" 0) 10:47 DDE-COMMAND assoc parse (= (+ Yc_Fn_BALL_WALL1_1_40 Yc_Fn_BALL_WALL2_1_40) 0) 10:47 DDE-COMMAND assoc error MISSING-FORCES-IN-Y-AXIS-SUM 10:47 DDE-COMMAND assoc step (EQN (= (+ Yc_Fw_BALL_EARTH_1_40 Yc_Fn_BALL_WALL2_1_40 Yc_Fn_BALL_WALL1_1_40) 0)) 10:47 DDE-COMMAND assoc op WRITE-NFL-COMPO 10:47 DDE-RESULT |NIL|... 10:50 DDE-RESULT |!show-hint There is a force acting on the ball at T0 that you have not yet drawn.~e|... 16:38 END-LOG problem name session time student action (equation) error analysis: intended action student action (draw axes) interpretation: compare to model green red

31. 31 Demonstration by Tim Nokes # Log of Andes session begun Wednesday, April 18, 2007 21:08:07 by [user] on [computer]... 0:02DDE (read-problem-info "FARA9" 0 0)... 0:13Help-Hint 0:13DDE (Get-Proc-Help) 0:13DDE-COMMAND assoc (NSH NEW-START-AXIS 0) 0:13DDE-RESULT |!show-hint It is a good idea to begin most problems by drawing an axis. This helps to ground your work and will be useful later on in the process.~e| … 0:17Begin-draw 50001 Axes-1 185 331... 0:30New-Variable resistance... 0:39DDE (define-variable "R" |NIL| |resistance| |R| |NIL| |NIL| Var-2 "20 ohm") 0:39DDE-COMMAND assoc step (DEFINE-VAR (RESISTANCE R)) 0:39DDE-COMMAND assoc op DEFINE-RESISTANCE-VAR 0:39DDE-COMMAND assoc parse (= R_R (DNUM 20 ohm)) 0:39DDE-COMMAND set-score 3 0:39DDE-RESULT |T|.... 0:50DDE (lookup-vector "B" Unspecified B-field |s| NIL 0 |NIL| Vector-3) 0:50DDE-COMMAND assoc entry (VECTOR (FIELD S MAGNETIC UNSPECIFIED TIME NIL) ZERO) 0:50DDE-COMMAND assoc error DEFAULT-SHOULD-BE-NON-ZERO 0:50DDE-COMMAND assoc step (VECTOR (FIELD S MAGNETIC UNSPECIFIED TIME NIL) OUT-OF) 0:50DDE-COMMAND assoc op DRAW-FIELD-GIVEN-DIR 0:50DDE-COMMAND set-score 2 0:50DDE-RESULT |NIL|... 9:51DDE-RESULT |T| 9:55END-LOG

32. 32 Agenda I.LearnLab methodology II.Demonstration of Andes, an intelligent homework tutor III.Log File Analysis Next

33. 33 Model Solution Set Solution 0 Principle A Op 1 Op 3 Op 6 Op 7 Principle B Op 2 Op 3 Op 5 Op 8 Op 10 Solution 1 Principle C Op 10 Op 11 Op 12 Principle A Op 1 Op 3 Op 6 Op 7 Principle D Assumption: Op i = KC

34. 34 # Log of Andes session begun Friday, July 27, 2007 14:29:38 by bobh on BOBH … 0:02 DDE (read-problem-info "S2E" 0 0) … 11:45 Vector-dlg Vector-673 || … 11:48 CLOSE type instantaneous 11:48 SEL type 1 instantaneous 11:51 BTN-CLICK 1 OK 11:51 DDE (lookup-vector "a" instantaneous Acceleration |ball| NIL 0 |T0| Vector-673) 11:51 DDE-COMMAND assoc step (VECTOR (ACCEL BALL :TIME 1) ZERO) 11:51 DDE-COMMAND assoc op ACCEL-AT-REST 11:51 DDE-RESULT |T| … problem name student actions match model solution: assoc step = entry Assoc op = operator green

35. 35 14:03E 8 Fearth_y = m*g 14:11EQ-SUBMIT 8 14:11DDE (lookup-eqn-string "Fearth_y = m*g" 8) 14:11DDE-COMMAND assoc parse (= Yc_Fw_BALL_EARTH_1_0 (* m_BALL g_EARTH)) 14:11DDE-COMMAND assoc error MISSING-NEGATION-ON- VECTOR-COMPONENT 14:11DDE-COMMAND assoc step (EQN (= Fw_BALL_EARTH_1 (* m_BALL g_EARTH))),(EQN (= Yc_Fw_BALL_EARTH_1_0 (- Fw_BALL_EARTH_1))) 14:11DDE-COMMAND assoc op WT-LAW,COMPO-PARALLEL-AXIS 14:11DDE-COMMAND set-score 74 14:11EQ-F 8 14:11DDE-RESULT |NIL| student actions red guess intended error interpretation

36. 36 Review Video Match steps in video to log file

37. 37 Researchable questions Timing Sequencing (order of steps) Hint Usage Problem solving skills Errors as window to mental state Self-correction of errors

38. 38 DataShop https://learnlab.web.cmu.edu/ > Launch DataShop > New user? Sign up now! > (Create account)