1. Boredom Across Activities, and Across the Year, within Reasoning Mind William L. Miller, Ryan Baker, Mathew Labrum, Karen Petsche, Angela Z. Wagner

2. In recent years Increasing interest in modeling more about students than just what they know

3. In recent years Increasing interest in modeling more about students than just what they know Can we assess a broad range of constructs

4. In recent years Increasing interest in modeling more about students than just what they know Can we assess a broad range of constructs In a broad range of contexts

5. Boredom A particularly important construct to measure

6. Boredom is Common in real-world learning (D’Mello, 2013)

7. Boredom is Common in real-world learning (D’Mello, 2013) Associated with worse learning outcomes in the short-term (Craig et al., 2004; Rodrigo et al., 2007)

8. Boredom is Common in real-world learning (D’Mello, 2013) Associated with worse learning outcomes in the short-term (Craig et al., 2004; Rodrigo et al., 2007) Associated with worse course grades and standardized exam performance (Pekrun et al., 2010; Pardos et al., 2013)

9. Boredom is Common in real-world learning (D’Mello, 2013) Associated with worse learning outcomes in the short-term (Craig et al., 2004; Rodrigo et al., 2007) Associated with worse course grades and standardized exam performance (Pekrun et al., 2010; Pardos et al., 2013) Associated with lower probability of going to college, years later (San Pedro et al., 2013)

10. Online learning environments Offer great opportunities to study boredom in context – Very fine-grained interaction logs that indicate everything the student did in the system

11. Automated boredom detection Can we detect boredom in real time, while a student is learning? Can we detect boredom retrospectively, from log files?

12. Automated boredom detection Can we detect boredom in real time, while a student is learning? Can we detect boredom retrospectively, from log files? Would allow us to study affect at a large scale – Figure out which content is most boring, in order to improve it

13. Affect Detection: Physical Sensors? Lots of work shows that affect can be detected using physical sensors – Tone of voice (Litman & Forbes-Riley, 2005) – EEG (Conati & McLaren, 2009) – Posture sensor and video (D’Mello et al., 2007) It’s hypothesized – but not yet conclusively demonstrated – that using physical sensors may lead to better performance than interaction logs alone

14. Sensor-free affect detection Easier to scale to the millions of students who use online learning environments In settings that do not have cameras, microphones, and other physical sensors – Home settings have parents bought equipment? can they set it up and maintain it? – Classroom settings can school maintain equipment? do students intentionally destroy equipment? parent concerns and political climate

15. Sensor-free boredom detection Has been developed for multiple learning environments – Problem solving tutors (Baker et al. 2012; Pardos et al. 2013) – Dialogue tutors (D’Mello et al. 2008) – Narrative virtual learning environments (Sabourin et al. 2011; Baker et al. 2014) – Science simulations (Paquette et al., 2014) The principles of affect detection are largely the same across environments But the behaviors associated with boredom differ considerably between environments

16. This talk We discuss our work to develop sensor-free boredom detection for Reasoning Mind Genie 2 (Khachatryan et al, 2014) Self-paced blended learning mathematics curriculum for elementary school students – Youngest population for sensor-free affect detection so far Used by approximately 100,000 students a year

17. Reasoning Mind Genie 2 Combines – Guided Study with a pedagogical agent “Genie” – Speed Games that support development of fluency Used in schools 3-5 days a week for 45-90 minutes per day

18. Reasoning Mind Genie 2 (a) (b) (c)

19. Reasoning Mind Genie 2 Better affect and more on-task behavior than most pedagogies, online or offline (Ocumpaugh et al., 2013) Still a substantial amount of boredom Reducing boredom is a key goal

20. Role for affect detection If we can detect boredom in log files We can determine which content is more boring, and improve that content

21. Related Work Evidence that specific design features associated with boredom in Cognitive Tutors for high school algebra (Doddannara et al., 2013)

22. Related Work Evidence that specific design features associated with boredom in Cognitive Tutors for high school algebra (Doddannara et al., 2013) Evidence that some disengaged behaviors increase during the year (Beck, 2005) – Important to verify that differences in affect due to actual content/design, not time of year

23. Approach to Boredom Detection Collect “ground truth” data on student boredom, using field observations Synchronize log data to field observations Distill meaningful data features of log data, hypothesized to relate to boredom Develop automated detector using classification algorithm Validate detector for new students/new lessons/new populations

24. BROMP 2.0 Field Observations (Ocumpaugh et al., 2012) Conducted through Android app HART (Baker et al., 2012) Protocol designed to reduce disruption to student – Some features of protocol: observe with peripheral vision or side glances, hover over student not being observed, 20-second “round-robin” observations of several students, bored-looking people are boring Inter-rater reliability around 0.8 for behavior, 0.65 for affect 64 coders now certified in USA, Philippines, India

25. Data collection 408 elementary school students

26. Data collection Diverse sample important for model generalizability (Ocumpaugh et al., 2014) 11 different 8 th grade classes 6 schools – 2 urban in Texas, predominantly African-American – 1 urban in Texas, predominantly Latino – 1 suburban in Texas, predominantly White – 1 suburban in Texas, mixed ethnicity/race – 1 rural in West Virginia, predominantly White

27. Affect coding 3 expert coders observed each student using BROMP Coded 5 categories of affect – Engaged Concentration – Boredom – Confusion – Frustration – ? 4891 observations collected in RM classrooms

28. Building detectors Observations were synchronized with the logs of the students interactions with RM, using HART app and internet time server For each observation, a set of 93 meaningful features describing the student’s behavior was engineered Computed on actions occurring during or preceding an observation (up to 20 seconds before)

29. Features: Examples Individual action features – Whether an action was correct or not – How long the action took Features across all past activity – Fraction of previous attempts on the current skill the student has gotten correct Other known models applied to logs – Probability student knows skill (Bayesian Knowledge Tracing) – Carelessness – Moment-by-Moment Learning Graph

30. Automated detector of boredom Detectors were built using RapidMiner 5.3 For each algorithm the best features were selected using forward selection/backward elimination Data was re-sampled to have more equal class frequencies; models were evaluated on original class distribution Detectors were validated using 10-fold student-level cross-validation

31. Automated detector of boredom Detectors were built using 4 machine learning algorithms that have been successful for building affect detectors in the past: – J48 – JRip – Step Regression – Naïve Bayes

32. Best One Detectors were built using 4 machine learning algorithms that have been successful for building affect detectors in the past: – J48 – JRip – Step Regression – Naïve Bayes

33. Machine learning Performance of the detectors was evaluated using A’ – Given two observations, probability of correctly identifying which one is an example of a specific affective state and which one is not – A’ of 0.5 is chance level and 1 is perfect – Identical to Wilcoxon statistic – Very similar to AUC ROC (Area Under the Receiver- Operating Characteristic Curve)

34. Results A’ = 0.64 Compared to similar detectors in other systems, validated in similar stringent fashion SystemA' Cognitive Tutor Algebra (Baker et al. 2012) 0.69 ASSISTments (Pardos et al. 2013) 0.63 EcoMUVE (Baker et al. 2014) 0.65 Inq-ITS (Paquette et al. 2014) 0.72

35. CoefficientFeature +0.212 The standard deviation, across the clip, of student correctness (1 or 0) on each action. -0.013 The number of actions in the clip that occurred on Speed Game items. -0.070 The fraction of the total clip duration spent on Speed Game items. -0.073 The number of actions in the clip on items where the answer input was made by selecting an item from a drop-down list. +0.290 The minimum slip parameter (P(S) in Bayesian Knowledge Tracing) on skills in the clip. -0.260 The standard deviation, across the clip, of the action duration, normalized across all students, times the presence (1) or absence (0) of a hint request on the previous action. +0.123 Y-intercept.

36. Using detectors Model applied to entire year of data from these classrooms 2,974,944 actions by 462 students Includes 54 additional students not present during observations Aggregation over pseudo-confidences rather than binary predictions – Retains more information

38. Apparent downward trend

39. Is it statistically significant?

40. Apparent downward trend Is it statistically significant? Yes. Students are less bored later in the year F-test controlling for student p<0.001

41. Is it practically significant?

42. No. r = -0.06

43. Is it practically significant? No. r = -0.06 With large enough samples, anything is statistically significant

44. Kind of a positive thing At minimum, students aren’t getting more bored as the year goes on In other systems, students get more disengaged as the year goes on (Beck, 2005) And the overall level of boredom (~14%) is not very high

45. Beyond this Curriculum is self-paced

46. Beyond this Curriculum is self-paced Which means that predicting boredom by date may obscure real variation

47. Beyond this Curriculum is self-paced Which means that predicting boredom by date may obscure real variation Instead, look at boredom by learning objective

49. Predicting boredom by objective p<0.001 r=0.343

50. If we cluster objectives into two groups “High boredom” “Low boredom” Ignoring the one point in between the two groups Cohen’s D = 0.67

51. Future work So… what is it that differentiates the higher boredom lessons from the lower boredom lessons?

52. Future work So… what is it that differentiates the higher boredom lessons from the lower boredom lessons? – Nothing obvious, unfortunately…

53. Future work So… what is it that differentiates the higher boredom lessons from the lower boredom lessons? – Nothing obvious, unfortunately… – May be necessary to develop a taxonomy of potential differences, and see which are predictive – May be possible to build off prior work by (Doddannara et al., 2013) that did exactly this for Cognitive Tutor

54. Future Work Can we fix the more boring lessons? – Either by determining why they are boring – Or just by adding a little more “fun content”

55. Eventual Goal Use precise assessments of boredom to help us enhance Reasoning Mind – Improving engagement – Improving learning outcomes

56. Thank you twitter.com/BakerEDMLab Baker EDM Lab See our free online MOOT “Big Data and Education” All lab publications available online – Google “Ryan Baker” “Data, Analytics, and Learning” – EdX, Fall 2014