Bayes Net Toolbox for Student Modeling (BNT-SM) 21/10/10 Bayes Net Toolbox for Student Modeling (BNT-SM) Kai-min Chang & Yanbo Xu EDM2015 SMART Tutorial June 26, 2015 This work was supported by the National Science Foundation under Cyberlearning Grant IIS1124240. The opinions expressed are those of the authors and do not necessarily represent the views of the Institute, or the National Science Foundation.

Download BNT-SM Matlab Octave http://www.cs.cmu.edu/~listen/BNT-SM Check your university’s license Octave http://www.gnu.org/software/octave

Bayes Net Toolkit for Student Modeling (BNT-SM) Estimates of model parameters BayesNet.xml BNT-SM BNT Observations of student performance Estimates of student knowledge

Model Knowledge Tracing with Some Bayes Nets Packages Popular Bayes Nets packages (at the time…) BNT, BUGS, GMTK Complicated coding effort to use the packages Create toolkit to hide the complexity from ITS researchers. Extend Bayes Nets Toolkit (BNT; Murphy, 1998)

Revision History First developed 9 years ago (Chang et al., 2006) Incorporated LR-DBN (Xu & Mostow, 2011) Updated for the tutorial (SMART, 2015) To run on Octave (used a workaround since xmlread.m is not implemented in Octave). Attempted to use the most updated BNT (unsuccessful).

Knowledge Tracing (KT) Corbett & Anderson (1995) Infers a student’s proficiency on a skill from observation of applying the skill. Maintain 4 skill-specific parameters: Learning parameters: Already know & learn Performance parameters: guess & slip Knowledge of the skill “cat” 0.40 0.32 0.21 0.29 0.45 0.38 Correct  

Bayesian Knowledge Tracing (BKT) Affect future attempt Student Knowledge (K0) Student Knowledge (Kn) Already know learn, forget Affect current attempt guess, slip Student Performance (C0) Student Performance (Cn)

Bayes Net Toolkit for Student Modeling (BNT-SM) Step 4 Step 1 Step 3 Estimates of model parameters BayesNet.xml BNT-SM Step 2 BNT Step 5 Observations of student performance Estimates of student knowledge

1a. Specify Network Structure <nodes> <node> <name>StudentKnowledge</name> <values>2 </values> <type> discrete </type> <observed> latent</observed> <within> StudentPerformance </within> <between>StudentKnowledge </between> </node> …… Student Knowledge (K0) Kn Student Performance (C0) Cn

1b. Initialize Model Parameters Already know Student Knowledge (K0) Kn Learn XML Guess, Slip Student Performance (C0) Cn

2. Provide Observations of Student Performance Ordered attempts within a skill Comprehensive observational data Skill User Performance Time Student Knowledge Student Performance my Kevin 2002-08-14 15:30:21 ?  cat 2002-08-14 15:30:22 …

3. Run BNT-SM Estimates of model parameters BayesNet.xml BNT-SM BNT Observations of student performance Estimates of student knowledge I will describe the output of BNT-SM more in the evaluation section.

4. Examine Model Parameters Parameter values can be used to infer which skill is more difficult Skill # Users # Cases LL L0 Guess Slip Learn Forget My 14 23 -5.15 0.46 0.73 0.04 0.20 0.00 Cat 46 219 -94.07 0.44 0.70 0.11 0.24 Average log likelihood can be used to compare across different models

5. Examine Model Inferences User Skill Knowledge Correct Alice My 0.463045 NULL Bob Cathy … John Cat 0.440226 Kevin 0.620417 Estimates the progression of student knowledge that can be used by the tutor

Matlab / Octave >> cd BNT-SM/src >> setup >> cd ../model/kt >> [property evidence hash_bnet] = RunBnet('property.xml'); Sample Bayes net models An XML file that specifies the Bayes net we are constructing.

Examine Some Files >> vim property.xml (Matlab) >> vim load_property.m (Octave) >> vim evidence.train.xls >> vim evidence.test.xls >> vim param_table.xls >> vim inference_result_header.xls >> vim inference_result.xls Input Output

Model 1: KT Corbett & Anderson (1995) … … l0 l f l f K(1) K(t) K(t+1) g s g s g s C(1) C(t) C(t+1)

Model 1: KT  ? ? ?   Corbett & Anderson (1995) … … l0 l f l f g s

Model 2: LR-DBN An Extension of BNT-SM to trace multiple subskills (Xu & Mostow, 2012) S1(t+1) S1(t) S2(t+1) S2(t) Sm(t) Sm(t+1) … … K(1) l0 le fe K(t) le fe K(t+1) … … g s g s g s C(1) C(t) C(t+1) 19

Model 3: EEG-KT Insert a binary EEG measure into KT (Yuan et al., 2014) l0 le fe le fe K(1) K(t) K(t+1) … … E(1) E(t) E(t+1) ge se ge se ge se ge se C(1) C(t) C(t+1)

Model 3: EEG-KT Insert a binary EEG measure into KT (Yuan et al., 2014) le fe le fe l0 ? ? ? … … ge se ge se ge se ge se    Idea Model Evaluate Discuss

References Chang, K.M., Beck, J.E., Mostow, J., & Corbett, A. (2006). A Bayes net toolkit for student modeling in intelligent tutoring systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, 104-113. Xu, Y., & Mostow, J. (2011). Using Logistic Regression to Trace Multiple Subskills in a Dynamic Bayes Net. Proceedings of the 4th International Conference on Educational Data Mining Eindhoven, Netherlands, 241-245. Yuan, Y., Chang, K.M., Xu, Y., & Mostow, J. (2014). A Public Toolkit and ITS Dataset for EEG. Proceedings of the 12th International Conference on Intelligent Tutoring Systems Workshop on Utilizing EEG Input in Intelligent Tutoring Systems, Honolulu, Hawaii.

Questions? Kai-min Kevin Chang Yanbo Xu Language Technologies Institute, Carnegie Mellon University E-mail: kkchang@cs.cmu.edu Yanbo Xu Robotics Institute, Carnegie Mellon University E-mail: yanbox@cs.cmu.edu