1. Bayes Net Toolbox for Student Modeling (BNT-SM)
21/10/10
Bayes Net Toolbox for Student Modeling (BNT-SM)
Kai-min Chang & Yanbo Xu
EDM2015 SMART Tutorial
June 26, 2015
This work was supported by the National Science Foundation under Cyberlearning Grant IIS The opinions expressed are those of the authors and do not necessarily represent the views of the Institute, or the National Science Foundation.

2. Download BNT-SM Matlab Octave http://www.cs.cmu.edu/~listen/BNT-SM
Check your university’s license
Octave

3. Bayes Net Toolkit for Student Modeling (BNT-SM)
Estimates
of model
parameters
BayesNet.xml
BNT-SM
BNT
Observations
of student
performance
Estimates
of student
knowledge

4. Model Knowledge Tracing with Some Bayes Nets Packages
Popular Bayes Nets packages (at the time…)
BNT, BUGS, GMTK
Complicated coding effort to use the packages
Create toolkit to hide the complexity from ITS researchers.
Extend Bayes Nets Toolkit (BNT; Murphy, 1998)

5. Revision History First developed 9 years ago (Chang et al., 2006)
Incorporated LR-DBN (Xu & Mostow, 2011)
Updated for the tutorial (SMART, 2015)
To run on Octave (used a workaround since xmlread.m is not implemented in Octave).
Attempted to use the most updated BNT (unsuccessful).

6. Knowledge Tracing (KT)
Corbett & Anderson (1995)
Infers a student’s proficiency on a skill from observation of applying the skill.
Maintain 4 skill-specific parameters:
Learning parameters: Already know & learn
Performance parameters: guess & slip
Knowledge of the skill “cat”
0.40
0.32
0.21
0.29
0.45
0.38
Correct  

7. Bayesian Knowledge Tracing (BKT)
Affect future attempt
Student Knowledge (K0)
Student Knowledge (Kn)
Already know
learn, forget
Affect current attempt
guess, slip
Student Performance (C0)
Student Performance (Cn)

8. Bayes Net Toolkit for Student Modeling (BNT-SM)
Step 4
Step 1
Step 3
Estimates
of model
parameters
BayesNet.xml
BNT-SM
Step 2
BNT
Step 5
Observations
of student
performance
Estimates
of student
knowledge

9. 1a. Specify Network Structure
<nodes>
<node>
<name>StudentKnowledge</name>
<values>2 </values>
<type> discrete </type>
<observed> latent</observed>
<within> StudentPerformance
</within>
<between>StudentKnowledge
</between>
</node> ……
Student Knowledge (K0) Kn
Student Performance (C0) Cn

10. 1b. Initialize Model Parameters
Already
know
Student Knowledge (K0) Kn
Learn
XML
Guess,
Slip
Student Performance (C0) Cn

11. 2. Provide Observations of Student Performance
Ordered attempts within a skill
Comprehensive observational data
Skill
User
Performance Time
Student Knowledge
Student Performance my
Kevin
:30:21 ? 
cat
:30:22 …

12. 3. Run BNT-SM Estimates of model parameters BayesNet.xml BNT-SM BNT
Observations
of student
performance
Estimates
of student
knowledge
I will describe the output of BNT-SM more in the evaluation section.

13. 4. Examine Model Parameters
Parameter values can be used to infer which skill is more difficult
Skill
# Users
# Cases LL L0
Guess
Slip
Learn
Forget My 14 23
-5.15
0.46
0.73
0.04
0.20
0.00
Cat 46
219
-94.07
0.44
0.70
0.11
0.24
Average log likelihood can be used to compare across different models

14. 5. Examine Model Inferences
User
Skill
Knowledge
Correct
Alice My
NULL
Bob
Cathy …
John
Cat
Kevin
Estimates the progression of student knowledge that can be used by the tutor

15. Matlab / Octave >> cd BNT-SM/src >> setup
>> cd ../model/kt
>> [property evidence hash_bnet] = RunBnet('property.xml');
Sample Bayes net models
An XML file that specifies the Bayes net we are constructing.

16. Examine Some Files
>> vim property.xml (Matlab) >> vim load_property.m (Octave) >> vim evidence.train.xls >> vim evidence.test.xls >> vim param_table.xls >> vim inference_result_header.xls >> vim inference_result.xls
Input
Output

17. Model 1: KT Corbett & Anderson (1995) … … l0 l f l f K(1) K(t) K(t+1)
g s
g s
g s
C(1)
C(t)
C(t+1)

18. Model 1: KT  ? ? ?   Corbett & Anderson (1995) … … l0 l f l f g s

19. Model 2: LR-DBN
An Extension of BNT-SM to trace multiple subskills (Xu & Mostow, 2012)
S1(t+1)
S1(t)
S2(t+1)
S2(t)
Sm(t)
Sm(t+1) … …
K(1) l0 le fe
K(t) le fe
K(t+1) … …
g s
g s
g s
C(1)
C(t)
C(t+1) 19

20. Model 3: EEG-KT
Insert a binary EEG measure into KT (Yuan et al., 2014) l0 le fe le fe
K(1)
K(t)
K(t+1) … …
E(1)
E(t)
E(t+1)
ge se
ge se
ge se
ge se
C(1)
C(t)
C(t+1)

21. Model 3: EEG-KT
Insert a binary EEG measure into KT (Yuan et al., 2014) le fe le fe l0 ? ? ? … …
ge se
ge se
ge se
ge se   
Idea
Model
Evaluate
Discuss

22. References
Chang, K.M., Beck, J.E., Mostow, J., & Corbett, A. (2006). A Bayes net toolkit for student modeling in intelligent tutoring systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan,
Xu, Y., & Mostow, J. (2011). Using Logistic Regression to Trace Multiple Subskills in a Dynamic Bayes Net. Proceedings of the 4th International Conference on Educational Data Mining Eindhoven, Netherlands,
Yuan, Y., Chang, K.M., Xu, Y., & Mostow, J. (2014). A Public Toolkit and ITS Dataset for EEG. Proceedings of the 12th International Conference on Intelligent Tutoring Systems Workshop on Utilizing EEG Input in Intelligent Tutoring Systems, Honolulu, Hawaii.

23. Questions? Kai-min Kevin Chang Yanbo Xu
Language Technologies Institute, Carnegie Mellon University
Yanbo Xu
Robotics Institute, Carnegie Mellon University