1. Advanced Methods and Analysis for the Learning and Social Sciences PSY505 Spring term, 2012 January 30, 2012

2. Today’s Class Bayesian Knowledge Tracing

3. How does BKT differ from PFA?

4. Assesses latent knowledge as well as probability of correctness Only handles one skill per item (extensions can handle this)

5. How does BKT differ from IRT?

6. Takes learning into account Ignores item-level differences in difficulty

7. What is the typical use of BKT? Assess a student’s knowledge of topic X Based on a sequence of items that are dichotomously scored – E.g. the student can get a score of 0 or 1 on each item Where each item corresponds to a single skill Where the student can learn on each item, due to help, feedback, scaffolding, etc.

8. Key assumptions Each item must involve a single latent trait or skill – Different from PFA Each skill has four parameters From these parameters, and the pattern of successes and failures the student has had on each relevant skill so far, we can compute latent knowledge P(Ln) and the probability P(CORR) that the learner will get the item correct

9. Key Assumptions Two-state learning model – Each skill is either learned or unlearned In problem-solving, the student can learn a skill at each opportunity to apply the skill A student does not forget a skill, once he or she knows it

10. Model Performance Assumptions If the student knows a skill, there is still some chance the student will slip and make a mistake. If the student does not know a skill, there is still some chance the student will guess correctly.

11. Corbett and Anderson’s Model Not learned Two Learning Parameters p(L 0 )Probability the skill is already known before the first opportunity to use the skill in problem solving. p(T)Probability the skill will be learned at each opportunity to use the skill. Two Performance Parameters p(G)Probability the student will guess correctly if the skill is not known. p(S)Probability the student will slip (make a mistake) if the skill is known. Learned p(T) correct p(G)1-p(S) p(L 0 )

12. Bayesian Knowledge Tracing Whenever the student has an opportunity to use a skill, the probability that the student knows the skill is updated using formulas derived from Bayes’ Theorem.

13. Formulas

14. BKT Only uses first problem attempt on each item (just like PFA) What are the advantages and disadvantages? Note that several variants to BKT break this assumption at least in part – more on that later

15. Knowledge Tracing How do we know if a knowledge tracing model is any good? Our primary goal is to predict knowledge

16. Knowledge Tracing How do we know if a knowledge tracing model is any good? Our primary goal is to predict knowledge But knowledge is a latent trait

17. Knowledge Tracing How do we know if a knowledge tracing model is any good? Our primary goal is to predict knowledge But knowledge is a latent trait But we can check those knowledge predictions by checking how well the model predicts performance

18. Fitting a Knowledge-Tracing Model In principle, any set of four parameters can be used by knowledge-tracing But parameters that predict student performance better are preferred

19. Knowledge Tracing So, we pick the knowledge tracing parameters that best predict performance Defined as whether a student’s action will be correct or wrong at a given time

20. Fit Methods There are many fit methods Before we go into them in detail, let’s discuss the homework solutions

21. Homework Solutions

22. Sweet’s BKT Solution Sweet, can you please talk us through your spreadsheet? Also, tell us (but no need to show us) how you computed the parameter values

23. Zak’s BKT Solution Zak got the exact same parameter values as Sweet did Zak, why did this happen?

24. Goodness Zak PFA 12,122.77 (dummy values) Sweet PFA 10,896.09(fit in Matlab) Sweet BKT 8140.995 Zak BKT 8140.995

25. Thoughts? Why might BKT have worked better than PFA?

26. Excel issues Several folks had Excel crashing issues on this assignment Anyone want to discuss the problems you had? We can discuss how to fix them

27. Fit Methods Hill-Climbing Hill-Climbing (Randomized Restart) Iterative Gradient Descent (and variants) Expectation Maximization (and variants) Brute Force/Grid Search

28. Hill-Climbing The simplest space search algorithm Start from some choice of parameter values Try moving some parameter value in either direction by some amount – If the model gets better, keep moving in the same direction by the same amount until it stops getting better Then you can try moving by a smaller amount – If the model gets worse, try the opposite direction

29. Hill-Climbing Vulnerable to Local Minima – a point in the data space where no move makes your model better – but there is some other point in the data space that *is* better Unclear if this is a problem for BKT – IGD (which is a variant on hill-climbing) typically does worse than Brute Force (Baker et al., 2008) – Pardos et al. (2010) did not find evidence for local minima (but with simulated data)

30. Pardos et al., 2010

31. Let’s try Hill-Climbing On assignment data set For one skill Let’s use 0.1 as the starting point for all four parameters

32. Hill-Climbing with Randomized Restart One way of addressing local minima is to run the algorithms with randomly selected different initial parameter values

33. Let’s try Hill-Climbing On assignment data set For one skill Let’s run four times with different randomly selected parameters

34. Iterative Gradient Descent Find which set of parameters and step size (may be different for different parameters) leads to the best improvement Use that set of parameters and step size Repeat

35. Conjugate Gradient Descent Variant of Iterative Gradient Descent (used by Albert Corbett and Excel) Rather complex to explain “I assume that you have taken a first course in linear algebra, and that you have a solid understanding of matrix multiplication and linear independence” – J.G. Shewchuk, An Introduction to the Conjugate Gradient Method Without the Agonizing Pain. (p. 5 of 58)

36. Expectation Maximization (Thanks to Joe Beck for explaining this to me) 1.Starts with initial values for L0, T, G, S 2.Estimates student knowledge P(Ln) at each problem step 3.Estimates L0, T, G, S using student knowledge estimates 4.If goodness is substantially different from last time it was estimated, and max iterations has not been reached, go to step 2

37. Expectation Maximization EM is vulnerable to local minima just like hill- climbing and gradient descent Randomized restart typically used

38. Let’s try Expectation Maximization By hand in Excel Log likelihood is typically used, but for ease of real-time calculation we will use SSR

39. Brute Force/Grid Search Try all combination of values at a 0.01 grain-size: L0=0, T=0, G= 0, S=0 L0=0.01, T=0, G= 0, S=0 L0=0.02, T=0, G= 0, S=0 … L0=1,T=0,G=0,S=0 … L0=1,T=1,G=0.3,S=0.3 I’ll explain this soon

40. Which is best? EM better than CGD – Chang et al., 2006  A’= 0.05 CGD better than EM – Baker et al., 2008  A’= 0.01 EM better than BF – Pavlik et al., 2009  A’= 0.003,  A’= 0.01 – Gong et al., 2010  A’= 0.005 – Pardos et al., 2011  RMSE= 0.005 – Gowda et al., 2011  A’= 0.02 BF better than EM – Pavlik et al., 2009  A’= 0.01,  A’= 0.005 – Baker et al., 2011  A’= 0.001 BF better than CGD – Baker et al., 2010  A’= 0.02

41. Maybe a slight advantage for EM The differences are tiny

42. Model Degeneracy (Baker, Corbett, & Aleven, 2008)

43. Conceptual Idea Behind Knowledge Tracing Knowing a skill generally leads to correct performance Correct performance implies that a student knows the relevant skill Hence, by looking at whether a student’s performance is correct, we can infer whether they know the skill

44. Essentially A knowledge model is degenerate when it violates this idea When knowing a skill leads to worse performance When getting a skill wrong means you know it

45. Theoretical Degeneracy P(S)>0.5 – A student who knows a skill is more likely to get a wrong answer than a correct answer P(G)>0.5 – A student who does not know a skill is more likely to get a correct answer than a wrong answer

46. Empirical Degeneracy Actual behavior by a model that violates the link between knowledge and performance

47. Empirical Degeneracy: Test 1 (Concrete Version) (Abstract version given in paper) If a student’s first 3 actions in the tutor are correct The model’s estimated probability that the student knows the skill Should be higher than before these 3 actions.

48. Test 1 Passed P(L 0 )= 0.2 Bob gets his first three actions right P(L 3 )= 0.4

49. Test 1 Failed P(L 0 )= 0.2 Maria gets her first three actions right P(L 3 )= 0.1

50. Empirical Degeneracy: Test 2 (Concrete Version) (Abstract version in paper) If the student makes 10 correct responses in a row The model should assess that the student has mastered the skill

51. Test 2 Passed P(L 0 )= 0.2 Teresa gets her first seven actions right P(L 7 )= 0.98 The system assesses mastery and moves Teresa on to new material

52. Test 2 Failed P(L 0 )= 0.2 Ido gets his first ten actions right P(L 10 )= 0.44 Over-practice for Ido

53. Test 2 Really Failed P(L 0 )= 0.2 Elmo gets his first ten actions right P(L 10 )= 0.42 Elmo gets his next 300 actions right P(L 310 )= 0.42

54. Test 2 Really Failed P(L 0 )= 0.2 Elmo gets his first ten actions right P(L 10 )= 0.42 Elmo gets his next 300 actions right P(L 310 )= 0.42 Elmo’s school quits using the tutor

55. Model Degeneracy Joe Beck has told me in personal communication that he has an alternate definition of Model Degeneracy that he prefers I don’t know which paper it’s in, and could not find it before class – I emailed him, and will let you know what he says

56. Extensions There have been many extensions to BKT We will discuss some of the most important ones in class on February 8

57. BKT.vs. PFA Should we use BKT or PFA within educational software? Why? What situations might BKT be better for? What situations might PFA be better for?

58. BKT Questions? Comments?

59. Next Class Wednesday, February 1 3pm-5pm AK232 Diagnostic Metrics: ROC, A', AUC, Precision, Recall, LL, BiC Fogarty, J., Baker, R., Hudson, S. (2005) Case Studies in the use of ROC Curve Analysis for Sensor-Based Estimates in Human Computer Interaction. Proceedings of Graphics Interface (GI 2005), 129-136. Davis, J., Goadrich, M. (2006) The relationship between Precision- Recall and ROC curves. Proceedings of the 23rd international conference on Machine learning. Russell, S., Norvig, P. (2010) Artificial Intelligence: A Modern Approach. Ch. 20: Learning Probabilistic Models.

60. The End