1. Detecting the Learning Value of Items In a Randomized Problem Set
Zachary A. Pardos, Neil T. Heffernan
Worcester Polytechnic Institute
Department of Computer Science

2. The Problem What is the learning value of content in ITS?
- does it promote learning?
Ways to find out:
Run an RCE
Data mine responses (using this method)
100s of Items of learning content
1000s of students’ responses
Pile of learning content (100s of assistments)
Created by IQP students, Cris, Leena, me, not teachers in schools though
Is it any good? It could all be great or all terrible, we are looking for what is the best content relative to the rest
What content is the top of the heap? (conical two triangles example)

3. Dataset
Student main problem responses (correct/incorrect) to 25 problem sets of 2,3 and 4 questions
Questions within a problem set relate to the same skill
students completed each problem set in school year data
2,400 students total with 54,000 responses (14-16 year olds)
Questions in the problem sets were presented in a randomized order (required for this analysis)
Main problem
hint

4. Confound
Since only main question responses are being analyzed, the learning from the main question is confounded with the learning from the scaffolding and hints of the problem.
learning could be attributed to
The immediate feedback to the main problem of question 1
The scaffolding of question 1
Applying concepts from question 1 the next main problem
Main problem
hint

5. Model S S S Parameters can be learned with the EM algorithm! .. ?
Modeling or measuring learning requires modeling knowledge
Knowledge Tracing used to model learning
Parameters
(probability of learning)
(guess/slip)
P(Skill: 0 → 1)
P(Skill: 0 → 1) S S S
Latent
(skill knowledge)
(dichotomous)
P(correct| Skill = 0)
P(incorrect| Skill = 1)
Observables
(question answers)
incorrect
correct
correct

6. Model
Knowledge tracing assumes that learning rate is the same between each opportunity
Our model associates the learning rate with the particular problem that was encountered
Knowledge Tracing
Learning rate between opportunities are the same regardless of which problem the student saw
0.12
0.12
Our Item Effect Model
Learning rates are an attribute of specific problems
Learning rates must be associated with problem for all permutations.
0.11
0.15

7. Model
Three question sequence permutations modeled with shared Bayesian parameters
Also known as Equivalence classes of CPTs (conditional probability tables )

8. Reliability measure
Data for a problem set randomly split into 20 equal size bins by student
Each bin was evaluated separately by the model
Binomial test used to estimate the probability of the null hypothesis, that each item is equally likely to have the highest learning rate
ie: binopdf(best_choice_mode,20,0.25)
Item learning rates 1 2 3 4
Split 1
0.0732
0.0267
0.0837
0.0701
...
Split 20
0.0849
0.0512
0.0550
0.0710

9. Method Application Which one is BEST?
Compute the learning rates of the three questions in the problem set
Which one is BEST?
Definition of BEST in this analysis: The question in a problem set that has the highest probability of learning.

10. Results
Problem sets with four questions were analyzed and the parameters of prior, guess/slip and learning rates were learned using the described method
The question with the highest probability of learning was identified
Problem set
Number of users
Best question
p value
prior
q1 rate
q2 rate
q3 rate
q4 rate 16
800 2
0.0652
0.6738
0.1100
0.1115
0.1017
0.1011 11
560 4
0.0170
0.5909
0.0958
0.0916
0.0930
0.1039 14
480 3
0.6499
0.1365
0.0977
0.1169
0.1063 25
440 1
0.7821
0.1392
0.0848
0.1157
0.1242
282
220
0.0039
0.7365
0.1574
0.0999
0.0991
0.1004 33
200
0.4394
0.7205
0.1124
0.1028
0.1237
0.1225 39
160
0.6180
0.0853
0.1192
0.1015
0.0819
Method needs validation. Another method may report different results and reliability.
Ground truth of the parameters is necessary to validate the method and results

11. Simulation Validation
Since ground truth of learning rates in the real world are impossible to know, a simulation study was run
The simulation set a variety of values for the parameters of prior, guess/slip and learning rates and then simulated user responses
These responses could then be analyzed by the method using the same technique as was used on real data
An error analysis could done since the underlying simulation parameters of the data were known (did the method pick the right best question?)
Opportunity to learn what the method can & can’t do

12. Simulation Results More students increases chance of a result
Larger learning difference between questions also increases the change of a result
Of the 160 experiments evaluated, 89 were reported as reliable (56%)
Of the 89 reported reliable results (using p < 0.05), seven were incorrect (7.8% FP)

13. Limitations
Only problem sets of five questions or less can be reasonably evaluated
Larger problem sets become intractable to compute due to the exponential increase in nodes and permutations as question count increases
for a four question set (4+4)*24 = 192 nodes
for a five question set (5+5) *120 = 1,200 nodes
Possible optimization is to only model the sequences for which there is data
Randomization of question order must be present to control for factors including problem difficulty and allow for detecting learning rates of all item pairs in the problem set

14. Contribution No methods previous to this
Estimate learning rates per problem
Allows for the best (and worse) content to be identified without RCEs
Extends knowledge tracing to support randomization of problem order
Uses permutations of sequences to estimate stable Bayesian parameters with EM

15. Conclusions & Future Work
We think that this method, and ones built off of it, will facilitate better tutoring systems
Randomization gives many of the properties of a RCE. This method can perform a similar function but in the form of data mining to find what content works best
Method could be applied to aid in improving accuracy of question skill tagging