1. Addressing the Assessing Challenge with the ASSISTment System
Mingyu Feng, Worcester Polytechnic Institute (WPI)

2. The “ASSISTment” Project
Teachers are being asked to use formative assessment data to inform their instructions
The ASSISTment project integrates assistance & assessment
The original question
a. Congruence
b. Perimeter
c. Equation-Solving
The 1st scaffolding question
Congruence
Main goals:
Help student Learning [1][2][3]
Assess precisely and present to teachers
The 2nd scaffolding question
Perimeter
A buggy message
A hint message

3. Online Report [4] How do we improve the prediction?
the “Grade book” report
The prediction of score is primitive
made totally based on percent correct on original questions.
Big offset from real MCAS score
avg. 7.5 points ( )
How do we improve the prediction?

4. Thoughts on Improving our Prediction
What if we take into consideration of performance on scaffolding questions?
What if we assess student longitudinally over time?
Questions have been tagged with skills in different skill models. How do we take advantage of that?
Do students learn on different skills at the same rate?
Can we make better prediction by incorporating skills?
Which skill model did better on predicting students’ score?
Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

5. Thoughts on Improving our Prediction
What if we take into consideration of performance on scaffolding questions?
What if we assess student longitudinally over time?
Questions have been tagged with skills in different skill models. How do we take advantage of that?
Do students learn on different skills at the same rate?
Can we make better prediction by incorporating skills?
Which skill model did better on predicting students’ score?
Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

6. Research Questions
Can we do a more accurate job of predicting student's MCAS score using the online assistance information (concerning time, performance on scaffoldings, #attempt, #hint)?
Yes!
the more hint, attempt, time a student need to solve a problem, the worse his predicted score would be. [Feng, Heffernan & Koedinger, 2006a]

7. Thoughts on Improving our Prediction
What if we take into consideration of performance on scaffolding questions?
What if we assess student longitudinally over time?
Questions have been tagged with skills in different skill models. How do we take advantage of that?
Do students learn on different skills at the same rate?
Can we make better prediction by incorporating skills?
Which skill model did better on predicting students’ score?
Difficulty of questions differ. Will our prediction be improved if we take into consideration of difficulty of each single question?

8. Research Questions Track students’ learning longitudinally
Can our system detect performance improving over time?
Can we tell the difference on learning rate of students from different schools? Teacher?
Can we do a better job of predicting students’ performance by modeling longitudinally?

9. 17 Student from one class predicted-score (Y-Axis) over months (X Axis)

11. Research Questions Track learning longitudinally
Can our system detect performance improving over time?
Can we tell the difference on learning rate of students from different schools? Teacher?
Can we do a better job of predicting students’ performance by modeling longitudinally?
Yes, we can. (Feng, Heffernan & Koedinger, 2006a, 2006b )

12. Thoughts on Improving our Prediction
What if we take into consideration of performance on scaffolding questions?
What if we assess student longitudinally over time?
Questions have been tagged with skills in different skill models. How do we take advantage of that?
Do students learn on different skills at the same rate?
Can we make better prediction by incorporating skills?
Which skill model did better on predicting students’ score?
Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

13. Skill Models

14. Research Questions
Can we track the learning of different skills?

16. Research Questions Can we track the learning of different skills?
Yes, we can!
In (Feng, Heffernan & Koedinger, 2006a), we constructed models with 5 skills and showed that students started as being comparably good at Algebra while bad at Measurement. They were learning Data Analysis approximately at the same level of speed as Number Sense, significantly faster than learning other skills.

17. Research Questions
Can we improve our prediction of student test scores by introducing skills as predictor in our models?
The grain sizes of the skills models differ. How does the finer-grained skill model (WPI-78) do on estimating test scores comparing to the coarse skill?
Similar work has been done in (Pardos et al., 2006) using Bayesian Network approach.

18. Data Item level online data
students’ binary response (1/0) to items that are tagged in different skill models

19. Approach
Fit mixed-effects logistic regression model on the longitudinal online data
using skills as a factor
predicting prob(response=1) on an item tagged with certain skill at certain time
The fitted model gives learning parameters (initial knowledge + learning rate) of each skill of individual student
All questions in the external test have also been tagged in all skill models. So we can apply the fitted model to get predicted test scores.

20. Result > Is 12.12% any good for assessment purpose?
Real
MCAS
score
Assistment Predicted Score
Skill Models
WPI-1
WPI-5
WPI-78
Mary 29
28.59
27.65
27.05
Tom 28
27.58
26.43
25.35 …
Sue 25
26.56
24.94
24.10
Dick 22
23.70
22.78
21.31
Harry 33
27.54
26.37
28.12
Absolute Difference between Real Score and Assistment Predicted Score
WPI-1
WPI-5
WPI-78
0.41
1.35
1.95
0.42
1.57
2.65 …
1.56
0.06
0.90
1.70
0.78
0.69
5.46
6.63
4.88
MAD
4.552
4.343
4.121
%Error
13.39%
12.77%
12.12%
>
P-values of both Paired t-tests are below 0.05
Is 12.12% any good for assessment purpose?
MCAS-simulation result: 11.12%

21. Thoughts on Improving our Prediction
What if we take into consideration of performance on scaffolding questions?
What if we assess student longitudinally over time?
Questions have been tagged with skills in different skill models. How do we take advantage of that?
Do students learn on different skills at the same rate?
Can we make better prediction by incorporating skills?
Which skill model did better on predicting students’ score?
Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

22. Research Question
Does introducing item difficulty information help to build a better predictive model on top of skills?

23. Approach Getting difficulty parameters
Fit one-parameter logistic (1PL) IRT model (also referred to as Rasch model) on our online data (from a different group of students)
the dependent variable: probability of correct response for a particular person to a specified item.
The independent variables: the person’s trait score and the item’s difficulty level .
was used as a covariate in the mixed-effects logistic regression models

24. Results
By introducing as a predicting factor in addition to skills, we can construct statistically significantly better fitted models on the online data. Yet, the prediction of the MCAS scores was not improved (Feng et al., 2006)
When using separately, skill learning tracking can better predict MCAS score than simply using item difficulty parameter (Feng & Heffernan, 2007b)

25. Summary of Approaches on Improving Assessment
What if we take into consideration of performance on scaffolding questions?
What if we assess student longitudinally over time?
Questions have been tagged with skills in different skill models. How do we take advantage of that?
Do students learn on different skills at the same rate?
Can we make better prediction by incorporating skills?
Which skill model did better on predicting students’ score?
Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

26. Reference
[1] Razzaq, L., Feng, M., Nuzzo-Jones, G., Heffernan, N.T., Koedinger, K. R., Junker, B., Ritter, S., Knight, A., Aniszczyk, C., Choksey, S., Livak, T., Mercado, E., Turner, T.E., Upalekar. R, Walonoski, J.A., Macasek. M.A., Rasmussen, K.P. (2005). The Assistment Project: Blending Assessment and Assisting. Proceedings of the 12th Artificial Intelligence In Education, Amsterdam.
[2] Razzaq, L., Heffernan, N.T. (2006). Scaffolding vs. hints in the Assistment System. In Ikeda, Ashley & Chan (Eds.). Proceedings of the Eight International Conference on Intelligent Tutoring Systems. Springer-Verlag: Berlin. pp
[3] Razzaq, L., Heffernan, N. T. (2007, in press) What level of tutor feedback is best? In Luckin & Koedinger (Eds) Proceedings of the 13th Conference on Artificial Intelligence in Education. IOS Press.
[4] Feng, M., Heffernan, N.T. (2007a). Towards Live Informing and Automatic Analyzing of Student Learning: Reporting in the Assistment System. Special Issue of Journal of Interactive Learning Research (JILR) entitled "Usage Analysis in Learning Systems: Existing Approaches and Scientific Issues", 18(2).
[5] Pardos, Z. A., Heffernan, N. T., Anderson, B. & Heffernan, C. (in press). Using Fine-Grained Skill Models to Fit Student Performance with Bayesian Networks. Workshop in Educational Data Mining held at the Eight International Conference on Intelligent Tutoring Systems. Taiwan
[6] Feng, M., Heffernan, N., Mani, M., & Heffernan C. (2006). Using Mixed-Effects Modeling to Compare Different Grain-Sized Skill Models. AAAI'06 Workshop on Educational Data Mining, Boston, 2006.
[7] Feng, M. & Heffernan, N. T. (2007b). Assessing Students?Performance: Item Difficulty Parameter vs. Skill Learning Tracking, paper to be presented at NCME Annual Conference 2007.