Addressing the Assessing Challenge with the ASSISTment System Mingyu Feng, Worcester Polytechnic Institute (WPI)

The “ASSISTment” Project Teachers are being asked to use formative assessment data to inform their instructions The ASSISTment project integrates assistance & assessment The original question a. Congruence b. Perimeter c. Equation-Solving The 1st scaffolding question Congruence Main goals: Help student Learning [1][2][3] Assess precisely and present to teachers The 2nd scaffolding question Perimeter A buggy message A hint message

Online Report [4] How do we improve the prediction? the “Grade book” report The prediction of score is primitive made totally based on percent correct on original questions. Big offset from real MCAS score avg. 7.5 points (2004-2005) How do we improve the prediction?

Thoughts on Improving our Prediction What if we take into consideration of performance on scaffolding questions? What if we assess student longitudinally over time? Questions have been tagged with skills in different skill models. How do we take advantage of that? Do students learn on different skills at the same rate? Can we make better prediction by incorporating skills? Which skill model did better on predicting students’ score? Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

Thoughts on Improving our Prediction What if we take into consideration of performance on scaffolding questions? What if we assess student longitudinally over time? Questions have been tagged with skills in different skill models. How do we take advantage of that? Do students learn on different skills at the same rate? Can we make better prediction by incorporating skills? Which skill model did better on predicting students’ score? Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

Research Questions Can we do a more accurate job of predicting student's MCAS score using the online assistance information (concerning time, performance on scaffoldings, #attempt, #hint)? Yes! the more hint, attempt, time a student need to solve a problem, the worse his predicted score would be. [Feng, Heffernan & Koedinger, 2006a]

Thoughts on Improving our Prediction What if we take into consideration of performance on scaffolding questions? What if we assess student longitudinally over time? Questions have been tagged with skills in different skill models. How do we take advantage of that? Do students learn on different skills at the same rate? Can we make better prediction by incorporating skills? Which skill model did better on predicting students’ score? Difficulty of questions differ. Will our prediction be improved if we take into consideration of difficulty of each single question?

Research Questions Track students’ learning longitudinally Can our system detect performance improving over time? Can we tell the difference on learning rate of students from different schools? Teacher? Can we do a better job of predicting students’ performance by modeling longitudinally?

17 Student from one class predicted-score (Y-Axis) over months (X Axis)

Research Questions Track learning longitudinally Can our system detect performance improving over time? Can we tell the difference on learning rate of students from different schools? Teacher? Can we do a better job of predicting students’ performance by modeling longitudinally? Yes, we can. (Feng, Heffernan & Koedinger, 2006a, 2006b )

Thoughts on Improving our Prediction What if we take into consideration of performance on scaffolding questions? What if we assess student longitudinally over time? Questions have been tagged with skills in different skill models. How do we take advantage of that? Do students learn on different skills at the same rate? Can we make better prediction by incorporating skills? Which skill model did better on predicting students’ score? Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

Skill Models

Research Questions Can we track the learning of different skills?

Research Questions Can we track the learning of different skills? Yes, we can! In (Feng, Heffernan & Koedinger, 2006a), we constructed models with 5 skills and showed that students started as being comparably good at Algebra while bad at Measurement. They were learning Data Analysis approximately at the same level of speed as Number Sense, significantly faster than learning other skills.

Research Questions Can we improve our prediction of student test scores by introducing skills as predictor in our models? The grain sizes of the skills models differ. How does the finer-grained skill model (WPI-78) do on estimating test scores comparing to the coarse skill? Similar work has been done in (Pardos et al., 2006) using Bayesian Network approach.

Data Item level online data students’ binary response (1/0) to items that are tagged in different skill models

Approach Fit mixed-effects logistic regression model on the longitudinal online data using skills as a factor predicting prob(response=1) on an item tagged with certain skill at certain time The fitted model gives learning parameters (initial knowledge + learning rate) of each skill of individual student All questions in the external test have also been tagged in all skill models. So we can apply the fitted model to get predicted test scores.

Result > Is 12.12% any good for assessment purpose? Real MCAS score Assistment Predicted Score Skill Models WPI-1 WPI-5 WPI-78 Mary 29 28.59 27.65 27.05 Tom 28 27.58 26.43 25.35 … Sue 25 26.56 24.94 24.10 Dick 22 23.70 22.78 21.31 Harry 33 27.54 26.37 28.12 Absolute Difference between Real Score and Assistment Predicted Score WPI-1 WPI-5 WPI-78 0.41 1.35 1.95 0.42 1.57 2.65 … 1.56 0.06 0.90 1.70 0.78 0.69 5.46 6.63 4.88 MAD 4.552 4.343 4.121 %Error 13.39% 12.77% 12.12% > P-values of both Paired t-tests are below 0.05 Is 12.12% any good for assessment purpose? MCAS-simulation result: 11.12%

Thoughts on Improving our Prediction What if we take into consideration of performance on scaffolding questions? What if we assess student longitudinally over time? Questions have been tagged with skills in different skill models. How do we take advantage of that? Do students learn on different skills at the same rate? Can we make better prediction by incorporating skills? Which skill model did better on predicting students’ score? Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

Research Question Does introducing item difficulty information help to build a better predictive model on top of skills?

Approach Getting difficulty parameters Fit one-parameter logistic (1PL) IRT model (also referred to as Rasch model) on our online data (from a different group of students) the dependent variable: probability of correct response for a particular person to a specified item. The independent variables: the person’s trait score and the item’s difficulty level . was used as a covariate in the mixed-effects logistic regression models

Results By introducing as a predicting factor in addition to skills, we can construct statistically significantly better fitted models on the online data. Yet, the prediction of the MCAS scores was not improved (Feng et al., 2006) When using separately, skill learning tracking can better predict MCAS score than simply using item difficulty parameter (Feng & Heffernan, 2007b)

Summary of Approaches on Improving Assessment What if we take into consideration of performance on scaffolding questions? What if we assess student longitudinally over time? Questions have been tagged with skills in different skill models. How do we take advantage of that? Do students learn on different skills at the same rate? Can we make better prediction by incorporating skills? Which skill model did better on predicting students’ score? Difficulty of questions differ. Will our prediction be improved further if we take into consideration of difficulty of each single question?

Reference [1] Razzaq, L., Feng, M., Nuzzo-Jones, G., Heffernan, N.T., Koedinger, K. R., Junker, B., Ritter, S., Knight, A., Aniszczyk, C., Choksey, S., Livak, T., Mercado, E., Turner, T.E., Upalekar. R, Walonoski, J.A., Macasek. M.A., Rasmussen, K.P. (2005). The Assistment Project: Blending Assessment and Assisting. Proceedings of the 12th Artificial Intelligence In Education, 555-562. Amsterdam. [2] Razzaq, L., Heffernan, N.T. (2006). Scaffolding vs. hints in the Assistment System. In Ikeda, Ashley & Chan (Eds.). Proceedings of the Eight International Conference on Intelligent Tutoring Systems. Springer-Verlag: Berlin. pp. 635-644. 2006. [3] Razzaq, L., Heffernan, N. T. (2007, in press) What level of tutor feedback is best? In Luckin & Koedinger (Eds) Proceedings of the 13th Conference on Artificial Intelligence in Education. IOS Press. [4] Feng, M., Heffernan, N.T. (2007a). Towards Live Informing and Automatic Analyzing of Student Learning: Reporting in the Assistment System. Special Issue of Journal of Interactive Learning Research (JILR) entitled "Usage Analysis in Learning Systems: Existing Approaches and Scientific Issues", 18(2). [5] Pardos, Z. A., Heffernan, N. T., Anderson, B. & Heffernan, C. (in press). Using Fine-Grained Skill Models to Fit Student Performance with Bayesian Networks. Workshop in Educational Data Mining held at the Eight International Conference on Intelligent Tutoring Systems. Taiwan. 2006. [6] Feng, M., Heffernan, N., Mani, M., & Heffernan C. (2006). Using Mixed-Effects Modeling to Compare Different Grain-Sized Skill Models. AAAI'06 Workshop on Educational Data Mining, Boston, 2006. [7] Feng, M. & Heffernan, N. T. (2007b). Assessing Students?Performance: Item Difficulty Parameter vs. Skill Learning Tracking, paper to be presented at NCME Annual Conference 2007.