1. Comparing Decision Rules Decision accuracy of different decision rules combining multiple measures in a higher educational context Iris Yocarini, Samantha Bouwmeester, Guus Smeets, and Lidia Arends CEMO conference standard setting 23rd september 2015

2. The decision to be made End of first bachelor year Start of first bachelor year Student to second bachelor year Student leaves bachelor program BSA decision

3. Decision accuracy Given high stakes an accurate decision is required Comparison decision based on true score vs. observed score True Score Observed score Error

4. Decision accuracy Given high stakes an accurate decision is required Comparison decision based on true score vs. observed score Decision based on true score FailPass Decision based on observed score FailCorrect classificationMisclassification False negative PassMisclassification False positive Correct classification

5. Decision accuracy Total proportion of misclassifications (C + B / total sample) Decision based on true score FailPass Decision based on observed score FailCorrect classification AMisclassification False negative B PassMisclassification False positive CCorrect classification D

6. Decision accuracy Total proportion of misclassifications (C + B / total sample) False negative rate from all truly competent students those who are identified as fails (B/B+D) Decision based on true score FailPass Decision based on observed score FailCorrect classification AMisclassification False negative B PassMisclassification False positive CCorrect classification D

7. Decision accuracy Decision based on true score FailPass Decision based on observed score FailCorrect classification AMisclassification False negative B PassMisclassification False positive CCorrect classification D Total proportion of misclassifications (C + B / total sample) False negative rate from all truly competent students those who are identified as fails (B/B+D) False positive rate from all truly failing students those who are identified as passes (C/A+C)

8. Decision accuracy Decision based on true score FailPass Decision based on observed score FailCorrect classification AMisclassification False negative B PassMisclassification False positive CCorrect classification D Total proportion of misclassifications (C + B / total sample) False negative rate from all truly competent students those who are identified as fails (B/B+D) False positive rate from all truly failing students those who are identified as passes (A/A+C) Positive predictive value from all students who passed those who are correctly classified (D/C+D)

9. Testing system Compensatory testing system at Erasmus University Rotterdam Vs. standard conjunctive testing system in Dutch higher education Debate

10. Reasons behind implementation Educational views Psychometric argument Classical Test Theory (CTT): average more reliable  Assumption of parallel tests Equal true ability levels Similar test reliabilities

11. Factors influencing decision accuracy Reliability Decision accuracy True Score Observed score Error Decision based on true score FailPass Decision based on observed score FailCorrect classificationMisclassification False negative PassMisclassification False positive Correct classification Error

12. Decision rules in practice Educational setting: combinatory decision rules Compensatory aspect: required GPA Conjunctive aspect: required minimum grade Clusters First year psychology at Erasmus University Grading scale: 1.0 – 10.0 GPA: 6.0 Minimum grade: 4.0 Two clusters with each 8 courses

13. Our study Aim of study Comparing decision accuracy different decision rules that combine multiple tests Evaluating psychometric argument for implementation compensatory testing system CTT: average grade is more reliable than using individual test scores Context of first year Psychology students at Erasmus University Rotterdam

14. Decision rules Varying Conjunctive aspect: minimum required grade Compensatory aspect: required GPA Also included Fully conjunctive rule Fully compensatory rule Decision ruleMinimum gradeGPA Fully Conjunctive5.5 Fully Compensatory1.05.5 / 6.0 / 6.5 Complex rules3.0 / 4.0 / 5.05.5 / 6.0 / 6.5 *Grading from 1.0 to 10.0

15. Simulation Simulation study Manipulation of factors Decision based on true score FailPass Decision based on observed score FailCorrect classificationMisclassification False negative Pas s Misclassification False positive Correct classification

16. Results – minimum grade & GPA Minimum grade 1.0/ 3.0/ 4.0/ 5.0 GPA 5.5/ 6.0/ 6.5

17. Results – minimum grade & GPA

20. Results – average test reliability Proportion of Misclassifications

21. Results – average test reliability Positive Predictive Value

22. Results – number of retakes Proportion of Misclassifications

23. Results – number of retakes False Negative Rate

24. Results – number of retakes False Positive Rate

25. Results – number of retakes Positive Predictive Value

26. Comparison conjunctive & compensatory In compensatory decision rule: Fewer classification errors Fewer false negatives, more false positives Positive predictive value higher

27. Conclusion Increasing the degree of compensation results in less classification errors Within compensatory decision rule relatively fewer false negatives and more false positives Depends on specific setting & tests used Most important: test reliability and number of retakes Psychometric argument Standard setting

28. Take home message Decision accuracy important consideration Focus on both specific decision rule as well as tests

29. Thank you for your attention! Questions? yocarini@fsw.eur.nl

31. Results – proportion of misclassifications Decision Rule GPA Minimum Mean Proportion Errors Average Test Correlation Average Test Reliability Number of Tests Number of Retakes.1.3.5.7.4.6.881202 15.5.18.16.18.19.24.18.12.19.17.20.16 25.51.05.03.04.05.06.04.03.05.04.06.03 35.53.08.09.08.07.14.07.04.08.09.13.04 45.54.17.20.18.16.13.27.16.08.15.19.26.08 55.55.22.23.22.21.31.22.14.22.23.28.16 661.09.10.09.13.09.06.10.09.10.08 763.11.13.11.10.09.16.10.06.11.14.08 864.16.20.17.14.12.24.15.08.15.17.22.10 965.21.23.22.21.19.29.21.13.20.22.27.15 106.51.13.17.13.12.10.18.13.08.14.12.13 116.53.13.17.14.12.10.19.13.08.14.13 126.54.15.19.15.13.11.21.14.09.15.14.16.13 136.55.17.20.18.16.14.24.17.11.17.20.14

32. Results - sensitivity Decision Rule GPA Minimum Mean Sensitivity Average Test Correlation Average Test Reliability Number of Tests Number of Retakes.1.3.5.7.4.6.881202 15.5.60.52.59.64.67.45.60.76.65.56.44.77 25.51.97.98.97.96.98.99.97.98.96.99 35.53.93.91.92.93.95.87.94.98.94.92.88.98 45.54.83.79.81.84.87.71.84.93.85.80.71.94 55.55.67.61.66.69.72.52.68.82.71.63.51.83 661.95.94.95.93.95.97.94.95.92.98 763.92.90.91.93.94.87.93.96.92.87.97 864.85.80.83.87.90.74.86.93.87.83.75.95 965.68.61.67.71.75.53.69.83.73.64.52.84 106.51.92.89.91.93.94.89.92.94.91.92.88.96 116.53.90.86.90.92.93.86.91.94.90.85.95 126.54.86.80.85.88.91.78.87.93.87.85.78.94 136.55.73.64.71.76.81.59.74.86.77.69.58.88

33. Results - specificity Decision Rule GPA Minimum Mean Specificity Average Test Correlation Average Test Reliability Number of Tests Number of Retakes.1.3.5.7.4.6.881202 15.5.93.92.93.94.93.92.93.91.94.96.89 25.51.67.57.66.71.74.58.67.77.66.69.76.58 35.53.72.66.72.75.77.69.71.77.70.75.82.63 45.54.80.75.79.83.85.82.79.81.78.83.87.74 55.55.89.86.88.90.92.90.88.89.87.91.93.85 661.73.65.73.77.79.64.73.82.72.75.81.65 763.75.68.75.78.80.69.74.83.73.77.84.66 864.80.75.80.82.83.78.84.78.82.88.72 965.89.86.88.90.91.90.88.89.87.91.93.84 106.51.80.74.80.83.84.72.80.88.79.82.87.74 116.53.81.75.80.83.84.73.81.88.79.82.87.74 126.54.83.78.82.85.86.78.82.89.81.85.90.76 136.55.88.87.88.89.90.88.87.90.86.91.94.83

34. Results – positive predictive value Decision Rule GPA Minimum Mean Positive Predictive Value Average Test Correlation Average Test Reliability Number of Tests Number of Retakes.1.3.5.7.4.6.881202 15.5.82.68.80.88.93.79.82.86.83.81.79.86 25.51.98.99.98.97.96.97.98.97.98.97 35.53.98.99.98.97.98.97 45.54.97.96.97.98.97 55.55.90.82.88.93.96.88.89.91.89.87.92 661.93.95.93.91.93.96.93.94.93 763.94.95.94.93.92.94.96.93.94.95.93 864.94.93.95.93.94.93 965.89.81.88.92.95.88.89.91.90.89.87.91 106.51.85.82.85.87.88.80.85.91.85.86.87.84 116.53.86.82.85.87.88.81.85.91.85.86.87.84 126.54.86.82.86.87.88.82.86.91.85.87.88.85 136.55.85.77.85.89.91.83.84.89.85.86.85

35. Results – average test reliability False Negative Rate

36. Results – average test reliability False Positive Rate

37. Previous studies Douglas & Mislevy (2010) Van Rijn, Béguin, & Verstralen (2012) McBee, Peters, & Waterman (2014 )