Confidence-Based Assessment of Two-Alternative Format Tests AHMED A. BELAL (1) & DIALA F.AMMAR (2) (1)Computer Engineering and Informatics- Beirut Arab University, Lebanon (2)Psychology department, Lebanese American University, Lebanon abelal@bau.edu.lb

Computer Aided Assessment Multiple Choice Exams : No Partial Credit Penalty for Wrong Answers

Designing good distracters Multiple Select Exams Penalty for Wrong Answers Multiple T/F Exams Question may have no correct answers No partial credit Individual T/F questions Weak resistance to guessing

Correct =+ X No Answer = 0 Wrong Answer= -Y Poor discrimination Confidence-based Assessment

Uncertainty improves Discrimination The Element Of Uncertainty improves Discrimination Three Questions +10 Correct 1 2 3 Grade 10 20 30 Correct 1 2 3 Grade -15 15 30 Scaled 45 +10 -5 Correct 1 2 3 Grade -30 -10 10 30 Scaled 20 40 60 +10 -10

Adding an element of uncertainty Correct +10 Wrong -5 No answer Correct 1 2 3 Wrong No answer 9 Different Grades Grade -15 15 30 -10 5 20 10 -5 Scaled 45 35 25 Correct +10 Wrong -10 No answer Grade -30 -10 10 30 -20 20 Scaled 40 60 50 7 Different Grades

Incorporating uncertainty in the test Confidence Level Correct Answer Wrong Answer 1 +3 -1 2 +4 -2 3 +5 -5 Leclereq [1,2] Hard to correctly estimate one’s confidence level Humans are more oriented to judge their uncertainty in things relative to each other

Relative Uncertainty Student ranks questions relative to each other. Value of correct answer decreases as level of confidence decreases. Wrong answers change the confidence level. Use a step size and a reduction function to assign values for a correct answer.

Relative Uncertainty : An Example Answer Vector 000 001 010 011 100 101 110 111 R=5 5 10 15 20 30 R=10 -10/0 R=15 -20/0 -5/0 S=1

Relative Uncertainty (Cont) Confidence Level C1 C2 C3 C4 C5 Reduction Value 2 10 8 6 4 2 Reduction Value 5 5 -5 -10 10 Questions : answer vector 1101001100 Step size = 1 R =2 Score = 10+10+0+8+0+0+4+4+0+0 = 36 R = 5 Score = 10+10+5 -5-5 = 15 Step size = 2 R =2 Score = 10+10+10+8+8 = 46 R = 5 score = 10+10+10+5+5 = 40

Answering Form

Absolute (10-2p) (10-p) (10-5p) R=1;s=2 R=2;S=2 Answer Vector 16 24 32 10 37 34 1001011000 38 44 47 35 50 1101110000 25 40 15 56 52 1010111010 30 0110101100 62 64 72 76 1111001111 80 90 1111111110 66 73 45 1011111110 42 46 1011110000 18 36 20 39 1011010000 51 0111111110 60 1110011011 86 78 84 1110111111 65 1100101111 1101100000 58 67 55 70 1111011100 49 1110111100 1011111100 22 5 0100111010 41 0111110000 0101101000 27 0011100000 1 26 29 28 0110100000 1011001000 4 1110001000 0101110100 54 68 1111100110 61 1110011101 Table 1

Strategy Average Standard Deviation Absolute 37.46428571 22.57084672 10-2P 45.28571429 18.21854396 10-P 51.75 17.3901313 10-5P 29.46429 20.19989 R=1 S=2 56.14286 17.98912 R=2 S=2 54.42857 18.44597 Table 2

Thank you