How does item writing flaws (IWF), cognitive level and re-use of items (RI) affect the quality of multiple choice questions (MCQ) and the students’ performance? Bjørn Mørkedal, Tobias Schmidt Slørdahl, Torstein Vik

Background MCQs was introduced at our faculty in 2005 The first four year classes has 60% MCQ in the end-of-year-examination (in addition to 40% MEQ) As an on-going quality control we assessed the 460 MCQs delivered in 2008

MCQ-type used in Trondheim Vignette A-type 3-5 alternatives, 1 is the best No penalty for wrong answer Question Alternatives

Measurements 1 Evaluation Synthesis Analysis Application Understanding Cognitive level Evaluation Synthesis Analysis Application Understanding Knowledge K2 K1 Bloom’s six levels of cognition

Measurements 2 Item writing flaws (IWF) Hand cover test Convergence Longest answer best Longest answer not best Word repetition Logical clues Vague terms «Never», «Always», etc.

Measurements 3 Re-used items (RI) The faculty allows the re-use of items on a proportion of the exam Analysed objectively using plagiarism software, comparing items from 2008 with previous exams Items from previous examinations are publically available to students

Analyses of outcome Item difficulty Discrimination index (DI) the proportion of students with correct answer Discrimination index (DI) ability of an item to distinguish less well from well performing students Mean test score equivalent to item difficulty for the entire exam This information was gathered after examinations using item analysis

Results 1 Adjusted R2 of different item analyses indices in relation to performance of items   Difficulty Discriminating power R2 Re-used item 0.08 0.07 K2 0.01 Any IWF 0.00

Results 2 Proportion of re-used items Stage Items Total Proportion 1 38 120 32 % 2 27 23 % 3 12 100 12 % 4 17 14 % Overall 94 460 20 %

Results 3 Student score by stage and RI-status Total New items Re-used items Stage Score SD 1 86.8 5.6 82.2 7.5 96.8 3.5 2 78.6 9.3 74.7 10.0 92.0 9.4 3 74.8 6.3 72.4 6.9 92.4 8.0 4 76.4 7.8 8.3 86.3 8.5 Overall 79.3 76.6 8.8 91.9 P < 0.001 Score is the number of correct answers times 100 divided by number of items. SD: Standard deviation.

Conclusions IWFs and cognitive level had little effect on the students’ performance Re-use of items influenced the results significantly Students had significantly higher scores in re-used than in new items Increasing the question bank or avoiding re-use of items is important in order to maintain high quality of MCQ-examinations

Backup-slide 1 Difficulty Discriminating power R2 Re-used item 0.08   Difficulty Discriminating power R2 Re-used item 0.08 0.07 K2 0.01 Any IWF 0.00 Number of IWF Handcover Convergence Short vignette Longest answer best (strict) Longest answer not best (strict) Longest answer best (Tobias) Longest answer best (eyeballing) Longest answer not best (eyeballing) Word repeat Logical clue “Never” Vague terms

Backup-slide 2 Student score by stage and IWF-status Total IWF-items Non-IWF-items Stage Score SD 1 86.8 5.6 85.4 6.9 87.5 2 78.6 9.3 77.9 10.3 78.9 9.4 3 74.8 6.3 77.8 6.8 72.6 7.5 4 76.4 7.8 73.9 9.6 78.0 Overall 79.3 8.5 79.2 9.1

Backup-slide 3 Proportion of items with one or more item writing flaws Stage All items New items Re-used items 1 53 % 67 % 76 % 2 43 % 52 % 63 % 3 42 % 50 % 4 38 % 46 % 41 % Overall 51 %

Backup-slide 4 Proportion of items with K2 Stage Items Proportion 1 5 4 % 2 7 6 % 3 41 41 % 4 62 52 % Overall 115 25 %