1. How does item writing flaws (IWF), cognitive level and re-use of items (RI) affect the quality of multiple choice questions (MCQ) and the students’ performance?
Bjørn Mørkedal, Tobias Schmidt Slørdahl, Torstein Vik

2. Background MCQs was introduced at our faculty in 2005
The first four year classes has 60% MCQ in the end-of-year-examination (in addition to 40% MEQ)
As an on-going quality control we assessed the 460 MCQs delivered in 2008

3. MCQ-type used in Trondheim
Vignette
A-type
3-5 alternatives, 1 is the best
No penalty for wrong answer
Question
Alternatives

4. Measurements 1 Evaluation Synthesis Analysis Application Understanding
Cognitive level
Evaluation
Synthesis
Analysis
Application
Understanding
Knowledge K2 K1
Bloom’s six levels of cognition

5. Measurements 2 Item writing flaws (IWF) Hand cover test Convergence
Longest answer best
Longest answer not best
Word repetition
Logical clues
Vague terms
«Never», «Always», etc.

6. Measurements 3 Re-used items (RI)
The faculty allows the re-use of items on a proportion of the exam
Analysed objectively using plagiarism software, comparing items from 2008 with previous exams
Items from previous examinations are publically available to students

7. Analyses of outcome Item difficulty Discrimination index (DI)
the proportion of students with correct answer
Discrimination index (DI)
ability of an item to distinguish less well from well performing students
Mean test score
equivalent to item difficulty for the entire exam
This information was gathered after examinations using item analysis

8. Results 1
Adjusted R2 of different item analyses indices in relation to performance of items
Difficulty
Discriminating power R2
Re-used item
0.08
0.07 K2
0.01
Any IWF
0.00

9. Results 2 Proportion of re-used items Stage Items Total Proportion 138
120
32 % 2 27
23 % 3 12
100
12 % 4 17
14 %
Overall 94
460
20 %

10. Results 3 Student score by stage and RI-status Total New items
Re-used items
Stage
Score SD 1
86.8
5.6
82.2
7.5
96.8
3.5 2
78.6
9.3
74.7
10.0
92.0
9.4 3
74.8
6.3
72.4
6.9
92.4
8.0 4
76.4
7.8
8.3
86.3
8.5
Overall
79.3
76.6
8.8
91.9
P < 0.001
Score is the number of correct answers times 100 divided by number of items. SD: Standard deviation.

11. Conclusions
IWFs and cognitive level had little effect on the students’ performance
Re-use of items influenced the results significantly
Students had significantly higher scores in re-used than in new items
Increasing the question bank or avoiding re-use of items is important in order to maintain high quality of MCQ-examinations

13. Backup-slide 1 Difficulty Discriminating power R2 Re-used item 0.08
Difficulty
Discriminating power R2
Re-used item
0.08
0.07 K2
0.01
Any IWF
0.00
Number of IWF
Handcover
Convergence
Short vignette
Longest answer best (strict)
Longest answer not best (strict)
Longest answer best (Tobias)
Longest answer best (eyeballing)
Longest answer not best (eyeballing)
Word repeat
Logical clue
“Never”
Vague terms

14. Backup-slide 2 Student score by stage and IWF-status Total IWF-items
Non-IWF-items
Stage
Score SD 1
86.8
5.6
85.4
6.9
87.5 2
78.6
9.3
77.9
10.3
78.9
9.4 3
74.8
6.3
77.8
6.8
72.6
7.5 4
76.4
7.8
73.9
9.6
78.0
Overall
79.3
8.5
79.2
9.1

15. Backup-slide 3 Proportion of items with one or more item writing flaws
Stage
All items
New items
Re-used items 1
53 %
67 %
76 % 2
43 %
52 %
63 % 3
42 %
50 % 4
38 %
46 %
41 %
Overall
51 %

16. Backup-slide 4 Proportion of items with K2 Stage Items Proportion 1 5
4 % 2 7
6 % 3 41
41 % 4 62
52 %
Overall
115
25 %