1. Assessment
David Taylor

2. Last session’s evaluation
Was it worth it?
ROI
Did it make a difference?
Results
Did they use anything they had learnt?
Impact
Did they learn anything?
Learning
Did they enjoy it?
Satisfaction

3. Bloom’s Taxonomy Creating Evaluating Analysing Applying Understanding
evaluation
Evaluating
synthesis
Analysing
analysis
Applying
application
Understanding
comprehension
Remembering
knowledge

4. Miller’s Pyramid
Action
Does
Performance
Shows how
Competence
Knows how
Knowledge
Knows

5. Dreyfus & Dreyfus proposed a 5 stage model
Novice
Advanced beginner
Competent
Proficient
Expert
Strictly adheres to rules
Limited situational perception
Copes with crowdedness
Plans…
Decision making
Holistic view, prioritizes
Transcends reliance on rules
Intuitive

6. The 7 stage model
Student
Novice
Advanced beginner
Competent
Proficient
Expert
Master
We should be expecting our students to be reaching advanced beginner to competent status by the time of their assessment
Student
Novice
Advanced beginner
Competent
Proficient
Expert
Master

7. Key paper

8. Many different types of assessment
Multiple choice questions
cloze
Extended matching items
Essay
Portfolios
Short answer
OSCE
Mini CEX
Case based discussion
OSTE
DOPS
Long case
OSLER
Dissertation
WBA
OSPE
oral

9. Miller’s Pyramid
Observation, portfolios, logs, peer observation
Does
OSCE
Shows how
MCQ type A or R
Knows how
MCQ type A
Knows

10. The five targets Feasibility Impact Reliability Acceptability Validity
After Van der Vleuten
Feasibility
Impact
Reliability
Acceptability
Validity

11. Which is most important?
Impact
Validity
Reliability
Feasibility
Acceptability
It is usually a trade-off

12. Key paper

13. Three elements MCQs Subject/Level
The Stem - A question – either long or short
Between 5 and 7 possible answers

14. And which learning outcome you are testing!
Subject/level
Before you start, state what level of student will be answering the question
And which learning outcome you are testing!

15. Key paper

16. The stem – a statement Azer 2003
Reflects a single task to the examinee (a micro-skill)
Aims at testing “rote” learning
Encourages recall of memorized facts
Does not allow integration of disciplines
Suits a traditional curriculum

17. The stem – a scenario Azer 2003
Carefully written questions stimulate a number of higher levels of thinking, e.g. hypotheses generation, making decisions on the basis of evidence, and critical thinking
Allows a thinking process
Matches with real-life situations and applications of basic sciences in a clinical format
Allows integration
Matches with students’ expectations in a PBL curriculum

18. Clues to the correct answer Abbreviations, eponyms, synonyms
Stems - avoid
Double-negatives
Clues to the correct answer
Abbreviations, eponyms, synonyms
Vague terms (sometimes, often, occasionally)

19. Key paper

20. In alphabetical or other systematic order Each of similar length
The items
Between 5 and 7 items
In alphabetical or other systematic order
Each of similar length
Related to the learning objectives/outcomes!
Distractors should be potential answers

21. Always ask a colleague (or several) to check the question/answers
The whole thing…
Always ask a colleague (or several) to check the question/answers
Then compile the paper and check that there are not clues to the right answers elsewhere
And after the paper has been taken, analyse the individual answers

22. In your groups construct one (or two) MCQs.
So now
In your groups construct one (or two) MCQs.
And be prepared to share them with the rest of the course

23. Three elements MCQs Subject/Level
The Stem - A question – either long or short
Between 5 and 7 possible answers

24. Discussion of the different MCQs
Main observations …