Assessment David Taylor

Last session’s evaluation Was it worth it? ROI Did it make a difference? Results Did they use anything they had learnt? Impact Did they learn anything? Learning Did they enjoy it? Satisfaction

Bloom’s Taxonomy Creating Evaluating Analysing Applying Understanding evaluation Evaluating synthesis Analysing analysis Applying application Understanding comprehension Remembering knowledge

Miller’s Pyramid Action Does Performance Shows how Competence Knows how Knowledge Knows

Dreyfus & Dreyfus proposed a 5 stage model Novice Advanced beginner Competent Proficient Expert Strictly adheres to rules Limited situational perception Copes with crowdedness Plans… Decision making Holistic view, prioritizes Transcends reliance on rules Intuitive

The 7 stage model Student Novice Advanced beginner Competent Proficient Expert Master We should be expecting our students to be reaching advanced beginner to competent status by the time of their assessment Student Novice Advanced beginner Competent Proficient Expert Master

Key paper

Many different types of assessment Multiple choice questions cloze Extended matching items Essay Portfolios Short answer OSCE Mini CEX Case based discussion OSTE DOPS Long case OSLER Dissertation WBA OSPE oral

Miller’s Pyramid Observation, portfolios, logs, peer observation Does OSCE Shows how MCQ type A or R Knows how MCQ type A Knows

The five targets Feasibility Impact Reliability Acceptability Validity After Van der Vleuten Feasibility Impact Reliability Acceptability Validity

Which is most important? Impact Validity Reliability Feasibility Acceptability It is usually a trade-off

Key paper

Three elements MCQs Subject/Level The Stem - A question – either long or short Between 5 and 7 possible answers

And which learning outcome you are testing! Subject/level Before you start, state what level of student will be answering the question And which learning outcome you are testing!

Key paper

The stem – a statement Azer 2003 Reflects a single task to the examinee (a micro-skill) Aims at testing “rote” learning Encourages recall of memorized facts Does not allow integration of disciplines Suits a traditional curriculum

The stem – a scenario Azer 2003 Carefully written questions stimulate a number of higher levels of thinking, e.g. hypotheses generation, making decisions on the basis of evidence, and critical thinking Allows a thinking process Matches with real-life situations and applications of basic sciences in a clinical format Allows integration Matches with students’ expectations in a PBL curriculum

Clues to the correct answer Abbreviations, eponyms, synonyms Stems - avoid Double-negatives Clues to the correct answer Abbreviations, eponyms, synonyms Vague terms (sometimes, often, occasionally)

Key paper

In alphabetical or other systematic order Each of similar length The items Between 5 and 7 items In alphabetical or other systematic order Each of similar length Related to the learning objectives/outcomes! Distractors should be potential answers

Always ask a colleague (or several) to check the question/answers The whole thing… Always ask a colleague (or several) to check the question/answers Then compile the paper and check that there are not clues to the right answers elsewhere And after the paper has been taken, analyse the individual answers

In your groups construct one (or two) MCQs. So now In your groups construct one (or two) MCQs. And be prepared to share them with the rest of the course

Three elements MCQs Subject/Level The Stem - A question – either long or short Between 5 and 7 possible answers

Discussion of the different MCQs Main observations …