Constructing a test

Aims To consider issues of: Writing assessments Blueprinting

Methods of assessment MCQsMultiple true/false Best of five Extended matching questions Short answer questionsModified essay questions Objective Structured Examination Mini-sampling examinations Prolonged observationSummative at end of year Multiple throughout years Oral examinationNon-structured vs. structured Assessment of professional behavioursUtility?

The professional progress Knows Knows how Shows

The professional progress Knows Knows how Shows Knowledge Can state how Demonstrates information is used competence MCQs MEQs MEQs EMQs EMQs EMQs Multi-station exam Multi-station exam OSCE Work–based test Multi-source data Professionalisms

Subjects DANGER

Subjects Many curricula Assessments DANGER

SubjectsAssessmentsLearning DANGER Subjects Many curricula

SubjectsAssessmentsLearning Disconnected teaching leads to disconnected assessments Leads to disconnected learning DANGER Subjects Many curricula

Future directions in assessment Testing competencies by ability to handle complex tasks van der Vleuten and Schurworth 2005 Assessment in context of ‘practice’ – work-place based assessments Rethans et al, 2002 Multiple sampling by multiple assessors Petrusa, 2002 Peer review Do OSCEs do all that you need? Professional behaviours assessed longitudinally and at many times

Methods of assessment MCQsMultiple true/false Best of five Extended matching questions Short answer questionsModified essay questions Objective Structured Examination Mini-sampling examinations Prolonged observationSummative at end of year Multiple throughout years Oral examinationNon-structured vs. structured Assessment of professional behavioursUtility?

Methods of assessment in use MCQsMultiple true/false Best of five Extended matching questions Short answer questions‘Key feature case’ Modified essay questions OSCE Mini Clinical Examination (Mini-CEX) Long caseSummative at end of year Multiple throughout years Oral examinationNon-structured vs. structured Assessment of professional behavioursMultiple throughout course

Methods of assessment in use in Sheffield Extended matching questions Modified essay questions OSCE Mini Clinical Examination (Mini-CEX) Observed long cases - multiple throughout years Assessment of professional behaviours -multiple throughout course Work-place based assessments (from 2009)

Methods of assessment in use in Sheffield Extended matching questions Modified essay questions OSCE Mini Clinical Examination (Mini-CEX) Observed long cases - multiple throughout years Assessment of professional behaviours -multiple throughout course Work-place based assessments (from 2009) All need: Resources Staff who are trained and able Exams prepared months in advance Evidence that the exams are good Student understanding Good statistics support

Validity: linking to the curriculum by blueprinting Blueprinting –aims to ensure that the defined curriculum (and only that) is assessed –ensures that a reliable sample of the curriculum is selected for testing –enables the most appropriate test methods to be used for different aspects to be tested

Assessment blueprinting Subjects/ modules taught Core problems

Assessment blueprinting Subjects/ modules taught Components to be tested Core problems

Assessment blueprinting Subjects/ modules taught Components to be tested Core problems Exam components and weighting

Assessment blueprinting Subjects/ modules taught Components to be tested Core problems Exam components and weighting Questions Balanced sampling between subjects, teaching components and exam components Marking appropriately weighted Increases validity Guides students and staff

Validity: linking to the curriculum by blueprinting Core clinical problems CVS RS GIENDO etc

Validity: linking to the curriculum by blueprinting Core clinical problems CVS RS GIENDO etc History Examination Case Management Communication skills Data interpretation

Feasibility Can the test be delivered? Time management –Blueprinting / writing questions in time –External validation of the test Educational –Can what you want to test be tested in the format proposed? Practical –Sufficient space; examiners, patients

Test security Keeping it confidential Who sees the examination paper? Reuse of previous papers/questions

Costs Human & financial Human resources –Academic and administration staff time –Simulated patients/actors Financial costs –OSCEs Manikins consumables

Information to candidates Ensure candidates know the purpose of the assessment Information about the test Results and (remediation/resit)

Feedback All assessments must lead to feedback All assessments are formative in part Without feedback, the educational loop is left open

Valid - test what they purport to test - link to the curriculum Reliable - reproducible Feasible- can be delivered Impact - preparation of students and feedback to them Leadership- with an overview Characteristics of good assessments

Question writing Link one question to one topic

Question writing Link one question to one topic Decide what you wish to test eg knowledge / reasoning / psychomotor skill / communication / response to emergencies Decide best format of question

Question writing Link one question to one topic Decide what you wish to test eg knowledge / reasoning / psychomotor skill / communication / response to emergencies Decide best format of question Write: - accurately - unambiguously and no negatives – single or double! - without trying to confuse or mislead - at the right level - to produce an exam that links directly to the curriculum - and show your work to colleagues / assessment group - with non-specialists - using the appropriate clinical context

Assessment is enhanced if it is Authentic Related directly to practice Seen by students to be fair, have a utility, be related to the curriculum and be a source of feedback Is more than an exercise in factual recall