Presented by Dr Safeera Hussainy OSCEology A primer on performance-based teaching, learning & assessment in pharmacy

Introduction ‘Worried, can’t wait’ role-play –Competent or not –Analytical checklist –Cut-score

Background Need to assess clinical skills –Not written exams –Performance-based assessment Traditional oral exam Appropriate use of PBA –Communication vs. Knowledge assessment

The OSCE –Performance in ‘real-world’ –Objective >Performance criteria pre-defined (checklist) –Structured –Content specificity issue >Multiple stations (improves reliability) –Reliable & valid (blueprint, peer-review)

Costly & complex OSCEs in Pharmacy in Canada –Teaching & assessment –High-stakes licensure perspective –Pre-internship

Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

Blueprinting What is it? –Content & process objectives “What do we expect of students?” (outcomes) The minimally competent student Ensures not just testing “niceness” –Pharmacy specific Validity built-in

Validation of blueprint –Face, content & ecological validity >Have 3 separate groups –Criticality, frequency & relevancy –To ensure buy-in & acceptance –Effort depends on stakes

Three separate groups A.Case writing B.Case reviewing & validation C.Standard setting Required for high-stakes exams

Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

Case writing- Station definition Blueprint to define station type –Content area (e.g. Cardiology) –Process skill (e.g. Age, ESL, lifestyle choices)

Things to consider: –“What should a minimally competent student be able to do?” –Blueprint, real-world, ethics, confirm correctness Do not: –Atypical, ambiguous, overloaded with details, based on lies

Objectives of station development –Realistic (base on real patient) –Relevant –Ecologically valid (blueprinting) –Focused (7 minutes) –Observable –Measurable –“One right answer”

Case writing template –The problem –The prompt/stem that the student reads outside –Background for SP –Time prompts for SP (e.g. if after 5 minutes the student has not asked about allergies, say “I am allergic to aspirin, will that be a problem?”)

Analytical checklists –11-17 observable ‘yes/no’ items –Must start with an action verb –Must be very specific (& case-specific) >‘and’ vs. ‘or’ >‘e.g.’ vs. ‘i.e.’ –Must provide training for use –Examiner must complete in real-time –High inter-rater reliability

Global rating scales –For communication/interpersonal skills –Generic scale for all cases –Examiner completes scale –Must provide training for use –Overall presentation: How satisfied does the patient appear at the end of the encounter?

Principles of effective case development –Realism! –Keep analytical checklist in mind –Aim for optimal number of checklist items, divided amongst relevant domains.

Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

Before reading –Volunteer leaves –Group reads/discusses –Role-play –Use checklist

Ensure relevance Ensure feasibility Modify case (including SP background information) & checklist Must pass the “reasonable pharmacist” test (average, not minimally competent) Second role-play (real SP if possible)

Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

Unable to change case, weighting only. Decide minimum performance level (for case or items) Quantify competence; > cut score. Many methods, no gold-standard. Subjective/judgment-based, many opinions. Calculate “cut-off” score, decide global vs. analytical weighting (high-stakes)

Defining minimum performance levels –Criticality/relevancy matrix method –The minimally competent student approach >Out of 100 minimally competent students, how many of them will actually get this item? –Decide individually, compare, re-adjust –Role-playing helpful –Compare scores between both methods

Criticality/relevancy matrix Low Criticality Moderate Criticality High Criticality Low Relevancy Moderate Relevancy High Relevancy

Problems –Criticality/relevancy matrix method >No “Must-have” items >Can “Pass” without solving –The minimally competent student method >May require more interpretation >Reduces inter-rater reliability >Is increasingly being used

Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

Training Standardised Patients Actors or lay people Standardized vs. Simulated Assessment (Rating scale vs. checklist) Feedback (teaching purposes) Effective use of SPs –Use in teaching before assessment (TOSCE) –Become the role

Clear case, thorough background Avoid jargon/provide definitions Clear checklist Train to checklist (e.g. one answer for one question) Role-play, correct problems in real-time. Group training (for consistent role-portrayal)

Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

Rooms Prompts Reference books Props Scoring sheets Timers Clear signage

Running the OSCE Exam site staffing –site administrator, chief examiner (pharmacist), SP trainer(s), hall monitors, track co-ordinators (pharmacist), timers, runners Communication & authority “Back up” for everything Training assessors (blinded to item weightings & cut-score) Brief students (explain timing)

Summary Costly, time-intensive & complex. Planning & organisation essential High stakes vs. Low stakes –time, effort, number of stations & security To fairly assess clinical competence in a realistic setting

OSCEs in the Monash BPharm course PAC2331 – Pharmacists as Communicators: –2 stations >Community >Hospital –Communication skills only assessed –Global Rating Scale only used

PAC2412 – Integrated Therapeutics – Cardiovascular: –3 stations >BP measurement >HT counselling in context of overall CV risk >Facilitating behaviour change –Clinical knowledge & communication/interpersonal & problem-solving skills assessed –Modified Analytical checklist only used

PAC3362 – Context in Practice 2 –4 stations >Extemporaneous preparation >Drug information query >Communication with HCP/patient >Medication reconciliation –Clinical knowledge & communication/interpersonal & problem-solving skills assessed –Assessment method TBC

PAC2331 Evaluation (2012) Objectives were clearly defined for each OSCE station 38.6% 7.1% 4.3%

Cases examined in each OSCE station fairly tested lecture & tutorial material 17.6%

I felt prepared for the OSCE after attending the Teaching OSCE (TOSCE) 34.3% 17.9% 6%

Use of the OSCE assessment method is significantly better than written assessment to help develop my competency for practice 51.5%

PAC2412 Evaluation (2011) Objectives were clearly defined for each OSCE station

Scenarios presented for each OSCE station were a fair representation of challenges faced in practice

Use of the OSCE assessment method is significantly better than written assessment to help develop my competency for practice.