1. www.pharm.monash.edu Presented by Dr Safeera Hussainy OSCEology A primer on performance-based teaching, learning & assessment in pharmacy

2. www.pharm.monash.edu Introduction ‘Worried, can’t wait’ role-play –Competent or not –Analytical checklist –Cut-score

3. www.pharm.monash.edu Background Need to assess clinical skills –Not written exams –Performance-based assessment Traditional oral exam Appropriate use of PBA –Communication vs. Knowledge assessment

4. www.pharm.monash.edu The OSCE –Performance in ‘real-world’ –Objective >Performance criteria pre-defined (checklist) –Structured –Content specificity issue >Multiple stations (improves reliability) –Reliable & valid (blueprint, peer-review)

5. www.pharm.monash.edu Costly & complex OSCEs in Pharmacy in Canada –Teaching & assessment –High-stakes licensure perspective –Pre-internship

6. www.pharm.monash.edu Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

7. www.pharm.monash.edu Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

8. www.pharm.monash.edu Blueprinting What is it? –Content & process objectives “What do we expect of students?” (outcomes) The minimally competent student Ensures not just testing “niceness” –Pharmacy specific Validity built-in

9. www.pharm.monash.edu Validation of blueprint –Face, content & ecological validity >Have 3 separate groups –Criticality, frequency & relevancy –To ensure buy-in & acceptance –Effort depends on stakes

10. www.pharm.monash.edu Three separate groups A.Case writing B.Case reviewing & validation C.Standard setting Required for high-stakes exams

11. www.pharm.monash.edu Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

12. www.pharm.monash.edu Case writing- Station definition Blueprint to define station type –Content area (e.g. Cardiology) –Process skill (e.g. Age, ESL, lifestyle choices)

13. www.pharm.monash.edu Things to consider: –“What should a minimally competent student be able to do?” –Blueprint, real-world, ethics, confirm correctness Do not: –Atypical, ambiguous, overloaded with details, based on lies

14. www.pharm.monash.edu Objectives of station development –Realistic (base on real patient) –Relevant –Ecologically valid (blueprinting) –Focused (7 minutes) –Observable –Measurable –“One right answer”

15. www.pharm.monash.edu Case writing template –The problem –The prompt/stem that the student reads outside –Background for SP –Time prompts for SP (e.g. if after 5 minutes the student has not asked about allergies, say “I am allergic to aspirin, will that be a problem?”)

17. www.pharm.monash.edu Analytical checklists –11-17 observable ‘yes/no’ items –Must start with an action verb –Must be very specific (& case-specific) >‘and’ vs. ‘or’ >‘e.g.’ vs. ‘i.e.’ –Must provide training for use –Examiner must complete in real-time –High inter-rater reliability

18. www.pharm.monash.edu

19. Global rating scales –For communication/interpersonal skills –Generic scale for all cases –Examiner completes scale –Must provide training for use –Overall presentation: How satisfied does the patient appear at the end of the encounter?

21. www.pharm.monash.edu Principles of effective case development –Realism! –Keep analytical checklist in mind –Aim for optimal number of checklist items, divided amongst relevant domains.

22. www.pharm.monash.edu Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

23. www.pharm.monash.edu Before reading –Volunteer leaves –Group reads/discusses –Role-play –Use checklist

24. www.pharm.monash.edu Ensure relevance Ensure feasibility Modify case (including SP background information) & checklist Must pass the “reasonable pharmacist” test (average, not minimally competent) Second role-play (real SP if possible)

25. www.pharm.monash.edu Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

26. www.pharm.monash.edu Unable to change case, weighting only. Decide minimum performance level (for case or items) Quantify competence; > cut score. Many methods, no gold-standard. Subjective/judgment-based, many opinions. Calculate “cut-off” score, decide global vs. analytical weighting (high-stakes)

27. www.pharm.monash.edu Defining minimum performance levels –Criticality/relevancy matrix method –The minimally competent student approach >Out of 100 minimally competent students, how many of them will actually get this item? –Decide individually, compare, re-adjust –Role-playing helpful –Compare scores between both methods

28. Criticality/relevancy matrix Low Criticality Moderate Criticality High Criticality Low Relevancy 0.10.30.5 Moderate Relevancy 0.30.50.7 High Relevancy 0.50.70.9

29. www.pharm.monash.edu Problems –Criticality/relevancy matrix method >No “Must-have” items >Can “Pass” without solving –The minimally competent student method >May require more interpretation >Reduces inter-rater reliability >Is increasingly being used

30. www.pharm.monash.edu Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

31. www.pharm.monash.edu Training Standardised Patients Actors or lay people Standardized vs. Simulated Assessment (Rating scale vs. checklist) Feedback (teaching purposes) Effective use of SPs –Use in teaching before assessment (TOSCE) –Become the role

32. www.pharm.monash.edu Clear case, thorough background Avoid jargon/provide definitions Clear checklist Train to checklist (e.g. one answer for one question) Role-play, correct problems in real-time. Group training (for consistent role-portrayal)

33. www.pharm.monash.edu Steps in development of an OSCE: 1.Blueprinting 2.Station development 3.Case writing 4.Case review & validation 5.Standard setting 6.Piloting 7.Assembly of OSCE

34. www.pharm.monash.edu Rooms Prompts Reference books Props Scoring sheets Timers Clear signage

35. www.pharm.monash.edu Running the OSCE Exam site staffing –site administrator, chief examiner (pharmacist), SP trainer(s), hall monitors, track co-ordinators (pharmacist), timers, runners Communication & authority “Back up” for everything Training assessors (blinded to item weightings & cut-score) Brief students (explain timing)

36. www.pharm.monash.edu Summary Costly, time-intensive & complex. Planning & organisation essential High stakes vs. Low stakes –time, effort, number of stations & security To fairly assess clinical competence in a realistic setting

37. www.pharm.monash.edu OSCEs in the Monash BPharm course PAC2331 – Pharmacists as Communicators: –2 stations >Community >Hospital –Communication skills only assessed –Global Rating Scale only used

38. www.pharm.monash.edu PAC2412 – Integrated Therapeutics – Cardiovascular: –3 stations >BP measurement >HT counselling in context of overall CV risk >Facilitating behaviour change –Clinical knowledge & communication/interpersonal & problem-solving skills assessed –Modified Analytical checklist only used

39. www.pharm.monash.edu PAC3362 – Context in Practice 2 –4 stations >Extemporaneous preparation >Drug information query >Communication with HCP/patient >Medication reconciliation –Clinical knowledge & communication/interpersonal & problem-solving skills assessed –Assessment method TBC

40. PAC2331 Evaluation (2012) Objectives were clearly defined for each OSCE station 38.6% 7.1% 4.3%

41. Cases examined in each OSCE station fairly tested lecture & tutorial material 17.6%

42. I felt prepared for the OSCE after attending the Teaching OSCE (TOSCE) 34.3% 17.9% 6%

43. Use of the OSCE assessment method is significantly better than written assessment to help develop my competency for practice 51.5%

44. PAC2412 Evaluation (2011) Objectives were clearly defined for each OSCE station

45. Scenarios presented for each OSCE station were a fair representation of challenges faced in practice

46. Use of the OSCE assessment method is significantly better than written assessment to help develop my competency for practice.