1. Objective Structured Clinical Examination (OSCE) Arnuparp Lekhakula M.D.,M.S. Faculty of Medicine Prince of Songkla University Hat Yai, Songkhla

2. What is OSCE? Multiple stations, each testing different competencies or tasks Time limit for each station Students rotate through all the station and are tested on the same material All are judges by the same preset standards – usually using checklists or rating scales

3. Why OSCE? Before OSCE (1975) Valid? Know how NOT show how Reliable? Different patients Different examiners OSCE - more valid, more reliable Show how Same task / patient / examiner / structured marking sheet

4. Advantages of OSCEs Candidate’s knowledge and skills over whole range of topics can be tested Use of different test methods Test not only knowledge and skills, but attitudes as well Efficient use of limited resources More objective Specific profiling of each component of performance

5. Climbing the pyramid...... Knows Shows how Knows how Does Knows Factual tests: MCQ, essay type, oral….. Knows how (Clinical) Context based tests: MCQ, essay type, oral….. Shows how Performance assessment in vitro: OSCE, SP-based test….. Does Performance assessment in vivo: Masked SPs, Video, Audits…..

6. Common Instruments Direct MCQ MEQ CRQ OSCE Observation Knowledge +++ ++ +++ + + Problem- solving + ++ ++ + ++ Practical skill +++ ++ Interpersonal skill ++ +++ Attitudes + + + ++

7. Common Instruments Direct MCQ MEQ CRQ OSCE observation Validity Content +++ ++ +++ ++ +++ Face + +++ ++ +++ +++ Construct + +++ + + +++ Reliability +++ ++ +++++ ++ Objectivity +++ ++ +++++ ++ Feasibility +++ + + + ++

8. Limitations of OSCEs Labor-intensive, lengthy preparation Costly Require trainees to perform isolated aspects of the clinical encounter Rely on task specific checklists and scoring What can be simulated constrain the type of patient problems used Need more observation skills of staff

9. Limitations of OSCEs Standardization of simulated patients and examiners Low inter-station correlation Test security Repetitive - boring

10. Factors Leading to Lower Reliability Too few stations or too little testing time Checklists or items that don’t discriminate Unreliable patients or inconsistent portrayals by standardized patients Examiners who score idiosyncratically Administrative problems

11. What Assessed by OSCE Various clinical skills – history taking, physical examination, technical procedure (on model), communication, interpersonal skills Knowledge and understanding Data interpretation Problem solving Attitudes

12. Types of OSCE stations Process - various skills, attitudes Product – data interpretation, diagnosis, problem solving, report writing, order sheet writing, drug description Mixed – process and product

13. Components of OSCE station Stem with clearly stated task Checklist - Appropriate for level of training - Task-based - Observable Training information

14. Stem John Smith, aged 37, arrived in the emergency room Complaining of acute abdominal pain that began 12 hours previously Instruction to student In the next five minutes, conduct a relevant physical examination

15. Checklist Examiner to fill in box to each item that student successfully completes Marks Drapes patient appropriately2 Inspects abdomen1 Auscultation abdomen1 Percussion abdomen1 Asks patient for the location of pain1 Start to palpate abdomen from the area which is not pain2 Lightly palpates each quadrant2 Deeply palpates each quadrant2 Checks for peritoneal irritation2 Stem

16. Training Information History of pain The pain started 12 hours ago Symptoms The pain is in the right lower quadrant, at “at least 9”, and is constant. His abdomen is tense even when palpate lightly. With deeper palpation, there is guarding in the RLQ, and McBurney point is acutely tender. Obturator (raising right knee against resistance) and psoas signs (extension of right leg at hip-kicking backward) are positive. Stem Checklist

17. Steps in Developing OSCEs Assembling working groups Defining competencies/content assessed Developing blueprint Writing an OSCE station – process or product - Writing a scenario - Develop instructions to the candidate: precise, concise, unambiguous - Write instructions to the simulator/SPs, a script for role-playing - Draw up a list of resources: space, people, equipment, materials

18. Steps in Developing OSCEs Developing checklist Draw a scoring scheme Revised by committee Setting standards/passing score Pilot testing Reviewing and rewriting

19. Table of Specification Area tested A1A2A3 A5A6A7A8….. Hx3 PE3 ProceduresXXXX4 Data interpret. XXXX4 Problem- solving 2 Com skill1 Pt Ed1 Attitudes2 Total2332332220

20. Organizing an OSCE What is to be assessed – produce a table of specification Duration of stations – 4-15 minutes, mostly 5 minutes Number of stations – 15-20 stations Format of stations – isolated, linked Use of examiners – fully briefed prior to examination Preparation of checklists Organization of examination

21. OSCE Station 1234 5 6789 10 1 2 2 34 10 min 1234 Basic Double- time Linked

22. Organizing an OSCE Assigning priority Resource requirement – examiners, patients (real, simulated), equipment, paperwork Plan of examination and direction Change signal Records

23. How to Improve Checklist Stem - Clear - Observable - Not to long Overall - Not too long Rating scale - More clarification of each scale - Done, complete/Done, incomplete/ Not done - Rater training

24. Observation Direct Indirect - One-way mirror - Monitor - Video

25. Instructions to Simulated Patient Situation Information about illness Role playing - Facial/emotional/non-verbal expression - Response to students : normal/abnormal - Questions asked to students - Answers prepared to students’ questions Dressing/Costume Make-up

26. Summary of OSCE stations Station Compe- tency Area tested Linked stationProcessProductSP Check -list Exami- ner Equip- ments

27. Timeline for OSCE 8 wkSelect date and appoint overall coordinator and site coordinator 6 wkDecide on the station tasks, book site and refreshments, make local arrangements 3 wkReview all station details 2 wkHave all station paperwork printed, signs made, equipment prepared, remind examiner

28. Timeline for OSCE 24 hrWalk through examination site with coordinator 2 hrAll coordinator on-site 1 hrFinal briefing of examiners and SPs 30 minAll examiners and SPs at stations Students’ orientation/briefing

29. Standard Setting - Angoff Method Performance levels at “minimally acceptable” or “borderline” is used to derive cut-off score Undertaken by a panel (usually a minimal of 6) Judges independently estimate what proportion (%) of minimally competent examinees would answer the item correctly

30. Example - Angoff ItemsFull mark JudgeAngoff Score Explain to patient180%0.8 Position of patient180%0.8 Inspection of skin280%1.6 Temperature change160%0.6 Pulsation : Femoral180%0.8 Popliteal150%0.5 Posterior tibial150%0.5 Dorsalis pedis150%0.5 Capillary refill150%0.5 Total106.6

31. Example - Angoff ItemsNot done Done poorly Done statis. Explain to patient00.51.0 Position of patient00.51.0 Inspection of skin01.02.0 Temperature change00.51.0 Pulsation : Femoral00.51.0 Popliteal00.51.0 Posterior tibial00.51.0 Dorsalis pedis00.51.0 Capillary refill00.51.0 Maximum total = 10.0Angoff score = 6.0

32. Standard Setting - Borderline Method Stu123456789 Score809060707555806585 Global rating SSSBUBSSUS S Marking form : checklist + global rating All categorized ‘borderline’ students Mean score of borderline group Passing score = (60+70+65)/3 = 65

33. Conclusions : OSCE What? Stations + tasks + checklist Why? More valid, more reliable, more objective How? How to organize? How to analyze?

34. OSCE – Task to Do Ahead (1) Create blue print Set timeline Get authors for case-writing workshop Review and finalize case Arrange workshop on setting standards Recruit standardized patients/examiners Train SPs Print scoring sheet, mark signs

35. OSCE – Task to Do Ahead (2) List all supplies for set-up of OSCE stations Remind everyone of date Make sure students have all the information Plan of the examination day - Diagram of station layout - Directions for examiners, SPs and staff - Registration table for examiners, SPs and students - Timing and signals (stopwatch, wristles or bell) - Procedures for ending examination