1. CLEAR 2008 Annual Conference Anchorage, Alaska Strategies and Outcomes of Pre-testing for Performance-Based Assessments Carol OByrne, PEBC

2. CLEAR 2008 Annual Conference Anchorage, Alaska Objective Structured Clinical Examination (OSCE) Why an OSCE? Examination Structure OSCE Stations –Patient interactions –Health professional interactions –Non-interactive tasks

3. CLEAR 2008 Annual Conference Anchorage, Alaska Stations Types – some examples SP Interactions –Provide information or advice –Counsel on new medication –Manage adverse drug reactions HP Interactions –Assist with medication management –Identify and resolve therapeutic problems (prescriptions, patient outcomes, etc.) Non-client Stations –Review new orders –Reconcile medication orders

4. CLEAR 2008 Annual Conference Anchorage, Alaska Scoring Candidates Performance Checklists –Critical items –Non-critical items –Incorrect items –Unique responses –Misinformation –Risk Rating scales –Communication –Problem solving / outcome –Overall performance

5. CLEAR 2008 Annual Conference Anchorage, Alaska Station Development Author workshop Editing Review workshop #1 Pre-test or bank for next/future exam? Review workshop #2 Exam Quality assurance, standard setting Revise, create variations

6. CLEAR 2008 Annual Conference Anchorage, Alaska Why Pretest? Check functionality of the station from the perspectives of –Candidates –Assessors –Standardized Clients Check station difficulty and length Confirm validity

7. CLEAR 2008 Annual Conference Anchorage, Alaska Why NOT Pretest? Exposure –Stations are memorable –Active grapevine Timeliness or criticality of content –Issue is time-sensitive, e.g. Health Canada or WHO advisory

8. CLEAR 2008 Annual Conference Anchorage, Alaska Selection Criteria for Pre-testing New station format, candidate response format New technology Ill-defined problem, e.g. ethical dilemma Complex station –More than one major objective –More than one problem Emerging, future-oriented practice or role

9. CLEAR 2008 Annual Conference Anchorage, Alaska Pre-test Strategies Include in the examination (entry to practice candidates) –One station per track (in addition to the 15 scored stations) –Each station pre-tested in two ore more sites (n=80+) –Data from actual candidates, under exam conditions –SP and Assessor feedback questionnaires Pre-test workshops (practising pharmacists)

10. CLEAR 2008 Annual Conference Anchorage, Alaska Feedback Questionnaires SPs –Questions, findings –Volunteer comments Assessors –Questions, findings –Volunteer comments Candidates –Questions, findings –Volunteer comments

11. CLEAR 2008 Annual Conference Anchorage, Alaska Case Studies Standardized Patient Interaction #1 –Pretest version, data and feedback –Revisions –Exam version, data and feedback –Recommendations Standardized Patient Interaction #2 –No pretest –Exam version, data and feedback –Recommendations

12. CLEAR 2008 Annual Conference Anchorage, Alaska Recommendations / Options Assessor and SP feedback –Revise questionnaires - capture more information –Ensure that assessors and SPs complete the questionnaires Sequester assessors and SPs to debrief and complete questionnaires Explain how feedback is used; provide example(s) Invite and record verbal comments

13. CLEAR 2008 Annual Conference Anchorage, Alaska Recommendations / Options SP Trainers and Chief Examiners –Develop feedback/report template/s pilot report template(s) include non-client and interactive stations –Gather questions/issues addressed during training and on exam day –Develop option to submit online in a secure manner

14. CLEAR 2008 Annual Conference Anchorage, Alaska Contact Information Carol OByrne Manager, PEBC Qualifying Examination – Part II (OSCE) Pharmacy Examining Board of Canada 717 Church Street Toronto, ON M4W 2M4 Tel: 416-979-2431, ext 226 Email: obyrnec@pebc.caobyrnec@pebc.ca Web Site: www.pebc.ca