CLEAR 2008 Annual Conference Anchorage, Alaska Strategies and Outcomes of Pre-testing for Performance-Based Assessments Carol OByrne, PEBC

CLEAR 2008 Annual Conference Anchorage, Alaska Objective Structured Clinical Examination (OSCE) Why an OSCE? Examination Structure OSCE Stations –Patient interactions –Health professional interactions –Non-interactive tasks

CLEAR 2008 Annual Conference Anchorage, Alaska Stations Types – some examples SP Interactions –Provide information or advice –Counsel on new medication –Manage adverse drug reactions HP Interactions –Assist with medication management –Identify and resolve therapeutic problems (prescriptions, patient outcomes, etc.) Non-client Stations –Review new orders –Reconcile medication orders

CLEAR 2008 Annual Conference Anchorage, Alaska Scoring Candidates Performance Checklists –Critical items –Non-critical items –Incorrect items –Unique responses –Misinformation –Risk Rating scales –Communication –Problem solving / outcome –Overall performance

CLEAR 2008 Annual Conference Anchorage, Alaska Station Development Author workshop Editing Review workshop #1 Pre-test or bank for next/future exam? Review workshop #2 Exam Quality assurance, standard setting Revise, create variations

CLEAR 2008 Annual Conference Anchorage, Alaska Why Pretest? Check functionality of the station from the perspectives of –Candidates –Assessors –Standardized Clients Check station difficulty and length Confirm validity

CLEAR 2008 Annual Conference Anchorage, Alaska Why NOT Pretest? Exposure –Stations are memorable –Active grapevine Timeliness or criticality of content –Issue is time-sensitive, e.g. Health Canada or WHO advisory

CLEAR 2008 Annual Conference Anchorage, Alaska Selection Criteria for Pre-testing New station format, candidate response format New technology Ill-defined problem, e.g. ethical dilemma Complex station –More than one major objective –More than one problem Emerging, future-oriented practice or role

CLEAR 2008 Annual Conference Anchorage, Alaska Pre-test Strategies Include in the examination (entry to practice candidates) –One station per track (in addition to the 15 scored stations) –Each station pre-tested in two ore more sites (n=80+) –Data from actual candidates, under exam conditions –SP and Assessor feedback questionnaires Pre-test workshops (practising pharmacists)

CLEAR 2008 Annual Conference Anchorage, Alaska Feedback Questionnaires SPs –Questions, findings –Volunteer comments Assessors –Questions, findings –Volunteer comments Candidates –Questions, findings –Volunteer comments

CLEAR 2008 Annual Conference Anchorage, Alaska Case Studies Standardized Patient Interaction #1 –Pretest version, data and feedback –Revisions –Exam version, data and feedback –Recommendations Standardized Patient Interaction #2 –No pretest –Exam version, data and feedback –Recommendations

CLEAR 2008 Annual Conference Anchorage, Alaska Recommendations / Options Assessor and SP feedback –Revise questionnaires - capture more information –Ensure that assessors and SPs complete the questionnaires Sequester assessors and SPs to debrief and complete questionnaires Explain how feedback is used; provide example(s) Invite and record verbal comments

CLEAR 2008 Annual Conference Anchorage, Alaska Recommendations / Options SP Trainers and Chief Examiners –Develop feedback/report template/s pilot report template(s) include non-client and interactive stations –Gather questions/issues addressed during training and on exam day –Develop option to submit online in a secure manner

CLEAR 2008 Annual Conference Anchorage, Alaska Contact Information Carol OByrne Manager, PEBC Qualifying Examination – Part II (OSCE) Pharmacy Examining Board of Canada 717 Church Street Toronto, ON M4W 2M4 Tel: , ext Web Site: