1. Standard Setting for a Performance-Based Examination for Medical Licensure Sydney M. Smee Medical Council of Canada Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

2. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona MCC Qualifying Examination Part II OSCE format - 12 short stations ●5 or 10 minutes per patient encounter ●Physicians observe and score performance Required for medical licensure in Canada Prerequisites ●Passed MCCQE Part I (Knowledge & Clinical reasoning) ●Completed 12 months of post-graduate clinical training Pass/Fail criterion-referenced examination Multi-site administration - twice per year Overall fail rate 10%-30% Implemented 1992

3. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Why do it? Requested by licensing authorities, largely in response to two issues: ●Increase in complaints, many centered around communication skills. ●Public accountability - OSCE to serve as an “audit” of training of all candidates seeking licensure in Canada.

4. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Blueprint Considerations Four domains *History-taking *Patient Interaction *Physical Examination *Management Multi-disciplinary / multi-system content Patient demographics Two formats * 5+5 couplets & 10 minute Each case based on an MCC Objective

5. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Standard for MCCQE Part II Acceptably competent for entry to independent practice Conjunctive standard ●Pass by total score AND ●Pass by minimum number of stations High performance in a few stations does not compensate for overall poor performance Just passing enough stations does not compensate for overall poor performance

6. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Translating a Standard to a Pass Mark Pilot exam: Ebel method ●Items rated for relevance and importance ●Pass based on most relevant and important items ●Failed 40% First two administrations: Angoff method ●Estimated score for the minimally competent candidate ●Pass based on average of estimates per instrument ●Pass marks varied more than the test committee liked ●Test committee did not like the task 1994: Adopted borderline group method

7. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Physicians as Scorers Three Assumptions: Clinicians do not require training to judge candidate behaviour according to checklists for basic clinical skills Most clinicians can make expert judgments about candidate performance Being judged by clinicians is vital for a high-stakes examination

8. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Physicians as Standard Setters

9. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Global Rating Question Did the candidate respond satisfactorily to the needs/problems presented by this patient? Borderline Satisfactory Good Excellent Borderline Unsatisfactory Unsatisfactory Inferior

10. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Numbers....  1,000-2,200 candidates per administration Examiners each observe 16-32 candidates 20-60 examiners per case Number of candidates identified as borderline per case ranges from  150-500 Collect >99% of data for global rating item

11. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Modified Borderline Group Method Examiners (content experts) identify borderline candidates based on the 6-point scale Scores of borderline candidates define performance that “describes” the pass standard Examiner judgments are translated into a pass mark by taking the mean score for the borderline candidates for each case

12. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Pass Marks by Case Across Exams Challenge to assess pass marks over multiple administrations ●Scoring instruments are revised post-exam ●Rating scale items have been revised ●Rating scale items have been added to cases As competency and difficulty of cases changes, so do cut scores

13. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Setting Total Exam Pass Mark Pass marks for cases are summed Add one standard error of measure (  3.2% ) Pass mark falls between 1 to 1.5 SD below mean score Station performance is reviewed by Central Examination Committee ●Then the standard for the number of stations passed is set Standard has been 8/12 since 2000

14. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Outcomes 15,331 candidates became eligible in 2000 – 2005 ●6,099 have yet to attempt MCCQE Part II ●8,514 have passed ●718 or 7.7% failed 2,243 candidates were eligible prior to 2000 and also took MCCQE Part II in 2000 – 2005 ●2,166 have passed ●77 or 3.4% failed and are likely out of the system Fail rates do not reflect impact on repeat takers ●Focused hundreds of candidates on remediation

15. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Limitation Current approach is easy to implement but it relies upon ●Large number of standard setters per case ●Large number of test takers in borderline group Smaller numbers would lead to more effort ●Increase training of examiners ●Impose stricter selection criteria on standard setters

16. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona What’s ahead? Increasing number of candidates to be assessed each year ●Modifications to the administration are needed Predictive validity study currently in progress ●Use non-physician examiners? Which type of cases, who sets standard? ●Add more administrations? Case development / challenge of piloting content

17. A Sample of References Burrows, P. J., Bingham, L., & Brailovsky, C. A. (1999). A modified contrasting groups method used for setting the passmark in a small scale standardized patient examination. Advances in Health Sciences Education: Theory and Practice, 4, 145-154. Cohen, A. S., Kane, M. T., & Crooks, T. J. (1999). A generalized examinee-centered method for setting standards on achievement tests. Applied Measurement in Education, 12, 343-366. Dauphinee, W. D., Blackmore, D. E., Smee, S. M., Rothman, A. I., & Reznick, R. K. (1997). Using the judgments of physician examiners in setting the standards for a national multi-center high stakes OSCE. Advances in Health Sciences Education: Theory and Practice, 2, 201-211. Haladyna, T. M. & Hess, R. (2000). An evaluation of conjunctive and compensatory standard-setting strategies for test decisions. Educational Assessment, 6, 129-153. Hambleton, R. K., Jaeger, R. M., Plake, B. S., & Mills, C. (2000). Setting performance standards on complex educational assessments. Applied Psychological Measurement, 24, 335-366. Jaeger, R. M. (1995). Setting standards for complex performances: An iterative, judgemental policy-capturing strategy. Educational Measurement: Issues and Practice, 14, 16-20. Kane, M. T., Crooks, T. J., & Cohen, A. S. (1999). Designing and evaluating standard-setting procedures for licensure and certification tests. Advances in Health Sciences Education: Theory and Practice, 4, 195-207. Kaufman, D. M., Mann, K. V., Muijtjens, A. M. M., & van der Vleuten, C. P. M. (2001). A comparison of standard-setting procedures for an OSCE in undergraduate medical education. Academic Medicine, 75, 267-271. Plake, B. S. (1998). Setting performance standards for professional licensure and certification. Applied Measurement in Education, 11, 65-80. Smee, S. M. & Blackmore, D. E. (2002). Setting standards for an objective structured clinical examination: The borderline group method gains ground on Angoff. Medical Education, 35, 1009-1010. Southgate, L., Hays, R. B., Norcini, J. J., Mulholland, H., Ayers, B., Woolliscroft, J., Cusimano, M. D., MacAvoy, P., Ainsworth, M., Haist, S., & Campbell, M. (2001). Setting performance standards for medical practice: A theoretical framework. Medical Education, 35, 474-481. Wilkinson, T. J., Newble, D. I., & Frampton, C. M. (2001). Standard setting in an objective structured clinical examination: use of global ratings of borderline performance to determine the passing score. Medical Education, 35, 1043-1049. Zieky, M. J. (2001). So much has changed: How the setting of cutscores has evolved since the 1980s. In G.J.Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 19-52). Mahwah, NJ: Lawrence Erlbaum Associates.

18. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Medical Council of Canada Ottawa Sydney M. Smee, M.Ed. Manager, MCCQE Part II www.mcc.ca