R 3 P Colloquium American Board of Pediatrics Jan. 31 – Feb. 2, 2007 The Past, Present and Future Assessments of Clinical Competence A Canadian Perspective Bob Hilliard MD EdD FRCPC Pediatric Specialty Committee

Many, many thanks for the invitation!! Overview – No Debate Whatsoever Assessments and evaluations in a pediatric residency program must be an integral part of the curriculum (teaching and learning experiences) and based on clear, obtainable and measurable educational objectives!! Examinations and evaluations influence, direct and drive resident learning: what, how, why. This includes not just the content of what is on the examination but also the format.

Pediatric Training In Canada Royal College of Physicians and Surgeons of Canada  Surveys & accredits programs, credentials & certifies specialists  Pediatric residencies are all university based (16)  Goals & Objectives – the pediatric consultant  Three (3) Core years +  one (1) additional year of general pediatrics, or  the first year (1) of pediatric subspecialty training  Prior to 2001 – the final exam was  Multiple Choice Questions (MCQ) exam  Two-part oral exam long case – observed + discussion / short case four cards selected from a box of questions

Objectives of Training Specialty Training Requirements in Pediatrics CanMEDS RolesABMS Competencies (Essential Key Competencies)  Medical Expert  Medical Knowledge  Systems Based Knowledge +!  Skills  Patient Care  Attitudes  Problems  Communicator  Interpersonal / communication  Collaboratorskills  Manager  Systems-based practice  Health Advocate  Scholar  Practice-based learning  Professional  Professionalism

In – Training Evaluations  Rotation Specific In-Training Evaluation Reports (Global Assessment of Performance) CanMEDS format Formative - face-to-face feedback  Written Multiple Choice Questions (MCQ) Exam  Oral / Clinical Exam  Observed History and Physical Exam / Mock Oral  Objective Structured Clinical Exam  “360º Evaluations”  Evaluation of Research / Quality Assurance Projects  Learning Portfolios

Final In-Training Evaluation Report  Certificate of Confirmation of Completion of Training (CCT)  FITER - Summative - compilation – CanMEDS roles  STACER (Standardized Assessment of a Clinical Encounter Report ) – Observed History and Physical Examination – two (2) observers / examiners, structured check list – may be repeated until satisfactory performance

The Final Examination in Pediatrics A condensed, comprehensive three-part exam  blue print – systems based content + CanMEDS roles Multiple Choice Questions (MCQ) Exam – 150 / 3 hrs Short Answer Questions (SAQ) – 60 – 75 / 3 hrs Structured Oral Questions (OSCE) – 10 X 15 mins - attempt to evaluate all CanMEDS roles - standardized patients – history taking, counseling - parents and adolescents (≥ 12 years) - telephone advice stations - structured oral stations - physical examination stations (standardized patients) - pictures / videos - critical appraisal stations - reliability ≈ 0.80

The Final Examination in Pediatrics (Reflections after six year’s experience)  Reliable – all exams - reliability  0.80  High face validity (credibility)  High content validity  Canadian trainees do very well  Non-Canadian trainees – much lower pass rate ? – language, lack of basic skills,  exam experience,  anxiety  Logistically – time consuming, labour intensive days in Ottawa 2 days – paired stations, both English and French 3 sessions – candidates sequestered to avoid contacts 35 examiners, 2006 – 170 candidates

The question still is: What should be the standards of a practicing pediatrician in Canada and how should these competencies be evaluated? One Grade Only and That the Best!!

Thank You Very Much for your Attention!! Any Comments or Questions??

Maintenance of Competence a.Canadian Perspective 80 hours of continuing education accredited CE based on adult learning principles individual personal learning projects accredited self-assessment programs practice audits, practice reviews and appraisal b.USA Perspective evidence of professional standing evidence of life-long learning evidence of cognitive expertise evidence of satisfactory performance in Pediatrics

Physicians who have not Trained in Canada a.Only training in a limited number of specific jurisdictions is accepted by the RCPSC. b.Many immigrant physicians have not had the same evaluations as Canadian residents. c.It is difficult for immigrant physicians to get further specialty training. d.Some provincial licensing authorities (CPS) grant temporary or restricted licenses to practice in supervised setting provided that the physician pass the RCPSC exams and complete a five year cycle of Maintenance of Certification. e.Pediatricians who have trained outside of Canada have difficulties passing the RCPSC certification exams.