Introduction to the CanAMS & the Internal Review Process May 13th, 2019

Agenda for the Morning 8:30-9:00am Registration & Breakfast What You Need to Know About CanERA Internal Reviews Parveen Wasi Teresa Vallera 10:00-10:15am Break 10:15-11:00am Tips & Tricks for Using the CanAMS Brenda Montesanto 11:00-11:30am CanAMS & the CanERA Standards Marg Ackerman

Learning Outcomes At the end of this workshop, you will be able to: Describe the requirements and procedure for the CanERA internal review Apply best practices to prepare for your program’s internal review Use the CanAMS to demonstrate your program’s achievement of CanERA standards

Review of the New CanERA System

Drs. Karen Finlay and Joanne Todesco Acknowledgements Royal College Drs. Karen Finlay and Joanne Todesco

CanWHAT? CanERA - Canadian Excellence in Residency Accreditation: The name given to the new system of accreditation CanRAC - Canadian Residency Accreditation Consortium: The conjoint group including the Royal College, CFPC, and CMQ tasked with the development and ongoing support of CanERA CanAMS - The digital Accreditation Management System: a fundamental component of CanERA) Bailie

Highlights of CanERA New Accreditation Standards Digitized Accreditation Management System (CanAMS) Accreditation Site Visits Changes to institution accreditation decisions 8-year review cycle Focus on Continuous Quality Improvement

When will this affect me? www.CanERA.ca

Previous Survey Cycle - OLD RCPSC Site Survey 6 1 Continuous program self-review (CQA) Faculty evaluations Rotation evaluations Curriculum evaluations Trouble-shooting Overall program review e.g. retreat 5 2 4 PG Office Internal Review 3

New Survey Cycle 8 years between regular on-site accreditation visits New electronic tools for surveyors / less paper & repetition / more flexibility 2 year follow-ups (some onsite, some not) Introduction of common software/database (AMS) Data collected by institutions throughout the cycle, including new sources of information Selected data provided to RCPSC throughout the cycle

The Survey Process (stays the same) University PSQs ➡ Program Profiles PSQs ➡ Program Profiles Specialty Committee Royal College Comments PSQs & Comments ➡ Program Profiles & Comments Comments Program Director Surveyor

Review of the Standards New standards, not so new Review the new standards with your Residency Program Committee What needs to be in place? Fully compliant; partially compliant? New standards will be used for the Internal Reviews Talk with other PDs/ PAs in your discipline and outside for tips on the ‘newer standards’

Program Standards 5 DOMAINS Program Organization Education Program ++ Blueprinted from B1-6 Updated, clarified, & reorganized Increased focus on outcomes, the learning environment, & CQI CanMEDS framework remains 5 DOMAINS Program Organization Education Program Resources Learners, Teachers, & Administrative Personnel Continuous Improvement

Standards Organization Framework Domains were defined by the Future of Medical Education in Canada-Postgraduate (FMEC-PG) Accreditation Implementation Committee to introduce common organizational terminology, to increase alignment of accreditation standards across the medical education continuum. The overarching outcome to be achieved through the fulfillment of the associated requirements. A category of the requirements associated with the overarching standard. A measurable component of a standard. A specific expectation used to evaluate compliance with a requirement (i.e. to demonstrate that the requirement is in place). Mandatory indicators must be met to achieve full compliance with a requirement. Exemplary indicators provide objectives beyond the mandatory expectations and may be used to introduce indicators that will become mandatory over time.

Requirement Rating Scale – NEW! Meets: all mandatory indicators met Partially meets: at least one, but not all mandatory indicators met Does Not Meet: none of the mandatory indicators met Key thing here is that each requirement is now receiving a rating, so there may be a number of partially meets, which can look scary It is important to understand that decisions (status/follow-up) are made in the context of the decision making principles, and as such, X # of AFIs does not equal external review vs. APOR vs. regular review; rather, it depends what the AFIs are, and which follow-up is most appropriate in the context of the AFIs. Communication with postgraduate offices re this – given that it will be a large culture shift.

Example - Domain: Program Organization Element 1.1: The program director effectively leads the residency program. Requirement 1.1.1: The program director is available to oversee and advance the residency program. Indicator 1.1.1.1 : The faculty of medicine and the academic lead of the discipline provide the program director with sufficient support, autonomy and required resources for effective operation of the residency program. Indicator 1.1.1.2 : Administrative support is organized and adequate to support the program director, the residency program and residents. [Link to Resources domain] Indicator 1.1.1.3 :The program director and residency program committee(s) have access to resources and data/information to support the monitoring of resident performance, residency program review, and continuous improvement. [Link to Resources and CI domain] Indicator 1.1.1.4 [Exemplary]: The program director and residency program committee(s) use an e-portfolio (or equivalent) to support the monitoring of resident performance, residency program review, and continuous improvement. Standard: There is an appropriate organizational structure, leadership and administrative personnel to effectively support the residency program, teachers and residents. Requirement Compliance Scale Score Non-compliance 1 Partial compliance 2 Full compliance 3 Exemplary compliance Description No mandatory indicators met Some of the mandatory indicators met All mandatory indicators met All mandatory indicators met and an exemplary indicator met

Area for Improvement (API) Replaces previous weaknesses New Terminology Area for Improvement (API) Replaces previous weaknesses Requirement level (PM or DNM) Leading Practice and Innovation (LPI) Replaces strengths

New Accreditation Categories Your Institution will now receive an accreditation category too New programs will now have an External Review Mandated Internal Reviews & Progress Reports are replaced by the APOR APOR = Action Plan Outcomes Report

POSSIBLE CATEGORIES OF ACCREDITATION - NEW A – RS A – APOR A – ER NOTICE OF INTENT WITHDRAWAL

APOR = Action Plan Outcomes Report Replaces A-IR and PR Living register tracking how weaknesses (AFIs) are being addressed Discussion with PGME and Program as how best to address AFIs

Accredited program with follow-up at next regular onsite survey (i. e Accredited program with follow-up at next regular onsite survey (i.e. in 8 yrs) Acceptable compliance with standards (could have AFIs) Expectation of good, ongoing CQI throughout the cycle

Accredited program with follow-up by APOR One (or more) significant area(s) for improvement impacting the overall quality of the program requiring follow-up prior to the next regular onsite review, and which can be evaluated via submission of evidence from the program. Predictable 2-year follow up

Accredited program with follow-up by External Review One (or more) significant area(s) for improvement impacting the overall quality of the program requiring follow-up prior to the next regular onsite review, and which can be best evaluated by external peer reviewers. Factors: Persistent area(s) requiring improvement; nature of the area(s) of improvement may require reviewer from outside university and/or from same discipline; concerns with program’s or institution’s oversight or CQI of the program Predictable 2-year follow up

Accredited program on Notice of Intent to Withdraw There are major and/or continuing concerns which call into question the educational environment and/or integrity of the residency program and its ability to deliver high quality residency education. OR Despite notifications and reminders, the program has failed to complete and submit the required accreditation follow-up by the deadline. Current residents & CaRMS applicants must be made aware. Predictable 2-year follow up - Onus is on the program to show why accreditation should not be withdrawn

INTERNAL REVIEWS Important evidence of Institutional CQI Accreditation standard for Institutions McMaster Process Similar process to the Royal College/CFPC on-site surveys Same documentation Similar accreditation decisions ( exception: notice of intent) Accreditation Committee to review all reports

Internal Review Does not need to be perfect Quality Assurance and Quality Improvement process Standardised Process for follow-up important Attention to program CQI Attention to Learning environment What we can learn collectively: New standards- what are frequent compliance issues Guide education and resources Best practices Areas of learning environments that need improvement

Survey Team Chair- usually a PD or past PD Faculty member Resident Some programs will have an external faculty member from the same discipline ( resource issues; large feeder programs; program identification of need)

Information Provided to Surveyors in Advance Access to Program Profile (incl response to previous weaknesses) Specialty specific documents Survey Report & Transmittal Letter from last RCPSC survey

Information Provided to Surveyors on Site Resident assessments Faculty/ rotation evaluation Face-to-face interviews RPC & Competence Committee Minutes (past 6 years) University’s Internal Review process

Program Profile (previously known as the PSQ) IT’S A BIG DEAL!

First impressions count! Describes how your program is meeting each standard – “evidence” Guides the surveyor’s questions Reviewed by many: PG Dean Your Surveyor(s), Accreditation Committee members

Be clear & thorough – If you are doing something a bit different or are dealing with a challenge, tell us all about it & defend your choices Attend to spelling, grammar, & formatting Get help from others. Give yourself lots of time. Hunt down all the numbers, institutional policies, & governance information. You should be the most informed person about your program!

Use abbreviations where necessary, but always include a legend Final draft should be reviewed by your RPC including resident reps & department head 7. Tell what is happening now rather than what you wish to happen

And finally … AVOID: We will be … We hope to … Only using “role modeling” & “observation” for the intrinsic CanMEDS Roles (OK in 2000, not OK in 2019)

Access the training modules here: Additional Resources CanERA has developed online training modules. The modules will allow you to: Familiarize yourself with the standards Understand and navigate the CanAMS Access the training modules here: http://www.royalcollege.ca/mssites/canera-uprh/index.html#/

McMaster Postgraduate Medical Education Office postgd@mcmaster.ca