1. Introduction to the CanAMS & the Internal Review Process
May 13th, 2019

2. Agenda for the Morning 8:30-9:00am Registration & Breakfast
What You Need to Know About CanERA Internal Reviews
Parveen Wasi
Teresa Vallera
10:00-10:15am
Break
10:15-11:00am
Tips & Tricks for Using the CanAMS
Brenda Montesanto
11:00-11:30am
CanAMS & the CanERA Standards
Marg Ackerman

3. Learning Outcomes At the end of this workshop, you will be able to:
Describe the requirements and procedure for the CanERA internal review
Apply best practices to prepare for your program’s internal review
Use the CanAMS to demonstrate your program’s achievement of CanERA standards

4. Review of the New CanERA System

5. Drs. Karen Finlay and Joanne Todesco
Acknowledgements
Royal College
Drs. Karen Finlay and Joanne Todesco

6. CanWHAT?
CanERA - Canadian Excellence in Residency Accreditation: The name given to the new system of accreditation
CanRAC - Canadian Residency Accreditation Consortium: The conjoint group including the Royal College, CFPC, and CMQ tasked with the development and ongoing support of CanERA
CanAMS - The digital Accreditation Management System: a fundamental component of CanERA)
Bailie

7. Highlights of CanERA New Accreditation Standards
Digitized Accreditation Management System (CanAMS)
Accreditation Site Visits
Changes to institution accreditation decisions
8-year review cycle
Focus on Continuous Quality Improvement

8. When will this affect me?

9. Previous Survey Cycle - OLD
RCPSC Site Survey 6 1
Continuous program self-review (CQA)
Faculty evaluations
Rotation evaluations
Curriculum evaluations
Trouble-shooting
Overall program review e.g. retreat 5 2 4
PG Office
Internal Review 3

10. New Survey Cycle
8 years between regular on-site accreditation visits
New electronic tools for surveyors / less paper & repetition / more flexibility
2 year follow-ups (some onsite, some not)
Introduction of common software/database (AMS)
Data collected by institutions throughout the cycle, including new sources of information
Selected data provided to RCPSC throughout the cycle

11. The Survey Process (stays the same)
University
PSQs
➡ Program Profiles
PSQs
➡ Program Profiles
Specialty Committee
Royal College
Comments
PSQs & Comments
➡ Program Profiles & Comments
Comments
Program Director
Surveyor

12. Review of the Standards
New standards, not so new
Review the new standards with your Residency Program Committee
What needs to be in place?
Fully compliant; partially compliant?
New standards will be used for the Internal Reviews
Talk with other PDs/ PAs in your discipline and outside for tips on the ‘newer standards’

13. Program Standards 5 DOMAINS Program Organization Education Program
++ Blueprinted from B1-6
Updated, clarified, & reorganized
Increased focus on outcomes, the learning environment, & CQI
CanMEDS framework remains
5 DOMAINS
Program Organization
Education Program
Resources
Learners, Teachers, & Administrative Personnel
Continuous Improvement

14. Standards Organization Framework
Domains were defined by the Future of Medical Education in Canada-Postgraduate (FMEC-PG) Accreditation Implementation Committee to introduce common organizational terminology, to increase alignment of accreditation standards across the medical education continuum.
The overarching outcome to be achieved through the fulfillment of the associated requirements.
A category of the requirements associated with the overarching standard.
A measurable component of a standard.
A specific expectation used to evaluate compliance with a requirement (i.e. to demonstrate that the requirement is in place).
Mandatory indicators must be met to achieve full compliance with a requirement. Exemplary indicators provide objectives beyond the mandatory expectations and may be used to introduce indicators that will become mandatory over time.

15. Requirement Rating Scale – NEW!
Meets: all mandatory indicators met
Partially meets: at least one, but not all mandatory indicators met
Does Not Meet: none of the mandatory indicators met
Key thing here is that each requirement is now receiving a rating, so there may be a number of partially meets, which can look scary
It is important to understand that decisions (status/follow-up) are made in the context of the decision making principles, and as such, X # of AFIs does not equal external review vs. APOR vs. regular review; rather, it depends what the AFIs are, and which follow-up is most appropriate in the context of the AFIs.
Communication with postgraduate offices re this – given that it will be a large culture shift.

16. Example - Domain: Program Organization
Element 1.1: The program director effectively leads the residency program.
Requirement 1.1.1: The program director is available to oversee and advance the residency program.
Indicator   : The faculty of medicine and the academic lead of the discipline provide the program director with sufficient support, autonomy and required resources for effective operation of the residency program.
Indicator   : Administrative support is organized and adequate to support the program director, the residency program and residents. [Link to Resources domain]
Indicator   :The program director and residency program committee(s) have access to resources and data/information to support the monitoring of resident performance, residency program review, and continuous improvement. [Link to Resources and CI domain]
Indicator  [Exemplary]: The program director and residency program committee(s) use an e-portfolio (or equivalent) to support the monitoring of resident performance, residency program review, and continuous improvement.
Standard:
There is an appropriate organizational structure, leadership and administrative personnel to effectively support the residency program, teachers and residents.
Requirement Compliance Scale
Score
Non-compliance 1
Partial compliance 2
Full compliance 3
Exemplary compliance
Description
No mandatory indicators met
Some of the mandatory indicators met
All mandatory indicators met
All mandatory indicators met and an exemplary indicator met

17. Area for Improvement (API) Replaces previous weaknesses
New Terminology
Area for Improvement (API)
Replaces previous weaknesses
Requirement level (PM or DNM)
Leading Practice and Innovation (LPI)
Replaces strengths

18. New Accreditation Categories
Your Institution will now receive an accreditation category too
New programs will now have an External Review
Mandated Internal Reviews & Progress Reports are replaced by the APOR
APOR = Action Plan Outcomes Report

19. POSSIBLE CATEGORIES OF ACCREDITATION - NEW
A – RS
A – APOR
A – ER
NOTICE OF INTENT
WITHDRAWAL

20. APOR = Action Plan Outcomes Report
Replaces A-IR and PR
Living register tracking how weaknesses (AFIs) are being addressed
Discussion with PGME and Program as how best to address AFIs

21. Accredited program with follow-up at next regular onsite survey (i. e
Accredited program with follow-up at next regular onsite survey (i.e. in 8 yrs)
Acceptable compliance with standards (could have AFIs)
Expectation of good, ongoing CQI throughout the cycle

22. Accredited program with follow-up by APOR
One (or more) significant area(s) for improvement impacting the overall quality of the program requiring follow-up prior to the next regular onsite review, and which can be evaluated via submission of evidence from the program. Predictable 2-year follow up

23. Accredited program with follow-up by External Review
One (or more) significant area(s) for improvement impacting the overall quality of the program requiring follow-up prior to the next regular onsite review, and which can be best evaluated by external peer reviewers. Factors: Persistent area(s) requiring improvement; nature of the area(s) of improvement may require reviewer from outside university and/or from same discipline; concerns with program’s or institution’s oversight or CQI of the program Predictable 2-year follow up

24. Accredited program on Notice of Intent to Withdraw
There are major and/or continuing concerns which call into question the educational environment and/or integrity of the residency program and its ability to deliver high quality residency education. OR
Despite notifications and reminders, the program has failed to complete and submit the required accreditation follow-up by the deadline.
Current residents & CaRMS applicants must be made aware.
Predictable 2-year follow up - Onus is on the program to show why accreditation should not be withdrawn

25. INTERNAL REVIEWS Important evidence of Institutional CQI
Accreditation standard for Institutions
McMaster Process
Similar process to the Royal College/CFPC on-site surveys
Same documentation
Similar accreditation decisions ( exception: notice of intent)
Accreditation Committee to review all reports

26. Internal Review Does not need to be perfect
Quality Assurance and Quality Improvement process
Standardised Process for follow-up important
Attention to program CQI
Attention to Learning environment
What we can learn collectively:
New standards- what are frequent compliance issues
Guide education and resources
Best practices
Areas of learning environments that need improvement

27. Survey Team Chair- usually a PD or past PD Faculty member Resident
Some programs will have an external faculty member from the same discipline ( resource issues; large feeder programs; program identification of need)

28. Information Provided to Surveyors in Advance
Access to Program Profile (incl response to previous weaknesses)
Specialty specific documents
Survey Report & Transmittal Letter from last RCPSC survey

29. Information Provided to Surveyors on Site
Resident assessments
Faculty/ rotation evaluation
Face-to-face interviews
RPC & Competence Committee Minutes (past 6 years)
University’s Internal Review process

30. Program Profile (previously known as the PSQ)
IT’S A BIG DEAL!

31. First impressions count!
Describes how your program is meeting each standard – “evidence”
Guides the surveyor’s questions
Reviewed by many:
PG Dean
Your Surveyor(s),
Accreditation Committee members

32. Be clear & thorough –
If you are doing something a bit different or are dealing with a challenge, tell us all about it & defend your choices
Attend to spelling, grammar, & formatting
Get help from others. Give yourself lots of time. Hunt down all the numbers, institutional policies, & governance information. You should be the most informed person about your program!

33. Use abbreviations where necessary, but always include a legend
Final draft should be reviewed by your RPC including resident reps & department head
7. Tell what is happening now rather than what you wish to happen

34. And finally … AVOID: We will be … We hope to …
Only using “role modeling” & “observation” for the intrinsic CanMEDS Roles (OK in 2000, not OK in 2019)

35. Access the training modules here:
Additional Resources
CanERA has developed online training modules. The modules will allow you to:
Familiarize yourself with the standards
Understand and navigate the CanAMS
Access the training modules here:

36. McMaster Postgraduate Medical Education Office