Milestones: Are You Ready for July 1st ? Kirk Lalwani, MD, FRCA, MCR Associate Professor, Chair, Resident Evaluation and Clinical Competence Committee, Anesthesiology and Perioperative Medicine. On behalf of the AMIGOS (Anesthesiology Milestones Implementation Group Operatives)

Disclosure No conflict of interest Some of my slides are borrowed from the faculty educational resources on the ACGME website

Implementing Milestones Objectives: Outline the Next Accreditation System (NAS) Define a Milestone Review our Milestones strategy Discuss assessment of Anesthesiology Milestones Three key points will be addressed in this presentation.

Abbreviations NAS- Next Accreditation System CLER- Clinical Learning Environment ACGME- Accreditation Council for Graduate Medical Education RRC- Residency Review Committee

The Next Accreditation System This slide presentation provides general information regarding the Next Accreditation System (NAS), including the reasons for its development, the components of NAS, and the differences between NAS and the “old” Accreditation System. This slide deck was created in December 2013 and care should be taken to ensure that updates and changes that have occurred since then are incorporated to provide an up to date and accurate presentation.

Why ‘Next Accreditation System’ (NAS)? “Self-regulation is a fundamental professional responsibility, and the system for educating physicians answers to the public for the graduates it produces.” “The Next GME Accreditation System – Rationale and Benefits” Nasca T.J., Philibert I., Brigham T., Flynn T.C. N Engl J Med 2012; 366:1051-1056

Goals of NAS Strengthen resident development in Professionalism Communication Skills Systems Based Practice Practice Based Learning Enhance public accountability More explicit definition of a good physician (Milestones) Patient safety (Clinical Learning Environment Review Program) Reduce burden required for accreditation

‘Next Accreditation System’ …in a Nutshell Accreditation on the basis of educational outcomes NAS : Advance from an episodic ` biopsy ’ model to annual data collection or ` continuous oversight ’. RRCs will measure compliance through the evaluation of annual program data elements.

The Next Accreditation System Continuous Observations Promote Innovation Assess Program Improvement(s) Identify Opportunities for Improvement The NAS seeks to provide continuous observations and identify opportunities for programs to improve and ensure that programs in good standing are free to innovate with regard to Detail requirements. Program Makes Improvement(s)

The Next Accreditation System We will now have annual data collection Trends in annual data Milestones Resident and Faculty Surveys Resident and Faculty Scholarly activities Case Log Data ITE / ABA Examination pass rates Programs will be evaluated annually and data reviewed include trends, Milestone levels, results of Resident and Faculty surveys, and scholarly activity for faculty members, residents and fellows. For clinical experience, Case Logs will be reviewed for those specialties that require Case Log reporting. For specialties that do not have Case Log reporting, questions that address the clinical environment have been added to the Resident Survey. In addition, programs that receive Continued Accreditation will be free to innovate on Detail requirements.

Clinical Learning Environment Review Visits Oversight of Transitions of Care Duty Hours Policy Fatigue Management and Mitigation Education and Monitoring of Professionalism Involvement in Institutional Quality Improvement and Safety initiatives Supervision policies Components of the site visit – encourage people to review the slides and presentations on the ACGME website concerning the CLER visits. Additional information available at: http://www.acgme-nas.org/CLER

Milestones

What are Milestones? Specific behaviors, attributes, or outcomes in the six general competency domains to be demonstrated by residents during residency. (ACGME and Specialty Boards) Skill and knowledge-based development that commonly occurs by a specific time.

Why ???? ACGME requires it Program Accreditation

Who, When, and How???? All residents in all ACGME-accredited programs Subspecialty fellows from July 2015 New assessment methodology as well as existing tools 6-monthly reporting of Milestones to ACGME in January and July Reporting starts July 2014

Expected Benefits of Milestone Assessments For Residents Clarify expectations Feedback should identify specific areas to work on Earlier identification of under-performers Aspirational goals for residents who exceed expectations In early experience with Milestone assessment, some observations have been that residents appreciate the more detailed and specific evaluations that milestones allow, compared to previous evaluations merely stating they had done a “good job.” Milestone evaluations provide a process to set expectations on what more there is to learn or what a resident has to accomplish to move to the next level. One of the goals in looking at each resident more closely is to identify residents who have gotten by because they are nice and easy to work with, but who actually had significant deficiencies in different skills and areas in the subcompetencies. The focus on specific skills and knowledge in the Milestones help to identify these residents and determine which in competency or subcompetency areas in which they need to improve.

Expected Benefits of Milestone Assessment For the Program Curriculum development Accreditation requirements Earlier identification of under-performers For the Public Better definition of what a physician can do at the completion of training Program is accountable to a common standard Possible use for board certification Some program directors have also suggested other benefits. In looking at the Milestones, a program may discover that the current curriculum may not be teaching a competency to the expectations set for Level 4. The benefits to the public may be less clear to individual programs, but allow policy makers to understand some of the specific skills that future physicians in practice are learning. Currently the ACGME only expects each program to collect and use the Milestones in assessing their residents. The ACGME does not expect that every resident meets a specific level at a particular point in residency. The goal is to use Milestone assessment to thoughtfully and honestly evaluate each resident. The Review Committee will not look at an individual resident’s Milestone attainment. Each specialty board will determine on its own whether and how to use Milestone data in board certification.

Pragmatic Milestones Development Strategy Minimalist Comprehensive Program implements new milestone aligned tools, which help residents understand expectations Comprehensive Program changes assessment system, invests time in faculty development and education to improve rater expertise and CCC validity Minimalist CCC assigns milestone levels using existing tools

Process: June 2013 Assembled AMIGOS Group from RECCC, faculty, and residents to represent all clinical areas Survey to rank importance of each Milestone, following which 17 / 25 chosen Small groups assigned 2-3 Milestones each to develop instruments or methodology Large group review and feedback Education office worked with Evalue to incorporate tools into system

General Strategy “Good now is better than perfect later….” Priorities Ease of use Should not be burdensome for residents, faculty, RECCC or education office staff No PAPER ! App for ‘on-the-go’ assessments - procedures, H&P Implementation feasibility Ready by July 2014 Integrate with existing system (Evalue) Ensuring adequate data to assess Milestones comprehensively A Milestone for every competency 17/25 Milestones chosen Work in progress, as refinements will be made and new tools added over time

Which Milestones did the AMIGOS select?

Medical Knowledge (MK): 1 MK1- Knowledge of biomedical, clinical, epidemiological, and social sciences as outlined in the American Board of Anesthesiology Content Outline ITE AKT ABA Basic Examination ACLS certification Direct Clinical Observation

Patient Care (PC): 8 of 10 PC1-Preanesthetic Evaluation, Assessment, and Preparation PC2- Anesthetic choice and conduct PC3- Periprocedural pain management PC4- Management of perianesthetic complications PC5- Crisis management PC6- Triage and management of critically ill patient in a non- operative setting PC7- Acute, chronic, and cancer related pain consultation and management PC8- Technical skills: Airway management PC9- Technical skills: Monitoring and Equipment PC10-Technical skills: Regional anesthesia

Example of PC-8 Assessment PC-8 (Technical skills: Airway management) Direct Clinical Observation (DCO) Daily evaluation form Airway management checklist Objective Structured Clinical Examination (OSCE) / Objective Structured Assessment Test (OSAT) Simulation / Standardized Patient (SP)

Professionalism (Prof): 3 of 5 Prof1- Responsibility to patients, families, and society Prof2- Honesty, integrity, and ethical behavior Prof 3- Commitment to institution, department and colleagues Prof4- Receiving and giving feedback Prof 5- Responsibility to maintain personal, emotional, physical and mental health

Example of Prof-1 Assessment Prof 1 (Responsibility to Patient, Family, and Society) 360 degree evaluations Patients Healthcare team members Peer evaluations Administrative staff Direct Clinical Observation (DCO) Daily evaluation form Written comments

Interpersonal and Communication Skills (ICS): 2 of 3 ICS1- Communication with patients and families ICS2- Communication with other professionals ICS3- Team and Leadership skills

Practice Based Learning (PBL): 2 of 4 PBL1 Incorporates quality improvement and patient safety initiatives into personal practice PBL2- Analysis of practice to identify areas in need of improvement PBL3- Self-directed learning PBL4- Education of patient, family, students, residents and other health professionals

Systems Based Practice (SBP) 1 of 2 SBP1- Systems-based approaches to patient care SBP2- Coordination of patient care within the healthcare system

How will we Assess Milestone Levels? Milestones are a summary of how a resident is progressing based on information from other sources Milestones are not assessment tools to be used for gathering detailed information They do not replace end-of-rotation forms, simulation, 360’s, Daily evaluations etc. More pieces of data allow for more precision There will be a minimum number of evaluations required for procedural assessments The Milestones will ultimately be a summary of the information gathered by several evaluations. Some subcompetencies may not be measured on every rotation or every experience. For example, if residents must complete one quality improvement project or one scholarly activity during residency, there may be just a single assessment during the program for that project. Some milestones may be difficult to measure with existing assessment tools or forms, and new methods may have to be developed. For example, a resident might need to serve on a quality committee with an assessment by the committee chair to address certain questions. An individual program may not have collected this information before, so a new assessment tool to address the milestones for this activity might be necessary.

Clinical Competency Committee (RECCC) Self Evaluations Case Logs Mock Orals Professionalism Indicators like pharmacy, billing, evals etc End-of-Rotation Evaluations Unsolicited Comments ITE Sim Lab Student Evaluations Nursing and Ancillary Personnel Evaluations Clinical Workplace Evaluations Clinical Competency Committee Overall the goal is to have the CCC review input from different sources. Assessment methods might include the evaluations that are currently being used by programs. Not all of the evaluations of one resident will be consistent, so the role of the CCC will be to consider them all and make an overall decision on how the resident is progressing, which milestones have been met, and which to continue to work toward. OSCE Peer Evaluations Patient/ Family Evaluations Assessment of Milestones

Anesthesiology Milestones Level 4 is the Graduation Target Not every resident will achieve Level 4 in every Milestone Residents required to substantially meet most Milestones at Level 4 Residents are not expected to achieve Level 5 during residency Residents may achieve a level of competency in specific Milestones sooner than expected Milestones permit the identification of outstanding performance exceeding what is expected and usually observed in a resident

Milestone Template This is what the five levels look like in the tabular form that will be used for Milestone reporting. Note that the form offers check-off boxes between Levels, recognizing that a resident might be best described at Level 2.5, for example, if he or she has met most of the Level 2 and some but not all of the Level 3 milestones. Level 5 milestones are aspirational goals that only a few exceptional residents will reach, but that not every resident would be expected to attain. Level 4 describes the graduation target, but every resident does not need to reach Level 4 in every competency or subcompetency in order to graduate. It is still the purview of the program director to determine if and when a resident is ready for independent practice.

Developmental Progression or Set of Milestones General Competency Developmental Progression or Set of Milestones Subcompetency Milestone PC1. Preanesthetic Evaluation, Preparation Level 1 Level 2 Level 3 Level 4 Level 5 Performs comprehensive H&P Identifies medical issues that may affect anesthesia care Identifies elements and process of informed consent Identifies medical issues relevant to anesthesia care Optimizes preparation of non-complex patients Obtains informed consent but recognizes when assistance is needed   Identifies medical issues relevant to subspecialty anesthesia care Optimizes preparation of complex or subspecialty patients Obtains informed consent of patients with complex problems or requiring subspecialty anesthesia care Performs assessment of complex or critically ill patients with conditional independence Obtains informed consent in complicated clinical situations with conditional independence  Independently performs assessment for all patients Independently serves as a consultant to other members of the healthcare team Consistently ensures that informed consent is obtained by using all available resources It will help to agree on some terminology. The “General Competency” refers to one of the six Core Competencies. In this example, “PC” is for “Patient Care”, one of six Core Competencies. “PC1. History (Appropriate for age and impairment)” is a subcompetency under the first of several subsections of “Patient Care”. A Milestone is actually the description of how a resident is doing and the entire set of milestones for one subcompetency can be called a Set of Milestones. Notice that the Milestone narratives do not indicate level of education or the years in the program but by levels merely to signify that the next Milestone is more advanced than the level below it. Residents are expected to progress at different rates.

Instruments to Measure Milestones Daily Evaluation Form for OR (Evalue) Daily Evaluation Form for ICU (Evalue) Daily Evaluation Form for Pain (Evalue) Technical Skills forms (Evalue) (‘App’?) Arterial line CVP Line Nerve Block / Epidural / Caudal Airway Management Peer evaluations Assessment of Patient Interaction Regional assessments Simulation assessments

‘QUASAR’ Project: Quality in Anesthesiology, Systems Assessment and Research

Resident ‘QUASAR’ Project: Quality in Anesthesiology, Systems Assessment and Research Quality Improvement Systems Based Practice Practice-Based Learning CA1- Background research, identify area of study CA2- Identify quality measure to be studied and change to be implemented, study effects, refine CA3- Write summary, submit abstract to national conference, or write a manuscript for publication

Summary Next logical extension of competency-based evaluation Mostly familiar, not burdensome Levels 1 and 2 do NOT signify a ‘Fail’ grade Evaluations will be more specific and focused Direct clinical observation to evaluate procedures and patient interactions

We Need Your Data ! APOM NEEDS YOU

Acknowledgements AMIGOS Nicole Conrad Oumou Diallo Judy Freeman Julio Gonzalez Karen Hand Izumi Harukuni Amy Miller Juve Diana Kim Ed Kahl Dean Lao David Larsen Kim Mauer Michele Noles Annie Riley Peter Schulman Katie Seligman Chris Swide Linda Wylie David Wilson Glenn Woodworth