Confidential – For Discussion Purposes Only Conflict of Interest  Ann Dohn – No conflicts of interest to disclose  Nancy Piro – No conflicts of interest to disclose

SES072 Mission NOT Impossible – Using Data to make You and Your Program Shine! Ann Dohn, MA DIO & GME Director Nancy Piro, PhD Program Manager/Education Specialist

Confidential – For Discussion Purposes Only Session Outcomes Participants will be able to:  Understand the many varied sources of data elements and how they can be organized and used in both resident (CCC) and program (APE) evaluations  Develop and utilize dashboard/report card structured tools to track resident performance and program performance across all aspects of their training requirements, both current and longitudinally  Identify ways to customize and adapt to individual program and institution needs  Take home tools to facilitate the implementation of these objectives

Confidential – For Discussion Purposes Only Overwhelmed by Data? Data elements can be organized and used in both resident (CCC) and program (PEC / APE) evaluations

Confidential – For Discussion Purposes Only NAS Data and YOU !  The program coordinator plays a crucial role in developing, implementing, collecting data, reporting milestone data, program evaluation data…

Confidential – For Discussion Purposes Only Let’s Break it Down…Different data elements for different purposes:

Confidential – For Discussion Purposes Only Do you provide the CCC members any evaluation data to review before the meeting? 7

Confidential – For Discussion Purposes Only Focusing on the data for your CCC (Clinical Competency Committee)

Confidential – For Discussion Purposes Only Pulling the Data Together Clinical Competency Committee End-of- Rotation Evaluations Safety Incident Reports Case Logs Patient/ Family Evaluations Clinical Skills Assessment Nursing and Staff / Techs Evaluations Progress on Milestones Sim Lab Sim Lab In-service training exams Quality Improvement Activities

Confidential – For Discussion Purposes Only Data used in CCC meetings for trainee assessment 10

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Step 1 – Defining what to track

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Step 1 – Defining what to track

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Step 1 – Defining what to track

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Step 1 – Defining what to track

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Step 1 – Defining what to track

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Step 1 – Defining what to track

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Step 2 - Link data sources to milestones

Confidential – For Discussion Purposes Only Milestone Data Translation to Numerical Data

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Step 3: CCC defines performance ranges At or Above Expectation: 2.8 and higher Below Expectation: 1.7 – 2.7 Remediation: Below 1.7 Example: For all aggregate milestone evaluation scores for a PGY 3, the CCC defines these ranges:

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Step 4 – Set conditional formatting

Confidential – For Discussion Purposes Only Example: Aggregate milestone evaluation data cells Highlight cells to apply the conditional formatting Creating a Resident Performance Profile Step 4 – Set conditional formatting

Confidential – For Discussion Purposes Only Click on “Conditional Formatting” “Highlight Cells Rules” Creating a Resident Performance Profile Step 4 – Set conditional formatting

Confidential – For Discussion Purposes Only Select, “Greater Than” “Less Than” or “Between” to Set Value Ranges Choose the corresponding fill color (e.g., red, yellow, green) Creating a Resident Performance Profile Step 4 – Set conditional formatting

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Compiling and centralizing data

Confidential – For Discussion Purposes Only Resident Performance Profile: Step 5: Enter in data

Confidential – For Discussion Purposes Only Creating a Resident Performance Profile Visual Trends and Detailed Data

Confidential – For Discussion Purposes Only There was no way we could have had an effective CCC meeting without completing pre-work. 27

Confidential – For Discussion Purposes Only Other Resources to make you shine!

Confidential – For Discussion Purposes Only Leveraging Resident Management System (RMS)Tools, if Available  RMSs – becoming more feature rich − CCC Online Functionality – Assign Files, Upload all required data for Preliminary Review, etc − Curriculum  Goals and Objectives and learning outcomes by rotation  Teaching and Assessment methodologies − Evaluation tool development  Sharing between programs and institutions

Confidential – For Discussion Purposes Only Leveraging Resident Management System (RMS)Tools, if Available  Conference attendance statistics − Core competencies linked to specified conferences − Attaching conference materials for later reference  Procedures and levels; linked procedure evaluations

Confidential – For Discussion Purposes Only Resident portfolio tools QI participation and outcomes Scholarly Activity logs Leveraging Resident Management System (RMS)Tools, if Available

Confidential – For Discussion Purposes Only Leveraging Resident Management System (RMS)Tools, if Available  Aggregate reporting and graphic summaries  Peer or departmental average, individual average, minimum and maximum scores, standard deviation or listing of all scores

Confidential – For Discussion Purposes Only Leveraging Calendaring and Task Management Software  Set “data gathering and reporting” appointments with yourself  Remember to start your timeline with the end in mind (e.g., CCC meeting dates)  Break down large tasks into smaller tasks to keep it manageable

Confidential – For Discussion Purposes Only Leveraging Calendaring and Task Management Software

Confidential – For Discussion Purposes Only PGY 1 Ranked at Graduation Level 1

Confidential – For Discussion Purposes Only When Remediation is Evident

Confidential – For Discussion Purposes Only Switching gears to APEs CCCs APEs

Confidential – For Discussion Purposes Only Where do I begin?

Confidential – For Discussion Purposes Only Program Evaluation Committee (PEC) Must Monitor and Track (V.C.2): 1.RESIDENT PERFORMANCE 2.FACULTY DEVELOPMENT 3.GRADUATE PERFORMANCE 4.PROGRAM QUALITY 5.PROGRESS ON THE PREVIOUS YEAR’S ACTION PLAN

Confidential – For Discussion Purposes Only RESIDENT PERFORMANCE  The most recent aggregated written evaluations of the residents submitted by faculty and other evaluators  In-training/In-service exam scores  Procedure logs (if applicable)  Scholarly activity (publications, presentations, grant awards, etc.)  Learning portfolios: documented quality improvement activities

Confidential – For Discussion Purposes Only FACULTY DEVELOPMENT  ABMS certification status for all faculty  Updated faculty CVs  Documentation (faculty survey; attendance logs) of faculty participation in: − CME-type activities directed toward acquisition of clinical knowledge and skills and also activities directed toward developing teaching abilities, professionalism, and abilities for incorporating the core competencies into practice − Teaching (conferences, grand rounds, journal clubs, lecture- based CME events, workshops, directed QI projects, practice- improvement self study).  Faculty actively involved in mentor relationships with residents/fellows.

Confidential – For Discussion Purposes Only GRADUATE PERFORMANCE  Aggregated board exam pass rates  Aggregated alumni survey results (typically, such surveys target physicians one year and five years after graduation – survey questions may inquire about such items as current professional activities of graduates and perceptions on how well prepared they are as a result of the program)  Other outcome measures − Practice location (underserved areas) − Academic Affiliations − Scholarly Activity

Confidential – For Discussion Purposes Only Alumni Surveys …

Confidential – For Discussion Purposes Only PROGRAM QUALITY  The most recent aggregated written evaluations of the program submitted by faculty  The most recent aggregated written evaluations of the program (and/or specific rotations) submitted by residents  The most recent aggregated written evaluations of the faculty submitted by residents  Faculty’s recent scholarly activity (publications, presentations, grant awards, etc.)

Confidential – For Discussion Purposes Only PROGRAM QUALITY - Continued  The most recent ACGME survey results  The most recent GME House Staff survey results  The most recent GMEC Internal Review Report  Any recent communications from the ACGME or RRC  Program Report Card/Scorecard − Trend Analyses

Confidential – For Discussion Purposes Only PROGRAM QUALITY - Continued  Curriculum  Overall and rotation-specific goals and objectives (Are they appropriate? Do they align with the core competencies?)  Didactic curriculum (Is there at least one regular conference targeted to the residents’ level?)  Opportunities for scholarly activity  Compliance with any new standards established by the ACGME, RRC, ABMS, etc. − Assessment Methods (Are evaluation tools appropriate? Do they align with the core competencies?) − Resources:Personnel (PD, PC, faculty), Affiliated Training Sites, Patient/Procedure Volume,  Learning Environment (space, call rooms, books, computers, etc.)

Confidential – For Discussion Purposes Only PROGRESS ON THE PREVIOUS YEAR’S ACTION PLAN Review progress / (attempts to resolve problems) with respect to last year’s Annual Review delineating identified areas of weakness.

Confidential – For Discussion Purposes Only Diamond in the Rough…

Confidential – For Discussion Purposes Only Making Your Program Shine  Use your data !! − ADS Updates  Major Changes − Applicants − Annual Reports − Scorecards − Monitor Program Improvement − Monitor Trends

Confidential – For Discussion Purposes Only Report Card Trending Example

Confidential – For Discussion Purposes Only How Can You Build a Scorecard? Easier than it looks!

Confidential – For Discussion Purposes Only External Measures

Confidential – For Discussion Purposes Only Internal Measures

Confidential – For Discussion Purposes Only Web ADS

Wednesday, June 11, 2014 PEDIATRIC GASTROENTEROLOGY ANNUAL REPORT Annual Reports

Confidential – For Discussion Purposes Only Managing it all: How will I pull this off?

Confidential – For Discussion Purposes Only Use Technology to Your Advantage…RECAP Know your program requirements and follow them unconditionally Use simple spreadsheet, calendaring and task organizational tools to manage, track and present resident performance data to your CCC Resident education is a cyclical process – revisit and revise tools and processes each year

Confidential – For Discussion Purposes Only Asking Why??  Continue to ask: “Why am I doing this??  Is there a better way…”Five Why’s”  Then streamline – no use collecting data that won’t be used i.e., “useless”

Confidential – For Discussion Purposes Only The Toolbox

Confidential – For Discussion Purposes Only Tools Can Be GME Community Templates – A3

Confidential – For Discussion Purposes Only Tools

Confidential – For Discussion Purposes Only Questions

Confidential – For Discussion Purposes Only Contact Information:  Ann Dohn:  Nancy Piro: