Program Directors Survey How are we evaluating our fellows? Ann Zimrin, MD University of Maryland Greenebaum Cancer Center

Disclosures for Ann Zimrin, MD Employment none Consultancy none Equity Ownership none Research Funding none Honoraria none Patents & Royalties none Speakers Bureau none Membership on Board of Directors/Advisory Committee none Other none Presentation includes a description of the following off-label use of a drug or medical device none In compliance with ACCME policy, ASH requires the following disclosures to the session audience: 52 nd ASH Annual Meeting ♦ Orlando, FL

Why do the survey?

Original survey SHARING BEST PRACTICES SURVEY General Information Name of program: Name of program director: Number of fellows per year: Number of months clinical work on average (out of 24 or 36) during fellowship: Strategy for assessing competencies CompetencyMethods Used Patient Care Medical Knowledge Practice-based Learning Interpersonal and Communication Skills Professionalism Systems-based Practice How do you assess a fellow’s capacity for analytic thinking and its application to patient problems?

Modified survey What evaluation methods are you currently using? Mini-CEXs Faculty global ratings 360 degree evaluations Faculty feedback on presentations (such as journal club, grand rounds, research presentations) Portfolios Procedure logs In-service exams (ASH, ASCO) QOPI Simulation center (patient interviews, physical exams) Simulation center (procedures) Case review conferences Anything else? How do you assess a fellow’s capacity for analytic thinking (optional)?

Who responded? Adult programs – Hematology/Oncology combined 69/126 (55%) – Hematology 3/7 (43%) – Oncology 4/16 (25%) Pediatric Hematology/Oncology 29/65 (45%)

Adult programs

Pediatric programs

Responses by Region

Evaluation Methods

Other Methods (Adults) A set of cases for which the fellows provide “consultation” report with critical analysis of the issues and relevant literature (q 6 mos) Written test and demonstration of manual competency in administrating chemotherapy Chart reviews by the faculty of the fellows’ notes; review of fellows’ overnight triage log Chart stimulated recall oral exams

Other Methods (Peds) Individualized Learning Plans Scholastic Oversight Committee presentations “Pretending” month as third year

Wish List On-line teaching modules concerning ethics, delivery of bad news, cultural sensitivity Better ways to mentor and assess research aspect of fellowship Simulation models for bone marrow biopsies

New ACGME Requirement “Fellows must have access to training using simulation.” “The committee does NOT expect each program to use a simulator or have a simulation center. Simulation means that learning about patient care occurs in a setting that does not include actual patients. This could include OSCEs, standardized patients, patient simulators, or electronic simulation of codes, procedures, and other clinical scenarios.”

Simulation – OSCE Objective Structured Clinical Examination – Students rotate from station to station – Interview patients, interpret lab findings and imaging – Observed by examiners and performance scored Useful way to test students’ grasp of common topics (“Approach to the Patient with Chest Pain”, for example)

OSCE - Application to Heme/Onc Possible topics – Approach to the patient with thrombocytopenia – Approach to the patient with anemia – Approach to the patient with a mediastinal mass – Gerontology issues Disadvantages – Expensive – Time-consuming to set up – Questionable utility

Computer-Based Simulation Schick and Burke (The Internet Journal of Hematology 2010 : Volume 7 Number 1) – Computer-based tutorial on the topic of bleeding disorders – After reading history, residents and fellows were asked to order tests from a menu and make recommendations for treatment – Authors noted that selection of diagnostic tests was the weakest area

Lab-based Simulation Insertion of CVP catheters Administration of intrathecal chemotherapy Bone marrow biopsy simulation – Sternal aspirate More on simulation in upcoming breakout session

Tools: Where to Look for Help ASH Website – Presentations from previous TPDWs Grants Clearinghouse Hematology curriculum Case studies for fellows (board-type questions with answers and references) Image Bank (slides to download or print) Teaching Cases for Medical Students Medical Student Hematology Course Learning Objectives

Tools: Where to Look for Help ASCO Website – Training Program Director sessions from previous annual meetings – Oncology Training Program Directors Retreat – presentations and extra material submitted by PDs – Medical Oncology Core Curriculum – Communication Skills – “Works for Me” – ASCO Tumor Boards

Tools: Where to Look for Help The American Society of Pediatric Hematology/Oncology – Site for training program directors (for sharing information, documents, links and files in an organized, searchable library) – password protected

Tools: Where to Look for Help European Hematology Organization ( – Passport (a hematology curriculum) – On-line cases ACGME – Outcomes Project Toolbox Educating Physicians for the 21 st Century RSVP (Recognize Success Via imPlementation)

My Conclusions Although many evaluation methods are used, faculty feedback is the cornerstone of the evaluation in essentially all programs surveyed Faculty development to make faculty better observers would be useful Simulation not yet incorporated into most programs Resources are available on-line