1 Meeting Milestone Teaching and Assessment Requirements for Problem-Based Learning and Improvement (PBLI) Allen F. Shaughnessy, PharmD, MmedEd Tufts University School of Medicine Department of Family Medicine Boston, Massachusetts David C. Slawson, MD The University of Virginia, Department of Family Medicine Charlottesville, Virginia

Objectives for today Describe the different aspects of the PBLI milestones List the curriculum elements necessary to develop resident competence in these areas Describe several resources available to teach the content or that provide faculty development Compare the available assessment methods available for documenting competence. 2

So... Where are we with Milestone assessment and documentation? PBLI teaching? 3

Overview of the family medicine PBLI milestones, their rationale, and their limitations 4

PBLI milestones 1. EBM: “Locates, appraises, and assimilates evidence from scientific studies related to the patients’ health problems” 2. Information Mastery: “Demonstrates self- directed learning” 3. Population Management: “Improves systems in which the physician provides care” 5

Anatomy of a milestone 6 Entering residencyEnd of residencyAspirationalDuring residency Not progressive; each level describes separate skills

Curriculum elements needed to teach PBLI content 7

EBM: Critical appraisal, which includes basics of clinical epidemiology, statistics Recognizing study design and quality, threats to validity, and critical appraisal of primary research, reviews (secondary literature), and clinical practice guidelines (tertiary literature). Relative generalizability of research (levels of evidence) and other issues of relevance and applicability. Demonstration of knowledge of different study designs, sources of bias in clinical research, and the concept of an evidence hierarchy/levels of evidence.

Information Mastery Lifelong learning skills: Identifying need for information Answering questions at the point of care Keeping up with changes in medicine

System/practice improvement Self-analysis of one’s practice System improvement and population management

So... Where are we with Milestone assessment and documentation? PBLI teaching? 11

Tour of Internet resources available for teaching PBLI, including FMDRL, STFM resources, online curricula, and blended curricula 12

Family Medicine Digital Resources Library (

Curricula Family Medicine 2012;44(4): Acad Med Jun;83(6): doi: /ACM.0b013e a5c.

BMC Medical Education 2009, 9: /9/59 BMC Medical Education 2004, 4:19

STFM Resource Library R1

STFM Resource Library R2, R3

Online Curricula

Michigan State University

20

21

22

23

24

Level 5 (your own doc) Know enough to provide basic system- based care Know one’s limits Demonstrate the ability to stay up to date find and apply the best information in the care of patients at the point of care 25

Assessment Methods EBM Fresno Test of Evidence-Based Medicine Berlin Questionnaire Knowledge, attitudes, access and confidence Online certification Information Mastery Online certification Documentation of actions Population management: Documentation similar to the MOC process

Fresno Test of EBM Basic EBM knowledge Short-answer; very difficult to grade and reproducibility is difficult Composite score does not link directly to milestones Ramos KD, Schafer S, Tracz SM: Validation of the Fresno test of competence in evidence based medicine. BMJ (Clinical research ed) 2003;326(7384):319.

Berlin Questionnaire Basic EBM knowledge and search skills Unable to get the actual questionnaire Adapted to psychiatry Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R: Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ (Clinical research ed) 2002; 325(7376):1338.

KACE (Knowledge Attitudes Confidence Access) Measure of implementation Have not used Hendricson WD, Rugh JD, Hatch JP, Stark DL, Deahl T, Wallmann ER:Validation of an instrument to assess evidence-based practice knowledge, attitudes, access and confidence in the dental environment. J Dent Educ 2011; 75(2):

Online Assessment: Information Mastery Curriculum and Assessment Program (our stuff) Assessment in development Mix of knowledge assessment, reflections, self-reported behavior Option: onsite assessment

Group discussion: reflection on content, sharing of ideas, planning for networking and support going forward 33