A Direct Observation tool for the next accreditation system Marking the Milestones A Direct Observation tool for the next accreditation system Welcome audience, show of hands about where audience is at in process of integrating reporting of milestones into evaluation processes, and in particular clinic? (brief)

Our Team Nancy Newman, MD Praveen Tupakula, MD Susan Hasti, MD Saif Choudhury, MD Peg Sullivan, BA (VIP!) James B. Anderson, PhD Kim Petersen, MD Dave Councilman, MD Ajay Grewal, MD Allyson Brotherson, MD WE HAVE NO RELEVANT FINANCIAL RELATIONSHIPS TO DISCLOSE.

Presentation Outline Rationale Proposed new process Milestone recording tool demo Direct observation trial results Take home lessons/ next steps Questions from audience

Rationale for Project Identified needs for more feedback to residents in clinic to document milestone progress to design something efficient and learner-centered Mention HCMC push for more feedback, hole in our clinic feedback noted in general and during beta testing for ACGME milestones. And as we presented these ideas faculty strong push for attention to time and value and respectful feedback process. ISSUE OF WHETHER IT IS RESPECTFUL OF RESIDENTS IS NOT STUDIED.

Clinic Direct Observation Process More faculty observation of encounters Adapted from “Field Notes” Focus on milestones that can ONLY be observed in clinic Core faculty participated in identifying Develop a computer-based tool

Software Tool Technical information: Used Microsoft Visual Studio 2012 V11 (with ASP.net 4.5 framework, C# language) Ready to use with: MS SQL capable online web hosting site, or IIS (Internet Information Services) enabled Local Area Network Tool available at: http://milestones.fammed.org/ Developed by P. Tupakula, MD whittierfm@gmail.com

Red ones are the milestones that must be observed directly, in room

Individual Resident Performance Summary

Pilot Study Convenience sample of faculty and residents One faculty-resident pair per clinic session Resident to request which milestone to focus on Faculty to observe one patient encounter An advantage that made administration happy was that we could often bill at a higher level when faculty went in the room. Some faculty chose to observe remotely and didn’t get this advantage.

Pilot Study, cont. December-February 2013-14 (8 weeks) 5 Faculty and 15 residents participated 30 direct observations About one /day

Outcome Measures Impact on resident work flow Impact on faculty precepting Qualitative comments about the experience Survey of all faculty and all residents (those participating in the pilot and those not.)

Outcome: Visit Length Question: Did patients have longer visits with direct observation? Number of Visits Avg. Vitals-to- AVS duration F- Value P-Value Control 23 40 minutes ---------- ----------- Pilot 43 minutes ------------- Control vs. Pilot --------- 0.29 0.60

Outcome: Faculty Precepting Question: Did pilot faculty precept smaller percentage of visits? Total Patients Precepted Patients Percentage Pilot 498 177 36 As-usual 533 183 34

Resident and Faculty Comments Like the opportunity for 1:1 feedback “Satisfying to give on the spot feedback” “Enables preceptors to take a more active role in teaching” Concern about efficiency “The requirement to multitask” “Tended to get behind” Desire for simplified & clearer process “Need a very specific limited list of criteria we observe for” “Would be nice to streamline the process”

Lessons Learned It did not seem to slow residents or faculty down Difficult with only one preceptor Flexibility is important Share the responsibility Use selected milestones

Next Steps Maintain and expand the new process Adapt tool to: Serve inpatient observations and other family medicine learning experiences Fine tune data collection to aid in milestones reporting

Questions?