1. A Direct Observation tool for the next accreditation system
Marking the Milestones
A Direct Observation tool for the next accreditation system
Welcome audience, show of hands about where audience is at in process of integrating reporting of milestones into evaluation processes, and in particular clinic? (brief)

2. Our Team Nancy Newman, MD Praveen Tupakula, MD Susan Hasti, MD
Saif Choudhury, MD
Peg Sullivan, BA (VIP!)
James B. Anderson, PhD
Kim Petersen, MD
Dave Councilman, MD
Ajay Grewal, MD
Allyson Brotherson, MD
WE HAVE NO RELEVANT FINANCIAL RELATIONSHIPS TO DISCLOSE.

3. Presentation Outline Rationale Proposed new process
Milestone recording tool demo
Direct observation trial results
Take home lessons/ next steps
Questions from audience

4. Rationale for Project Identified needs
for more feedback to residents in clinic
to document milestone progress
to design something efficient and learner-centered
Mention HCMC push for more feedback, hole in our clinic feedback noted in general and during beta testing for ACGME milestones. And as we presented these ideas faculty strong push for attention to time and value and respectful feedback process. ISSUE OF WHETHER IT IS RESPECTFUL OF RESIDENTS IS NOT STUDIED.

5. Clinic Direct Observation Process
More faculty observation of encounters
Adapted from “Field Notes”
Focus on milestones that can ONLY be observed in clinic
Core faculty participated in identifying
Develop a computer-based tool

6. Software Tool Technical information:
Used Microsoft Visual Studio 2012 V11 (with ASP.net 4.5 framework, C# language)
Ready to use with:
MS SQL capable online web hosting site, or
IIS (Internet Information Services) enabled Local Area Network
Tool available at:
Developed by P. Tupakula, MD

7. Red ones are the milestones that must be observed directly, in room

10. Individual Resident Performance Summary

11. Pilot Study Convenience sample of faculty and residents
One faculty-resident pair per clinic session
Resident to request which milestone to focus on
Faculty to observe one patient encounter
An advantage that made administration happy was that we could often bill at a higher level when faculty went in the room. Some faculty chose to observe remotely and didn’t get this advantage.

12. Pilot Study, cont. December-February 2013-14 (8 weeks)
5 Faculty and 15 residents participated
30 direct observations
About one /day

13. Outcome Measures Impact on resident work flow
Impact on faculty precepting
Qualitative comments about the experience
Survey of all faculty and all residents (those participating in the pilot and those not.)

14. Outcome: Visit Length
Question: Did patients have longer visits with direct observation?
Number of Visits
Avg. Vitals-to- AVS duration
F- Value
P-Value
Control 23
40 minutes
Pilot
43 minutes
Control vs. Pilot
0.29
0.60

15. Outcome: Faculty Precepting
Question: Did pilot faculty precept smaller percentage of visits?
Total Patients
Precepted Patients
Percentage
Pilot
498
177 36
As-usual
533
183 34

16. Resident and Faculty Comments
Like the opportunity for 1:1 feedback
“Satisfying to give on the spot feedback”
“Enables preceptors to take a more active role in teaching”
Concern about efficiency
“The requirement to multitask”
“Tended to get behind”
Desire for simplified & clearer process
“Need a very specific limited list of criteria we observe for”
“Would be nice to streamline the process”

17. Lessons Learned It did not seem to slow residents or faculty down
Difficult with only one preceptor
Flexibility is important
Share the responsibility
Use selected milestones

18. Next Steps Maintain and expand the new process Adapt tool to:
Serve inpatient observations and other family medicine learning experiences
Fine tune data collection to aid in milestones reporting

19. Questions?