1. AETC PRACTICE TRANSFORMATION PROJECT: EVALUATION PLAN SUMMARY AND IMPLEMENTATION CHECK-IN August 11 th, 2016 1

2. Outline 2  Background on AETC PT Project  Internally completed evaluation tools  Evaluation tools completed by clinic staff  Optional evaluative tools  Implementation of evaluation tools  Discussion of implementation challenges

3. 3 Background on the AETC PT Project Bazilio-Bellegarde, J., Doshi, R., and Ross, P. (2015). Practice transformation project: AIDS Education and Training Centers. Presentation delivered at AETC Reverse Site Visit Meeting on November 18, 2015.

4. Intent and Goal of the AETC Practice Transformation Project 4  Intent:  To increase the size and strength of the HIV clinical workforce and to improve outcomes along the HIV Care Continuum  Goal:  To transform selected clinical practices and to build the capacity of the selected organizations to provide quality HIV care

5. Desired Outcome and Evaluation of the AETC Practice Transformation Project 5  Desired Outcome:  As a result of participation in the AETC PT Project, targeted clinics will be able to provide quality, comprehensive care and treatment to PLWH as evidenced by progress along the HIV Care Continuum  Evaluation:  Each regional AETC must monitor and demonstrate progress towards the stated goals for each participating clinical site  The regional AETCs will contribute to a national evaluation of the PT project, led by the NEC

6. 6 National AETC PT Evaluation

7. Components of National PT Evaluation 7  Internally completed forms:  One-time clinic profile  One-time target of intervention  One-time clinic dropout  Event record (each event)  Forms completed by clinic staff  Annual participant information form  Annual organizational assessment  Annual aggregate data/performance measures  Annual provider/staff survey

8. 8 Internally Completed PT Evaluation Forms

9. One-Time Clinic Profile 9  Documents confidential clinic code assignment and basic information in relation to HRSA’s inclusion criteria  Profile (9 question max/8 question min) will link the clinic code to the other instruments  Regions must keep track of the codes they assign and use the same code for the clinic throughout the project  Completed once for each participating clinic, prior to collection of the baseline organizational assessment

10. One-Time Form on Identifying Target of PT Intervention 10  Form developed in response to clarifying questions from AETCs on the PT Learning Community call  Documents whether the target of the intervention is the whole clinic, an individual unit, or several sub-sites  This form (4 question max/3 question min) should be completed for each clinic  Completed once for each participating clinic  By Sept 9, 2016 for current clinics and prior to the baseline organizational assessment for newly added clinics

11. One-Time Clinic Dropout Form 11  Documents the time invested in clinic, reasons for dropping out, and lessons learned  Eight question form completed for each clinic that has dropped out  Completed once, preferably within 3 months of a clinic formally dropping out of the project

12. FY 2016-17 OMB-Approved Event Record Form 12  HRSA to release new version by Sept 1, 2016  Documents the activities conducted at the clinic/with clinic staff  AETC staff complete this form via their respective systems  The clinic code should be documented on this form and PTP funding should be indicated on the form as well  Completed for each event/activity

13. 13 PT Evaluation Forms Completed by Clinic Staff

14. Annual FY 2016-17 OMB-Approved Participant Information Form 14  HRSA to release new version by Sept 1, 2016  Documents demographics of training/TA recipients and if service provider, their patient populations  AETCs typically have trainees complete this form on paper at the training, have a laptop/tablet at the training for entry, or utilize event registration through their respective systems  Completed annually for all training and/or TA recipients

15. Annual Organizational Assessment (1 of 2) 15  Documents what participating sites are currently doing in relation to HIV care and patient-centered care, the site’s practice infrastructure, and their capacity  Participants include 2-3 individuals able to advocate for and enact identified changes needed at the clinic and have a working knowledge of staffing, services available, policies and procedures  Annual submission with slightly modified (shortened) version collected each year after baseline

16. Annual Organizational Assessment (2 of 2) 16  PT practice coaches should complete the assessment (67 question max/59 question min) in-person with the identified individuals and use the assessment to facilitate discussion on the clinic’s needs and capacity in relation to practice transformation  Examples of success include:  Changes in staffing  Movement toward interprofessional team-based care/task shifting  Changes in the HIV services provided  Better use of EHR data  Changes in HIV-related policies and procedures

17. Annual Clinic Aggregate Data/Performance Measures (1 of 2) 17  Documents the demographics of the population served and the quality of the HIV care services delivered (per HRSA/HAB performance measures)  Clinic’s quality team and/or their quality assurance manager will most likely query the clinic’s EHR system to produce this output  There is flexibility in who completes/submits the online version of this 31 question tool  Annual submission beginning with pre-implementation (2015) data due by Oct 31, 2016

18. Annual Clinic Aggregate Data/Performance Measures (2 of 2) 18  Examples of success include:  Increased testing  Increased number of patients with HIV seen at clinic  Increased patients with viral loads testing  Improvements on Continuum indicators (5 indicators measured)  Positive changes on performance measures (9 indicators measured)

19. Annual Provider/Staff Survey (1 of 4) 19  Documents provider/staff characteristics, attitudes, and practices that may influence the uptake of training leading to transformation along the HIV Care Continuum  Providers and staff working at a participating clinic will complete the survey if they are participating in the AETC project and have/will have contact with patients with HIV  Annual data collection. Due date to be determined once survey has been finalized and disseminated

20. Annual Provider/Staff Survey (2 of 4) 20  Survey will be programmed on Qualtrics and linked to Virtual Forum (VF) to allow AETCs ability to track who completed the survey and send reminders to those that have not  To administer in VF, you would create a dummy ER like “Clinic 01234 provider survey” and then set-up an event registration for the survey  We are planning on modifying the “registration” for this survey to include only the email address, confirmation of the email address, and the first name

21. Annual Provider/Staff Survey (3 of 4) 21  For ease of administration, we recommend sending the email out to all participants and targeting reminders to those essential to your PT efforts  The NEC will also keep you abreast of the professional groups missing from your sample  We will look at changes over time among individuals and across professional groups  Looking at professional groups will reduce our reliance on linking individual records, which can be difficult given staff turnover

22. Annual Provider/Staff Survey (4 of 4) 22  Awaiting feedback from HRSA on version 2 of the survey (42 question max/17 question min)  HRSA estimates sending feedback by Aug 15 th  Based on current draft, some examples of success include:  Improved testing practices  Increased ART prescription (including PrEP and nPEP)  Increased HIV services and ability to deliver them  Improved client assessment and care  Improved team functioning

23. 23 Optional Evaluative Tool to Use at Your Region’s Discretion

24. Optional Tools 24  Patient-Centered Medical Homes Assessment (PCMHA)  36-item validated instrument created to help sites understand their current level of “medical homeness” and identify opportunities for improvement  ACRE Immediate-Post  Documents changes in knowledge with a retrospective pre- post question and self-efficacy with a post-only question  ACRE Follow-Up  Documents changes in skills and translation of skills into clinical practice

25. 25 Implementation of National Evaluation Tools and Discussion of Challenges

26. PT Participating Clinics and Data Collected Thus Far 26 Region Total Clinics RW-Funded Clinics 330-Funded Clinics Clinic Dropout* Awaiting Baseline Data Awaiting Target Data MidAtlantic844020 Midwest1349004 Mountain West83 5048 New England633133 Northeast/CR954009 Pacific1055109 South Central1138300 Southeast1376003 TOTAL 7834445936 *All RW-funded

27. How is implementation of the evaluation going thus far? Any challenges for the group to discuss? 27  MidAtlantic  Midwest  Mountain West  New England  Northeast/Caribbean  Pacific  South Central  Southeast

28. 28 Questions? Concerns? Comments?