1. Using Program Data to Improve Services Kristin Stainbrook, Ph.D. Advocates for Human Potential, Inc. Albany, NY The Center for Substance Abuse Treatment National Conference Arlington, VA *October 22 nd, 2009

2. Goal of Presentation  Client data can be an important tool for improving program services  As the evaluator for several substance abuse and mental health services programs, we have created a feedback loop between evaluation activities and the program  This presentation will provide examples of ways we have employed data to assist with program management and quality assurance

3. AHP Relationship to Programs  AHP is a research and consulting firm  Worked with range of programs including: Substance Abuse Street Outreach; Trauma program for homeless families; HIV/Hep-C Education Program; Supported Housing and Supported Employment  In some cases we are the local evaluator to fulfill federal evaluation requirements, in others we are hired to help programs implement processes to measure outcomes

4. The ability to use data based is on established expectations  Program views Evaluators as a partner  Evaluation/data collection is not seen just as a requirement but a tool  Frequently review data/findings  Keep the data simple  Engage in discussion/interpretation of data at all staff levels  Ask staff about what else they would like to know

5. Mechanisms to provide feedback  Share data in management meetings, staff meetings on a regular basis  Prepare monthly service reports  Quarterly Client Focus Groups  Annual Staff Interviews  Annual Evaluation Report/Presentation

6. Data as an Indicator of Problems  Data can provide quick snapshot – and be used as a red flag for problems that we can explore more deeply through other methods  Intake Coverage Reports Enrollments down for several months  Through discussions with various staff found eligible people not acknowledging trauma in the screen Added monthly trauma information session and trauma specialist visits to shelter- enrollments increased

7. Sample Active Client Data Tables MonthEnrollmentsDischargesActive ClientsAverage Case Load/ 2.5 FTE May 1023313 June 1034016 July 654116 August 534317 September 574116 TOTAL/ Average 36194016

8. Using Data for Program Modifications  Length in program data (Trauma Program) Found some clients dropping out within a few months of intake- many with few contacts  Used Client focus groups to explore issue Changed process for introducing clients to program- including more outreach to shelters  Participation data (HIV/Hep-C Education) Found large number of clients not “graduating” because missed a session  Used staff interviews to explore issue Revamped program to offer a “make-up” session- and provide more time for staff follow-up/reminders

9. Using Data for Supervision  Staff service reports Created “Dose Log” that collects information on services provided by staff From data, we prepare monthly reports that detail the amount of services received in the past month by client, and the amount of services by staff Use in case conferences to discuss clients that are difficult to engage Used in supervision- helps identify staff who have a large number of clients who are not engaging in program  Descriptive and outcome data Present at Staffing retreats

10. Sample Monthly Dose Log Report Client Name Date Entered the Program Nature of Contact by Case Managers Contact with Trauma Specialists In person By Phone -not msgBy mailAttempt Other type of contact Trauma Groups Individual Session CLIENT A6/21/20091010000 CLIENT B6/29/20098601252 CLIENT C7/10/20096202040 CLIENT D7/19/20095002000 CLIENT E8/1/20093201032 CLIENT F9/5/20090013000

11. Using Data for Quality Improvement  Collect information on client outcomes We have Client interviews that include questions on Program Satisfaction and Working Alliance Questions, as well as outcomes  Quarterly Client Focus Groups Identify current issues: confidentiality in groups; perception of favoritism by clients  Annual staff interviews Ensure perspective of all program staff is heard

12. Evaluation supports sustainability  Presentation of data and sharing reports with key stakeholders Increase program visibility and share positive results  Process Evaluation- key informant interviews with stakeholders Use as a tool to develop/improve relationships and reinforce presence in the community

13. Summary  Relationship between evaluation and program is foundation for using data for quality improvement  Regular sharing of data so everyone can see changes- value Data does not need to be sophisticated- very simple tables work best  Engage in discussion with staff at all levels so that they understand what the data means and how it can work for them