1. Addictions Foundation of Manitoba Developing an evaluation of an accountability process in a provincial addiction agency David Patton, PhD. Addictions Foundation of Manitoba

2. Overview Goals of the presentation Inform process Present a model for discussion Offer insights into our process

3. Addictions Foundation of Manitoba AFM structure

4. Addictions Foundation of Manitoba Planning Teams Developed to provide consistent direction for each program area Management representative (SMR), chair usually supervisor of the program, staff Work together to develop accountability model – one of their goals (logic model?) Research Unit used as facilitator/guide to the process Reviewed by Management Team re: consistency with organizational goals

5. Addictions Foundation of Manitoba Developing Accountability Models Begin by asking staff a series of questions about programs (to generate content of each model). Who are the target clients? Assessment process (how do we determine that the program is appropriate?) Identify short-term goals (by the end of their program) How can we tell when a client has achieved these goals? What activities do staff engage in to achieve the short term goals?

6. Addictions Foundation of Manitoba Identify long-term goals (3 – 6 mths) How can we tell when a client has achieved these goals? How can we link short term goals with long term goals? How are these consistent with AFM’s mandate? Developing Accountability Models

7. Addictions Foundation of Manitoba Model of Program Accountability Staff-Client Activities Short-Term Objectives Long-Term Goals

8. Addictions Foundation of Manitoba Logic model developed Different from the accountability models Not program based More top-down (based on business plan) Link with strategic directions (staff also have input into this process)

9. Addictions Foundation of Manitoba Goals within planning teams Identify expected outcomes for programs Link these to accountability models Develop (with research group) measures/research protocols Continuous Improvement loop (feedback & pilots) Use feedback/pilot data to improve programs – not here just yet

10. Addictions Foundation of Manitoba Identify expected outcomes for programs (examples) Increased Knowledge about AODG (as appropriate) Better coping skills Realistic rehab plan Reduced harmful use (may include abstinence)

11. Addictions Foundation of Manitoba Link these to activities What is it that the staff do? Mostly counseling/education Some planning around process in clients’ lives Group sessions with speakers (e.g., financial planners for residential gambling program) Link with support groups

12. Addictions Foundation of Manitoba Develop measures/research protocols How would be be able to tell if clients are half way towards meeting their goals? – staff can tell us. Sometimes standardized measures can be used, sometimes we create our own (e.g., SFI) NOT client satisfaction measures

13. Addictions Foundation of Manitoba Continuous Improvement loop

14. Addictions Foundation of Manitoba Use data to improve programs and services Mostly pilot projects – evaluation reports used to convince partners to continue to fund (e.g., school-based projects, new gambling residential rehabilitation program) Starting some work with family therapists Services also being evaluated – e.g. library brochures, media links

15. Addictions Foundation of Manitoba Example of Model Show video Teaching Sessions 1 to 1 discussion Understanding the impacts of gambling Increased Knowledge about how gambling works Reduced Harm from Gambling Short-Term Objectives Staff-Client Activities Long-Term Goals

16. Addictions Foundation of Manitoba Example: Gambling Residential Program Long-Term Goals, Short-Term Goals, and Program Activities Pre-and Post-Tests for Short-Term Goals Several indicators for each main short- term objective Benchmark for Long-Term Goals These are asked of the client again 3 to 6 months after program involvement

17. Addictions Foundation of Manitoba Benefits of Program Accountability Models Program development and Evaluation Identify specific gaps in agency’s program offerings Job descriptions are more clear Referring agencies can be made aware of each program’s specific activities and objectives

18. Addictions Foundation of Manitoba Practical issues for counseling-based programs Mechanism for ongoing program improvement Contribution to Best Practices literature Issues and challenges

19. Addictions Foundation of Manitoba Summary Management buy in to process Planning teams (include staff, supervisors and management rep) Develop trust with line staff Listen to their input Work with the staff to develop models

20. Addictions Foundation of Manitoba Thank you Comments/questions?