1. Building a culture of measurement in a large not- for-profit organisation: Findings from three and a half years of Results Based Accountability implementation Desiree Nangle, Evaluation Manager Anglicare WA 2015

2. Agenda Background RBA implementation in Anglicare WA –Case Study: RBA on RBA Learnings and Challenges Where has it worked Future Directions

3. Anglicare WA works together with people, families and their communities to enhance their abilities to cope with their lives and relationships

4. From Surviving to Thriving Surviving where life is a real battle Coping where you are just getting by Building where life isn’t bad Thriving where life is going well

5. Planning Framework Five Year Strategic Plan Vision Corporate Purpose Beneficiary Performance Indicator Key Result Areas Three Year Business Plans Strategic Service Areas Housing, Community, Finances, Relationships Corporate People and Culture, Marketing, Fundraising, etc. Operational Plans Location, Team and Service Level Results Based Accountability Service Level

6. Where Results Based Accountability fits within Anglicare WA’s Strategic Direction

7. Why build a culture of measurement and evaluation? Evaluation Culture Benefits to an Organisation Better information to support program decision making Greater capacity to evaluate the merit, cost effectiveness and cost capacity of programs Delivery of programs that are relevant and focused on delivery of outcomes to the community Hanwright, J., & Makinson, S. (2008). Promoting Evaluation Culture: The development and implementation of an evaluation strategy in the Queensland Department of Education, Training and the Arts. Evaluation Journal of Australia, 8(1), 20-25.

8. Why build a culture of measurement and evaluation? Increased capability to reprioritise in a climate of static funding Production of data to verify results that can be used for public relations and promoting services Hanwright, J., & Makinson, S. (2008). Promoting Evaluation Culture: The development and implementation of an evaluation strategy in the Queensland Department of Education, Training and the Arts. Evaluation Journal of Australia, 8(1), 20-25.

9. Why are we measuring stuff? 1.To KNOW the effectiveness of our programs and ensure our clients are achieving their OUTCOMES It’s client-centred and inclusive 2.To identify what works! 3.To identify what could work better 4.Evidencing our value To ourselves, our community, our clients, existing and future funders

10. With thanks to Mark Friedman The Fiscal Policy Studies Institute Santa Fe, New Mexico

11. Results Based Accountability is made up of two parts: POPULATION ACCOUNTABILITY Is about the well-being of WHOLE POPULATIONS For Communities – Cities – States - Nations PERFORMANCE ACCOUNTABILITY Is about the well-being of CUSTOMER POPULATIONS For Programs – Agencies – Services

12. RBA Tool 1: The Quadrant Exercise

13. Three Questions 1.How much did we do? Outputs 2.How well did we do it? Quality measures 3.Is anyone any better off? Outcomes

14. RBA on RBA Quadrant RBA Self-Evaluation How Much Did We Do?How Well Did We Do It?  # of active services by phase of implementation  # of staff involved in active services by phase of implementation  % of staff satisfied with training and support  % of staff trained as RBA trainers  Time to self-sustainability o By Service  Time programs spend in each phase of implementation  % of measures reviewed for validity  % of programs implementing RBA Is Anyone Better Off?  % of staff who understands what RBA is  % of staff who support RBA within Anglicare WA  % of staff who support RBA within their own service area  % of staff reporting RBA improves their programs  % of programs with the capacity to use RBA unassisted  % of programs using RBA for program improvement

15. RBA Tool 2: Turn the Curve Report (One-page headline measure report)

16. Displaying evaluation results in a meaningful, user-friendly way One page, headline measure reports It has: Graphs for top measures that have data Story behind curves Action plan Partners Data development agenda It is good for: Staff Managers Boards Funders

17. Headline Measure Report: Components The top 3-5 measures that services have data for make up the Headline Measure Report. These are the things believed to be most important by the service staff Yellow graphs show client OUTCOMES while white graphs show outputs (numbers of things) or quality measures (things that add quality to the service but do not necessarily mean clients are ‘better off’

18. Story behind the Baseline Story behind the baselines describes why the graphs look the way they do. They explain what went right (in that what the reasons were that clients achieved the desired outcome) as well as diagnose potential reasons some clients did not achieve the outcome

19. Action Plan The Action Plan to Turn the Curves explains what more or different (new) the service will do in the coming 6 months to try to ensure all clients receive the desired outcomes of the service. They should also focus on low-cost/no-cost ideas as well as out of the box ideas to encourage innovation

20. Partners list internal and external partners who could have a role to play in helping turn the curves. This recognises that no one person or agency can be expected to do this work alone and opens up the possibilities for collaborations – Collective Impact Data Development Agenda highlights the additional data the service currently does NOT collect but thinks would be useful to have in the future.

22. 2014 Results Based Accountability Headline Measures Report Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have. Clients All Anglicare WA staff Data Development Agenda 1.Determine population outcomes our services contribute to and begin using these across service’s headline measure reports 2.Determine whether or not there are questions we can ask across services that use the same wording and same scale Story Behind the Baselines Three of our four headline measures increased in March 2014 from their previous data period in March 2013. Graph 1 shows that more staff in March 2014 understand what RBA is compared to the March 2013 data. This is also true for Graph 3 which shows an increase in the number of programs in phase 3 of RBA implementation. These increases are due to the fact that the RBA team expanded to include a new research officer to help reach more services; do more introductory RBA sessions with staff; and increase visits to regional offices to help staff with their headline reports. Graph 2 shows a 9% decrease in staff who think that RBA will help improve their programs in March 2014 compared to 2013. This could be because of the high turn-over of staff in programs and new staff not having enough exposure to RBA to see its benefits. This could also be due to the ever-changing data collection tools that staff were being asked to use. Because of this, there was no data consistency for some of these programs and could have left staff feeling that RBA was not meaningful for their programs. Interestingly, graph 4 shows an increase (2%) in staff who think that RBA is a good thing for AnglicareWA. It is possible, that staff think that RBA is good for the organisation as a whole but is not something they should be doing themselves. RBA was seen as ‘extra work’ on top of other admin and reporting duties in the open-ended questions where staff also noted that RBA took “time away from their clients”. Partners Who Can Help Us Do Better All Anglicare WA Programs and Staff The Community Sector Other Not-for-Profit Organisations National and Federal Government Trained RBA Trainers Action Plan to Continue Turning the Curves In an effort to continue improving performance on our four headline measures, Anglicare WA will: Work in-depth with one service in each of our sub-brand areas (Relationships, Community, Youth, Housing) and use them as case studies This will help us determine if more intensive support correlates to better understanding and, more importantly, high use of the RBA framework for service improvement Increase information to all staff on RBA success stories through the intranet and staff meetings Offer more RBA introduction sessions for staff who are new or need a refresher on the basics of the framework Encourage teams to present RBA at their team meetings and across teams: integrate RBA in daily business Share more examples with staff of RBA and where it works Make RBA a part of induction to help deal with staff turnover issues Implement an award for “Curve of the Month” to highlight progress being made by service RBA HEADLINE PERFORMANCE MEASURES Number of Clients Anglicare WA has 433 staff 123 staff completed the survey, a response rate of 28%

23. IMPLEMENTATION

24. Phase 1 Awareness raising Emails Presentations to groups Phase 2 Service Planning Scoping and readiness assessment Workshop Define clients Agree outcomes Agree measures Data strategy Plot baselines Phase 3 Service Implementation Data collection Plot curves Monitor and use Phase 4 Normalise Refine Use unassisted Four Phases of Implementation

25. IMPLEMENTATION CASE STUDY: RBA on RBA

26. 2011: Start of the RBA Journey First all staff survey to get a baseline 174 staff out of ~400 completed survey Two page highlight report sent to staff

27. Year Two: 2011-2012 Adoption of the one- page headline measure report! Second all staff survey Prioritised outcomes Report to all staff and Board

28. Year three: 2012-2013 Yellow graphs to highlight outcome measures Improved formatting Strategic focus on getting services to USE results RBA results in annual report for first time!

29. Year four: 2013-2014 Development and agreement of Population outcome measures Annual report case studies MUST have RBA data to be considered

30. RBA on RBA

31. How much did we do?How well did we do it? Is anyone better off? QuantityQuality Effect Effort # of services # of staff staff satisfaction with training and support # RBA facilitators trained time programs spend in each stage % services using RBA for program improvement validity of measures % staff that understand what RBA is % staff that support RBA within Anglicare WA % services implementing RBA % services with the capacity to use RBA unassisted % of curves turned ( 61 out of 68 services) = 89.7% 68% 86% 9 75% (1 service) = 2% RBA on RBA Updated: April 2014 68 430

32. Do staff understand what RBA is? Percentage who agree or strongly agree 2011= 53.6% 2012 = 70.6% 2013 = 80.9% 2014 = 85.7% 32%

33. Do staff think RBA is good for Anglicare ? Percentage who agree or strongly agree 2011= 55.2% 2012 = 63.1% 2013 = 66.1% 2014 = 68.4% 13%

34. OPPORTUNITIES, CHALLENGES, & LESSONS LEARNT

35. Staff Comments: challenges “One of the challenges is how to change the curve when it is falling behind”. “The challenges I feel are to do with how to set up and interpret outcomes”. “What I perceive to be meaningful data and what a funding body perceives are two different things”. “It is simply another reporting tool that takes staff away from doing what we do best, working with people”.

36. “I think RBA enables AnglicareWA to prove beyond a doubt that its programmes and services are high quality and successful!” “Reflecting on our achievements provides a sense of value to our work and helps us to focus on what is important”. Staff Comments: benefits “I have truly enjoyed the reflective journey of what is a meaningful outcome/how is the person better off compared to the old way of conforming data to fit your funding providers’ KPI’s”. “I find our teams like it after they realise they own it”.

37. Outcomes thinking make sense to staff It takes time…. lots of time It needs technical support High level support is vital Data is foreign to most Data needs systems Use or rigor debate and balance Lessons Learnt

38. Its not a fad Desire for long-term outcomes but short-term funding Enthusiastic services wanting to measure EVERYTHING  need balance Don’t make funders the only audience It is worth it…. really worth it Lessons Learnt

39. Evaluation Culture Must Do’s Support from the top and the middle Use a change management model Be organised – infrastructure and technical support Be aspirational Use the results Sharing is caring  share Make it fun!

41. WHERE HAS RBA WORKED?

42. Where has RBA worked? Young Parents 1 year funding to 3 year contract YES! Housing Foyer Oxford $ $ Improved service design and delivery

43. FUTURE DIRECTIONS

44. Now What Continuing the RBA journey with services Composite RBA analysis Adding rigour Other evaluation and research tools Exploring population outcomes RBA collaboration with Collective Impact

45. www.anglicarewa.org.au Desiree.nangle@anglicarewa.org.au

46. Results Based Accountability March 2013 Headline Measures Report Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have. Defined Service Users All Anglicare WA staff Data Development Agenda 1.Determine population outcomes our services contribute to and begin using these across service’s headline measure reports 2.Determine whether or not there are questions we can ask across services that use the same wording and same scale Story Behind the Baselines Three of our four headline measures increased in March 2013 from their previous baselines in August 2012 while one remained the same at 66% (staff reporting RBA in Anglicare WA as a good thing). The increase in percentage of services in Phase 3 has been driven by availability of July-Dec data (footwork for data collection plans were set-up Jan-June 2012 with data only now being available for a full 6 month period) and regular RBA monitoring meetings with General and Regional Service Managers. One of the obstacles to moving services in to Phase 3 is likely due to the time it takes to get there. Some services are unable to get together often to go through RBA and move forward with implementation while teams simply have not had enough time to see the usefulness of the RBA process yet. Staff reporting RBA is good for Anglicare WA has remained constant at 66% in both August of 2012 and March of 2013. This flat lining is a reflection of the large amount of staff who are still undecided (~26%); with time and continued opportunities to use RBA, these undecided staff will most likely be able to definitively answer this in the near future. This result could also be a reflection of teams viewing RBA as something they have to do to “tick the box” and not as something that could be useful to both them and their clients. Teams need to see the value of reflective practice as exciting to be sure they are implementing RBA in a timely manner before losing momentum. A final reason for this result could be that coordinators are not well versed enough in RBA to make it a routine part of reflection during team meetings. To be effective, RBA needs to be an ongoing discussing and incorporated into day-to-day operations at the service level. Partners Who Can Help Us Do Better All Anglicare WA Programs and Staff The Community Sector Other Not-for-Profit Organisations National and Federal Government Trained RBA Trainers Action Plan to Continue Turning the Curves In an effort to continue improving performance on our four headline measures, Anglicare WA will: Work in-depth with one service in each of our sub-brand areas (Relationships, Community, Youth, Housing) and use them as case studies This will help us determine if more intensive support correlates to better understanding and, more importantly, high use of the RBA framework for service improvement Increase information to all staff on RBA success stories through the intranet and staff meetings Offer more RBA introduction sessions for staff who are new or need a refresher on the basics of the framework Encourage teams to present RBA at their team meetings and across teams Share more examples with staff of RBA and where it works Make RBA a part of induction to help deal with staff turnover issues Implement an award for “Curve of the Month” to highlight progress being made by service RBA HEADLINE PERFORMANCE MEASURES