2012 Desiree Nangle, Evaluation Project Manager

SIMPLE COMMON SENSE PLAIN LANGUAGE MINIMUM PAPER USEFUL

With thanks to The Fiscal Policy Studies Institute Santa Fe, New Mexico

What is Results Based Accountability? Starts with ends and works backwards to means –Ends are how customers are better off when the program works the way it should RBA is meant to get you from talk to action quickly Used for continuous program improvement Inclusive process

Results Based Accountability is made up of two parts: POPULATION ACCOUNTABILITY Is about the well-being of WHOLE POPULATIONS For Communities – Cities – Counties – States - Nations PERFORMANCE ACCOUNTABILITY Is about the well-being of CUSTOMER POPULATIONS For Programs – Agencies – Services August, 2012

7 Performance Accountability Questions 1.Who are our customers? 2.How can we measure if our customers are better off? 3.How can we measure if we are delivering services well? 4.How are we doing on the most important of these measures? 5.Who are the partners that have a role to play in boing better? 6.What works to do better, including low- cost and no-cost ideas? 7.What do we propose to do?

7 Performance Accountability Questions 1.Who are our customers? 2.How can we measure if our customers are better off? 3.How can we measure if we are delivering services well? 4.How are we doing on the most important of these measures? 5.Who are the partners that have a role to play in boing better? 6.What works to do better, including low- cost and no-cost ideas? 7.What do we propose to do?

How much service did we deliver? Performance Measures How well did we deliver it? How much change / effect did we produce? What quality of change / effect did we produce? Quantity Quality Effect Effort

How much did we do? How well did we do it? Is anyone better off? Quantity Quality Effect Effort Skills and Knowledge Attitude and Opinion Behaviour Circumstance Client Satisfaction Standards Unit Costs Wait List Staff Quality Targets Number Clients Number Activities Types of Measures Found in Each Quadrant

How much did we do? How well did we do it? Is anyone better off? Quantity Quality Effect Effort  % of parents with more of a child focus o assessed using client self-reported perceptions in questionnaire o % of parents with less parent to parent conflict o % of parents with decreased litigation o Assessed via Family Court feedback questionnaire o % of clients seeking other services o % of parents with improved ability to co-parent o % of parents with improved self-awareness  Number of groups  Number of clients  No shows/Drop out rates  Bookings for intake  Intake group  Group to completion  Level of satisfaction Family Court has with MDF  Staff satisfaction  Reconnects Mums and Dads Forever 2012

How much did we do? Not All Performance Measures Are Created Equal How well did we do it? Is anyone better off? Least Important Quantity Quality Effect Effort Most Important Least Most Also Very Important

How much did we do? The Matter of Control How well did we do it? Is anyone better off? Quantity Quality Effect Effort Least Control Most Control

Program : _______________ Performance Measure Baseline Story behind the baseline Partners Three Best Ideas – What Works No-cost / low-cost ONE PAGE Turn the Curve Report: Performance Off the Wall

Example of a Turn the Curve Report

Results Based Accountability Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have. Defined Service Users All Anglicare WA staff Data Development Agenda 1.Determine population outcomes our services contribute to 2.Determine whether or not there are questions we can ask across services that use the same wording and same scale Headline Performance Measures % of staff who understand what RBA is % of programs in Phase 2 of RBA implementation % of programs in Phase 3 of RBA implementation % of staff who support RBA implementation within Anglicare WA Story Behind the Baselines While all four baseline indicators increased from April 2012 to August 2012, some had greater increases than others. While the percentage of staff understanding RBA increased significantly (from 54% to 74%) since April, the increase in staff supporting RBA use in Anglicare WA had a much less significant increase of only 2.5%. Part of the reason for the smaller increase is most likely due to the smaller response rate on this survey (~25% of staff). The percentage of programs in Phase two jumped considerably and progress was also made in moving services from Phase two to Phase three since April (an 8% increase). This increase in services moving to Phase three is most likely due to changes that were made in the process based on suggestions given by Mark Friedman while he was here in June. Partners Who Can Help Us Do Better All Anglicare WA Programs and Staff The Community Sector Other Not-for-Profit Organisations National and Federal Government Trained RBA Trainers Action Plan to Continue Turning the Curves In an effort to continue improving performance on our four headline measures, Anglicare WA will: Revise the length of the survey and possibly how often it is being sent out to staff in an effort to increase staff response rates Continue focusing on data that services do have to begin the process of RBA implementation and work around helping them develop tools to address each program’s data development agenda Highlight success stories and curves turned to encourage a culture of RBA monitoring among staff RBA HEADLINE PERFORMANCE MEASURES

RBA on RBA (roll out with services) 2012

How much did we do? How well did we do it? Is anyone better off? Quantity Quality Effect Effort # of active services by phases # of staff involved in active services by phase staff satisfaction with training and support # RBA facilitators trained time to self-sustainability time programs spend in each stage validity of measures validity of data % staff that understand what RBA is % staff that support RBA within Anglicare WA % services implementing RBA % services using RBA for program improvement % services with the capacity to use RBA unassisted % of curves turned 33% ( 48 out of 61 services) = 78% 66% 81% 19 in Phase % (1 service) = 2% RBA on RBA Updated: March 2013

Do staff understand what RBA is? December 2011: 53.6% Agree April 2012: 68.3% Agree August 2012: 73.5% Agree March 2013: 80.9% Agree 2013

Do staff think RBA is good for Anglicare ? December 2011: 55.2% Agree April 2012: 60.4% Agree August 2012: 66.3% Agree March 2013: 66.1% Agree 2013

Results Based Accountability March 2013 Headline Measures Report Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have. Defined Service Users All Anglicare WA staff Data Development Agenda 1.Determine population outcomes our services contribute to 2.Determine whether or not there are questions we can ask across services that use the same wording and same scale Headline Performance Measures % of staff who understand what RBA is % of services in Phase 3 of RBA implementation % of staff who think RBA will help improve their service % of staff who support RBA implementation within Anglicare WA Story Behind the Baselines Partners Who Can Help Us Do Better All Anglicare WA Programs and Staff The Community Sector Other Not-for-Profit Organisations National and Federal Government Trained RBA Trainers Action Plan to Continue Turning the Curves In an effort to continue improving performance on our four headline measures, Anglicare WA will: ………… ………….. RBA HEADLINE PERFORMANCE MEASURES

Open-ended response Challenges of RBA Opportunities with RBA 2012

Case Study YES! Housing Project

YES! Housing TURNING THE CURVE How Much Did We Do?How Well Did We Do It?  # of housed clients  # of referrals made by DCP  # of young people on the waitlist  # of allocations made by DoH  # of young people who disengage after being housed  # of days on waitlist  # of young people who complete the series of workshops  # of clients that receive a certificate for completion of ILS workshops  % of young people with case plans  % of young people who failed tenancies/eviction rate  % of qualified staff who have completed core training including: o Attending and participating in regular supervision o Proper ways training o TAS Training: Tenancy Law for Community Workers & Advocating for Public Housing Tenants o Other relevant training as identified in supervision  % of young people satisfied with the service  % of HSO from DoH who are satisfied with the program  % of clients referred to other agencies  % of waitlist population, housed population, and mainstreamed population who are: o Male/female o Indigenous o Parents  Unit Cost (cost/#of clients)  Referrals from DCP are regular and timely Is Anyone Better Off?  % of clients who access safe, secure, and stable housing at the completion of case management  % of clients that demonstrate improved knowledge of ILS skills (i.e. home, work, education, parenting, tenancy and healthy relationships)  % of clients that ustilse strategies developed through case management to address identified barriers to maintain their housing  % of clients successfully linked to appropriate specialist services  % of clients regularly engaging with their support worker to develop their case plans  % of clients who achieve set goals

YES! Housing July – December 2012: RBA Headline Program Measures Report Service Description: Anglicare WA’s YES! Housing program provides permanent housing for young people who are supported to establish independent living through long term housing, including public housing properties. RBA Headline Measures Story Behind the Baselines Graph 1: There continues to be an increase of young people on the waitlist trying to access safe, secure affordable accommodation. Unfortunately YES! Housing does not have the resources necessary to meet this ever-growing need. Graph 2: Data from graph 2 is based on the average difference from pre score to post score for the combined four ILS workshops (health, tenancy, parenting skills and home skills). The total average increase went down 1% in this past 6 months, mainly due to the health and relationships survey (deceased from a 31% increase Jan-June to 12% increase July-Dec). The Home Skills workshop increased along with Tenancy Support while the Parenting Skills results decreased in the past 6 months. Graph 3: 33.3% of clients with a case management plan mostly achieved or fully achieved their most important goal. Graph 4: 96.8% of clients engaged have a case management plan. Graph 5: 90% of clients self referred to the program or were referred through word of mouth. The remaining 10% were referred from DCP and Community agencies. Action Plan to Continue Turning the Curves In an effort to continue improving performance on our headline measures, YES! Housing will: Research other options for content, presentation methods and implementation ideas from clients participating in the Health & Relationship and Parenting workshops. Continue to implement the case management process to assist clients in achieving their most important goals through regular outreach visits. Continue to create case plans with housed and waitlisted clients through implementing the case management process and developing strategies to engage clients who have disengaged from the program. Develop stronger networks with DCP, DoH, Community Organisations, other housing organisations to offer support for their clients through case management process more accommodation options for clients in other housing situations to maintain a successful tenancy. Implement strategies developed in consultation with DCP during our recent Program Review to access alternative affordable, safe, secure housing for clients on the YES! Housing waitlist by establishing and developing relationships with private real estate. Review our current waitlist and remove ineligible clients. Partners Who Can Help Us Do Better Internal partners including Y-SHAC, Foyer, YPSG, Youth Services Manager, Fundraising External partners including Department of Housing, Department for Child Protection, Foundation Housing, Community Housing Real Estate,YACWA,, young people, Data Development Agenda Yes! Housing will work on developing and implementing new survey tools: to measure Independent Living Skills knowledge increase that is only post-workshop (not pre and post) to measure the Department of Housing’s satisfaction with the work done by YES! Housing Clients Young people years who are homeless or at risk of being homeless.

Phase 1 Awareness raising s Presentations to groups Phase 2 Service Planning Scoping and readiness assessment Workshop Define clients Agree outcomes Agree measures Data strategy Plot baselines Phase 3 Service Implementation Data collection Plot curves Monitor and use Phase 4 Normalise Refine Use unassisted Program: _______________ Performance Measure Performance Measure Baseline Story behind the baseline (List as many as needed) Partners (List as many as needed) Three Best Ideas – What Works No-cost / low-cost ONE PAGE Turn the Curve Report: Performance Off the Wall Four Phases of Implementation

What we do is an not a science. It can’t be measured What does the data say? I know I should be measuring but I don’t know how What if my program doesn’t show success? Check out these cool metrics!

The Task (1) Quadrant exercise: Performance measures How much did we do? How well did we do it? Is anyone better off?

How much did we do? How well did we do it? Is anyone better off? Quantity Quality Effect Effort Skills and Knowledge Attitude and Opinion Behaviour Circumstance Client Satisfaction Standards Unit Costs Wait List Staff Quality Targets Number Clients Number Activities Types of Measures Found in Each Quadrant