System Performance Measurement Webinar

System Performance Measurement Webinar
Indianapolis COC- June 30, 2016 Lisa Chapman, CSH ~

HEARTH performance measures & your progress
Today we will cover… HEARTH performance measures & your progress Setting community wide performance measures Today we will talk about system performance measures and hopefully it will be a continuation of conversation. We will introduce information and some of the expectations and also plant some seeds for what the COC will need to be thinking about, moving toward and taking action on in the very near future! We could do a whole day long workshop on this and scratch the surface.

A Message from Ann Oliva
“Under HEARTH, there will be an even greater emphasis on data and the use of HMIS. CoCs should assess their data tools, counting methodologies, and HMIS and determine if changes and/or improvements need to be made.”

Background: HEARTH Goal: One goal of the newly consolidated CoC Program under the HEARTH Act is to improve data collection and performance measurement. Experience: HUD has been measuring federal progress in reducing homelessness for many years. Localize: National performance is fundamentally a reflection of local progress. CoC performance measurement will focus on local performance as a system (including ESG programs). Success: Demonstration on these key measures reflects progress toward the goal of ending homelessness community-by-community.

HEARTH Core Measures Reduce new episodes of homelessness
Reduce returns to homelessness While all of the HEARTH Act measures are important, the intent of the Act seems to us to be summarized in three key goals and measures: 1) We need to reduce the number of people who newly become homeless—this includes people who use McKinney funded programs and those that don’t – we’re talking about impacting the overall flow into homelessness. 2) We need to try to eliminate as much as possible people who return to homelessness from having been homeless before, (recidivism) and 3) we need to reduce the time that people spend being homeless before they regain housing. The HEARTH act sets a national goal that no one will be homeless longer than 30 days. Stemming entries to homelessness in the first place and shortening the length of time homeless episodes last is the only way we can meet these ambitious national goals Other measures, such as our effectiveness at reaching people and our impact in helping them increase their incomes, play an important role in creating an effective system because they are linked to our ability to meet these core goals. Reduce lengths of homeless episodes 5

Performance Evaluation 101
How do you design one evaluation for everyone? Only the monkey can pass this test, many providers feel this way about the HUD requirements. Maybe joke that if all the animals work together they might be able to do it together. So, what are important things to consider when doing performance evaluation: Standard Outcomes Realistic Benchmarks Transparent Goals

What is Performance Measurement?
Performance Measurement is a process that systematically evaluates whether your efforts are making an impact on the clients you are serving or the problem you are targeting.

HEARTH Performance Measures
Reduce number of families and individuals who are homelessness Reduce average length of time persons are homeless Reduce returns to homelessness Improve employment rate and income amount of families and individuals who are homeless Reduce number of families and individuals who become homeless (first time homeless) Prevent homelessness and achieve independent living in permanent housing for families and youth defined as homeless under other Federal statutes What funders require. Here are the HEARTH measures

Other Relevant Measures
Successful Placements in Housing Permanent Housing Retention “Service-Volume” Coverage Note: these criteria can be “measured” only to the extent they are tracked in HMIS, but HUD may ask for additional information (e.g. in a narrative) to demonstrate a community’s ability to meet the criteria.

Project Performance Thinking about project(s): What do you already measure? What do you need to begin measuring? What are the challenges & opportunities?

Systems Perspective Why measure performance across the CoC?
• Ensure common understanding of system intent and goals, along with the programs that make up the CoC • Understand how individual programs (via ‘outputs’) result in positive change for persons served (‘outcomes’) • Understand how individual programs impact overall CoC performance • Understand how well CoC prevents & ends homelessness • Identify areas for improvement Difference between system and program performance targets – Program goals: may vary depending on target population, program purpose, services, etc.; use for measuring program performance individually, compare to similar programs – System goals: reflect aggregate performance; measure of system impact; use for measuring system achievement of CoC goals, compare to other communities

HEARTH Performance Measures
Thinking about your system/community/CoC: What do you already measure? What do you need to begin measuring? How would you establish your standard? What are the challenges & opportunities?

Homelessness Assistance Projects
Evaluation Project Project Project Now we’ll look at another of the major shifts in the HEARTH Act—the shift from looking at homelessness assistance as a set of separate projects to looking at it as a system. In many communities, homelessness assistance is more of a collection of associated projects than a coordinated system. Providers and funders may work well together on new efforts or refer to one another and generally have good relations but such a relationship is not the same thing as a system. For example, in an uncoordinated system When people need homelessness assistance, they typically must contact and negotiate with individual providers or projects. The onus is on the person or household that needs shelter, to call l shelters until they find one with available space. Without a systematic approach, whether a participant is accepted into a project is determined mostly by the operator of that project and can be subjective. And, evaluation is typically focused on the results of individual projects for the people served by that project, but not on the overall impact of all projects working together on the problem. A disjointed system like this not only creates burden on clients, it is very hard to know what impact the entire system is having. It also can create data quality and performance measurement challenges. For example, one of the most important data elements under the HEARTH Act will be knowing when a household moves into permanent housing. If the system is not working together to ensure that this data is collected and reported in a uniform way, it will have a very difficult time reporting on impact. The rate of “don’t know” or missing answers can be a serious challenge to evaluating effectiveness. 13

Homelessness Assistance System
Project Project Project Evaluation Project Project Project In a coordinated system, the focus of evaluation is on the impact of all the parts working together. Each part has a role to play but are working toward common ends – the ones we described before, reducing entries and reentries and shortening the overall length of time people spend homeless in our community. A benefit of operating a cohesive system like this is that people experiencing homelessness are more quickly connected to the intervention that is going to get them to permanent housing. When the entire system is operating by common standards with a shared or transparent data system, evaluation of client outcomes is part of standard operations. Data is available about the impact of the whole system and its component parts, and performance can be improved for the entire system because projects and resource can be adjusted based on common goals and objective and readily available outcome data. 14

System Evaluation What do you anticipate are the: Challenges in doing system-wide measurement? Advantages?

System Evaluation Challenges
Project Project Project Project Project Project Project Project Right now we’ll talk a little about challenges to system-wide measurement and evaluation. Even if you’ve got a more coordinated system with accountability for outcomes there will be some challenges: First of all, there are projects that are outside of the system in some way, and they may not participate in HMIS. Secondly, in most communities there are homeless individuals who are unsheltered and not attached to any project. Traditionally, such projects and people have just been left out of performance measurement and improvement efforts. But with the HEARTH Act’s new focus on system outcomes, they will have to be brought in. In addition, your community will have to make decisions about what to track and how to track it based on the HEARTH Act. Decisions will have to be made about data quality and sharing. Currently, projects track and report information independently of system goals and needs. They do these activities based on individual project goals and performance improvement needs. Moreover, many projects enter incomplete or incorrect data in to the HMIS, and they do not effectively share client information with partner agencies. This makes it impossible to track clients across your system. Under HEARTH, however, you will have to track people even as they move between projects, for example between different shelters, or as they are in multiple projects, for example receiving rapid re-housing while they are in shelter. 16

Establishing a CoC Performance Measurement Structure
Identify & Develop Performance Indicators Set Performance Targets Measure Performance Report Progress Identify and Make Improvements We will go over these 5 elements of establishing a system performance measurement structure for your COC

#1: Identify and Define Two key questions
What impact is the CoC trying to make? What indicators reflect the impact we want? • Key Considerations: – What impact is the CoC trying to make? • Primary: prevent and end homelessness • Secondary: increase well-being, stability…during and after – What indicators best reflect and convey CoC impact and achievement of the CoCs strategic plan? – What indicators are used by CoC funders (public/private)? • Are such indicators already defined and operationalized? – What is beyond system/project control? – What projects effect/impact an indicator and can therefore also be measured on the indicator? – Are the right projects collecting the right data? – Is data quality sufficient? #1: Identify and Define

1: Examples Prevention/Diversion – Are fewer people experiencing homelessness for the first-time? Are only persons who have no safe, appropriate housing option being admitted to shelter? Incidence of homelessness – Are overall rates of homelessness declining? Is street homelessness declining? Is chronic homelessness declining? Length of stay in system, across all homeless programs. Do people stay homeless for shorter periods of time? Historical trends…

#2: Set Performance targets
How is the CoC currently performing? What is reasonable and achievable and what is our stretch goal? What might affect performance? Set Performance Targets • Key Considerations: – How is the CoC currently performing? Individual programs? – How are other communities performing on the same or similar indicators? – What is reasonable and achievable…but requires effort to achieve? – What are longer term goals and how can interim performance targets be used to help move the CoC toward goal achievement? – What targets have funders (public/private) set for the CoC? Programs within the CoC? – For individual programs, how will the target population served and the services provided to the target population affect performance? How will the local economy and housing market affect performance? – Should performance targets be adjusted for certain programs based on target population, services provided and/or other factors?

2: Examples Decrease PIT count by 30% overall (once consistent methodology used) Increase emergency shelter diversions to 20% Reduce length of time homeless to 30 days Increase income of assisted households by 25% Increase permanent housing exits to 70% Reduce recidivism to 5% Examples of targets may want to set HUD expects CoCs to also establish appropriate local targets. HUD encourages CoCs to use the national performance targets as benchmarks for which the entire CoC, as a coordinated system, should aspire to achieve, while setting local targets that account for the unique needs of the homeless population and subpopulations and other circumstances within their communities. HUD recognizes, for example, that projects that serve homeless youth may have permanent housing placement rates that are lower than projects serving other populations. Similarly, projects specifically focusing on persons who are chronically homeless may have lower employment or income performance than the system as a whole. Therefore, CoCs are encouraged to consider these types of factors when setting local performance targets so that projects serving certain populations are not penalized, but still have performance targets that they should be striving to meet. HUD will not be measuring performance by subpopulations or subsystems, so it is important that as CoCs target these components, they carefully consider whether the performance is appropriate within that context or can be improved over time.

2: Specific Example System Goal: reduce average length of time homeless to 30 days by end of 2017 Current CoC performance: 45 day average CoC Performance Targets: – 2015: 40 days – 2016: 35 days – 2017: 30 days Programs that impact this goal: Street Outreach, Emergency Shelter, Transitional Housing

#3: Measure Performance
Can you use HMIS? How often will you measure? Key Considerations: – Is HMIS system ready? • Sufficient coverage, data quality for period examined? – Have reporting methodologies been tested? • Are results valid, reliable? – How often will performance be measured? How often do funders require performance to be measured?

System Performance Measurement Introductory Guide
Released July 24, 2014 System Performance Measures: An introductory guide to understanding system-level performance measurement Really terrific resource guide. HUD has a Performance measurement page it has guides, tools, videos and a 2016 guide released this month on submitting data in HDX.

Background on System Performance
A critical aspect of the HEARTH Act is a focus on viewing the local homeless response as a coordinated system of homeless assistance options as opposed to homeless assistance programs and funding sources that operate independently in a community. To facilitate this perspective the HEARTH Act now requires communities to measure their performance as a coordinated system, in addition to analyzing performance by specific projects or project types. CoCs are required to report to HUD their system-level performance. Coordinating with ESG Program Recipients to regularly measure their progress in meeting needs CoCs are required to provide information to complete the section in the Con Plan on homeless assistance, including data on performance measures.

Implementation HUD does not expect communities to fully implement these measures until detailed specification for HMIS administrators and vendors are issued. HUD’s NOFA will provide more detail regarding how CoCs will report performance measures data to HUD HUD may also establish performance targets From what I have heard, most if not all vendors have already done this work and systems are ready.

Length of Time Individuals and Families Remain Homeless
Demonstrate a reduction of the average and median length of time (LOT) persons enrolled in emergency shelter, transitional housing, or safe haven projects experience homelessness Metric 1.1 Change in average and median LOT in ES/SH Metric 1.2 Change in average and median LOT in ES/SH/TH Client Universe Persons… In ES and SH During Current Reporting Period In ES, SH, and TH Calculation HMIS Data, # of days each person was homeless Avg = Total days / total persons homeless “High Performance” Communities <20 days OR >= 10% less than prior year for persons in similar circumstances

Extent to Which Individuals/Families who Leave Homelessness Experience Additional Spells of Homelessness Demonstrate a reduction in the % of persons who have left homelessness (i.e. exited projects into permanent housing destinations) and who return to homelessness (i.e. return to projects for which homelessness is an eligibility criterion) Metric 2a.1 Returns to ES/SH/TH after exit to PH Metric 2a.2 Returns to ES/SH/TH/PH after exit to PH Client Universe Persons… In ES, SH, TH and any PH project type Who exited (system leavers) to PH destinations During Previous Reporting Period Calculation HMIS Data add # of persons in universe Of universe, add persons who were also in ES/SH/TH at both 6 and 12 months after exit to PH date Divide total from Step 2 by total from Step 1 Of universe, add persons who were also in ES/SH/TH/PH at both 6 and 12 months after exit to PH date “High Performing” Communities <5% within the next 2 years OR Decrease of >=20% over prior year for persons in similar circumstances within the next 2 years

Extent to Which Individuals/Families who Leave Homelessness Experience Additional Spells of Homelessness Demonstrate a reduction in the % of persons who have left homelessness (i.e. exited projects into permanent housing destinations) and who return to homelessness (i.e. return to projects for which homelessness is an eligibility criterion) Metric 2b.1 Returns to ES/SH/TH after exit to PH Metric 2b.2 Returns to ES/SH/TH/PH after exit to PH Client Universe Persons… In ES, SH, TH and any PH project type Who exited (system leavers) to PH destinations During the Fiscal Year 2 Years Prior to current reporting period Calculation HMIS Data add # of persons in universe Of universe, add persons who were also in ES/SH/TH at 24 months after exit to PH date Divide total from Step 2 by total from Step 1 Of universe, add persons who were also in ES/SH/TH/PH at 24 months after exit to PH date

Thoroughness of Grantees in Reaching Homeless Individuals/Families
Narrative Questions about Coordinated Assessment, Geographic Coverage of Projects, and street outreach efforts

Overall Reduction in the Number of Homeless Individuals and Families
Demonstrate a reduction in the number of homeless individuals and families identified in the PIT sheltered and unsheltered counts and annual sheltered data within the CoC over time Metric 3.1 Change in PIT counts of sheltered and unsheltered Metric 3.2 Client Universe Persons… Counted as sheltered and unsheltered in PIT During reporting period Calculation PIT Data add # of persons in universe HMIS Data add # of persons in universe by project type Add overall unduplicated # of people in universe

Jobs and Income Growth for Homeless Individuals and Families
Demonstrate that the % of homeless adults being served in CoC Program projects increased their earned income and/or other income from year-to-year and between their enrollment in the system and their exit (or follow-up assessment) Metric 4.1 Change in employment income Metric 4.2 Change in non-employment cash income Metric 4.3 Change in employment income from entry to exit for system leavers Metric 4.4 Change in non-employment cash income from entry to exit for system leavers Client Universe Adults… In CoC Program-funded TH/PH-RRH/SSO/PH-PSH During reporting period Who exited (system leavers) Calculation HMIS data, add # of adults Of adults, add # who gained/ increased earned income during period Divide Step 2 by Step 1 Of adults, add # who gained/ increased non-employment income during reporting period Of adults, add # who gained/ increased earned income from system entry to system exit Of adults, add # who gained/ increased non-employment income from system entry to system exit

Success at Reducing the Number of Individuals/Families who Become Homeless
Demonstrate a reduction in the number of persons experiencing homelessness for the first time Metric 5.1 Change in # of persons in ES/SH/ TH with no prior enrollments in HMIS Metric 5.2 Change in # of persons in ES/SH/ TH/PH with no prior enrollments in HMIS Client Universe Persons… In ES, SH, TH During current reporting period In ES, SH, TH, PH Calculation HMIS data, add # of persons in universe HMIS data, calculate # of persons who were in ES/SH/TH/ PH in HMIS 24 months prior to entry during reporting year Subtract Step 2 from Step 1

Successful Placement from Street Outreach
Demonstrate an increase in the % of persons served in street outreach projects exit to emergency shelter, safe haven, transitional housing, or permanent housing destinations Metric Metric 7a.1: Change in placements to ES, SH, TH, PH Client Universe Persons… In SO projects Who exited from SO During Current reporting period Calculation 1. HMIS data, add # of persons in universe 2. Of universe, add # of persons who exited to PH, temporary, institutional 3. Divide Step 2 from step 1

Successful Housing Placement To or Retention In a Permanent Housing Destination
Demonstrate an increase in the % of persons served in emergency shelter, safe haven, transitional housing, or rapid re-housing projects exit to permanent housing destination and persons served in permanent housing projects retain permanent housing or exit to permanent housing destinations Metric 7b.1 Change in exits to PH Metric 7b.2 Change in exit to or retention in PH Client Universe Persons… In ES, SH, TH, PH-RRH Who exited (system leavers) During current reporting period In all PH except PH-RRH Calculation HMIS data, add # of persons in universe Of universe, add # of persons in ES/SH/TH/PH-RRH who exited to PH Divide Step 2 from Step 1 Of universe, add persons who remained in PH (except PH-RRH) and exited to PH

#4: Report Progress 4: Report Progress
Who: who will receive & review the data? What: What will be provided (what’s public vs private)? When: How often – quarterly? Annually? Where: Where can you find the results (website? CoC meeting)? How: How will it be disseminated #4: Report Progress

4: Example, Columbus Ohio
Annual Evaluation Quarterly System & Program Indicator Reports Annual Community Report Reports issued to CSB Board of Trustees, CoC Steering Committee, funders, Annual Evaluation – System & program measures compared to period goal – Used to help determination annual funding – Data mostly derived from HMIS – Programs scored as: • High: no less than one not achieved • Medium: half or more achieved • Low: less than half achieved • Quarterly System & Program Indicator Reports – System & program performance measures compared to period goal – Used to identify system(s) and program(s) “of concern”, need for intervention • Annual Community Report on Homelessness – Annual and trend data – Point-in-time count data • Reports issued to CSB Board of Trustees, CoC Steering Committee, funders,

#5: Make Improvements 5: Make Improvements
Continuous quality & performance improvement What gets measured, gets done Implement some quality practices and keep getting better! Key Considerations: – How can performance reports be used to in a performance improvement framework? • Continuous quality and performance improvement mind set, practice Performance data should inform… • Annual HUD CoC project selection process • Local public/private funding decisions • Identification of system needs/gaps- what is the data telling you and how can you make adjustments? • Public policy development- how to best use the data to inform the community, funders, government • CoC goal adjustment #5: Make Improvements

Example Tools (hyperlinks)
State of Indiana: City of Chicago: City of Los Angeles: Memphis, TN: Columbus, OH: All hyperlinks lead to the main pages – from there you can find the evaluation instruments

Implementation Evaluation instrument for HUD projects (or city, county) Needs analysis for private funders Capacity building & technical assistance Program models and expectations revision Reallocation Many ways to get to the results you want- lot of guidance and resources available. Learn from others and import best practices to your community. There are great ways that the info you will be mining can be used.

Planning for Implementation
HUD recommends that CoCs take the following key action steps while waiting for specifications Become familiar with the selection criteria and process for measuring them Discuss the selection criteria and measurement process with the CoC Board and appropriate committees Work with the HMIS Lead and software provider to ensure they are familiar with the selection criteria and have established a plan for implementation when programming specifications are released Review the HUD HMIS Data Standards Upon release, review the programming specifications Review and test preliminary performance measure output to ensure the results are accurate and share the results with the CoC Hopefully you have been doing some of this in the past year. With the nofa out, you can see exactly what is being asked now for reporting. Begin testing perf measure outputs to make sure they are correct.

Things We Do Know All Project Types except Coordinated Assessment, Day Shelters, and Other will be required to be included in reports from HMIS Reporting Period will be from October 1 through September 30, same as AHAR Reporting Period The baseline year is October 1, 2012 through September 30, 2013 for good reliable data- won’t ask for data prior to that. HDX (PIT and HIC) and HMIS will be the Data Sources All Performance Measures at this time use data elements from the 2014 Data Standards Project Leavers/Stayers vs. System Leavers/Stayers vs. Persons/Adults HUD expects CoCs to also establish appropriate local targets High Data Quality is a must for good system performance measurement System leavers are persons who were in the system during the operating year but had exited from all applicable continuum project types being measured at the end of the reporting period. System stayers are persons who were in one of the applicable continuum project types being measured at the end of the reporting period, including persons who were continuously enrolled (i.e., had an open HMIS record during the entire reporting period). CoCs should assume that when HUD refers to “persons” or “adults” in a measure that both system stayers and system leavers should be included in the calculation unless there is language that clearly limits the measure to only system stayers or system leavers.

Final Tips Make a plan and start! Based on performance evaluation
Remain transparent Be consistent Decide method Create incentives Be firm Reporting: HUD expects data to be messy at first (maybe in year 1 & 2) – they recognize system performance is a long term process. They still plan to score system performance in coming NOFAs (starting in 2016) even if data is messy. Keep in mind, right now software providers are only providing system level measurement programming. CoCs would have to create their own/pay for their own adjustments for deeper data dives. Adjustments: Some people claiming that measures won’t work as well for youth. If weird incentives come up that incentivize the wrong things, HUD will find a way to get rid of incentives

System Performance Measurement Webinar

Similar presentations

Presentation on theme: "System Performance Measurement Webinar"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

System Performance Measurement Webinar

Similar presentations

Presentation on theme: "System Performance Measurement Webinar"— Presentation transcript:

Similar presentations

About project

Feedback