Performance Management Performance Mgmt and CSBG Performance Management Analysis & Use of Data JT National Association for State Community Services Programs

Workshop Objectives This session will focus on how a review and analysis of agency Community Needs Assessment data can be used to best develop the Strategic Plan and the Community Action Plan. Assuring the implementation of the full ROMA Cycle can be done while using these data to support and validate the agency’s Theory of Change. These activities taken together will ensure agencies are matching their ROMA outcome measures with identified needs, and are able to make meaning of their data.

Performance Management Framework Performance Measurement and CSBG Performance Management Framework JENAE National Association for State Community Services Programs

Performance Measurement and Management Principles Performance Mgmt and CSBG Performance Measurement and Management Principles Identify the elements that come together to produce success. Link data elements in a meaningful and useful way. Assess poverty needs (causes and conditions) within the community. Identify specific improvements, or results, to be achieved among low-income people and the community. Specify the protocols that support collection and analysis of the elements. JT Fall work… NASCSP

Performance Measurement and CSBG ROMA Logic Model Organization/Program _________ Level: __ family __ agency __ community Need Service or Activity Include who will be served, how many people/units of service and time frame. Outcome Indicator Of those who will be served, how many will achieve the outcome in the time frame. Evidence The tool that will be used to measure and document success. Data collection processes and personnel Mission Statement: Barbara Note: the “Actual Results” column is missing from this graphic. Once service is delivered, actual results must be compared with projected results. National Association for State Community Services Programs

Can Agencies Do This Now? Performance Mgmt and CSBG Can Agencies Do This Now? Implement full ROMA Cycle on a continual basis. Use systems that allow data to be aggregated, analyzed and reported quickly and efficiently. Achieve an unduplicated count without comprising program participant privacy Connect critical data from disparate systems to meet diverse program needs and reporting requirements. Monitor and manage the quality and effectiveness of services. Use data to make adjustments for improvement (what’s working, what’s not working, and WHY). BM National Association for State Community Services Programs

Performance Mgmt and CSBG Can States Do This Now? Assess the capacity of the local agencies (eligible entities) Ensure that agencies implement strategies to measure and record outcomes Monitor agency performance management Compare plans with actuals Use data to compare agency performance, statewide, and replicate best practices BM National Association for State Community Services Programs

How People, Services, and Outcomes Connect to Performance Measurement Analysis

Population Served - Characteristics Performance Mgmt and CSBG Population Served - Characteristics Community Action Agencies Provide a wide range of services Serve a wide range of people, Produce a wide range of outcomes . Analysis of outcome data must include clear identification of “who” received “what” services as well as the numbers served, and the outcomes achieved . If we served 16.5 million people last year, why did we only report 130 k jobs? This is .6% - just about half of a percent. This is not the right question, because the denominator of the fraction is only the set of those we served that were seeking jobs. This eliminates at least half of the population (previous slide). National Association for State Community Services Programs

Example: Outcomes and Clients Performance Mgmt and CSBG Example: Outcomes and Clients This graph shows all the people that were served for whom we have age characteristics. This is 13.7 million people. But notice how many are children! The number of people who were seeking employment is much less. We have that information for goal 1, so that makes it easy for us to contextualize this outcome. National Association for State Community Services Programs

Example: Outcomes and Clients Performance Mgmt and CSBG Example: Outcomes and Clients How many of the clients can actually be reasonably expected to obtain specific outcomes? If a third of the population served by CAAs are under 17, what happened to them? (family service or direct service to the youth) What percent of the total population received single or short term services? How many received multiple/integrated services? Of course, some children benefit from services such as housing, food, other family services. But others received direct services (Such as early childhood education or support for school success) What percent? National Association for State Community Services Programs

Performance Mgmt and CSBG Core Population The individuals, families, groups, or communities an organization = its core population The agency may not collect a full range of characteristics about everyone in this category, but some key characteristics will be collected. At the minimum, this population should have a unique agency identifier that would enable an unduplicated count of individuals and/or families. How does agency holds itself accountable for the achievement of specified outcomes for core populations? I.E. There is a system for follow up to identify outcomes (change). Agencies provide services in a variety of ways to the target population they serve. In some cases individuals and families receive a single service from the agency. It produces an outcome in one life domain area. In other cases, individuals and families may receive multiple services from the agency but these are not coordinated, integrated nor included in an overall service plan. The agency is able to identify the number of different services the individual/family has received but does not aggregate the information about the family. It is believed that this number of unduplicated individuals and families will represent the majority of those receiving service from an agency. However, it is also true that some individuals and families receive what has been characterized as “bundled services,” in which a plan for comprehensive and sequenced services has been created. In this case, the services are followed, assessed and the outcomes that result from the services are observed, documented and discussed as a part of future planning.   National Association for State Community Services Programs

Performance Mgmt and CSBG Service Population The individuals, families, groups or community members that an organization serves, but does so without expecting outcomes. agency may collect no characteristics about this population, and may not be able to identify if individuals and families are duplicated in the overall service count. Identified in 12.2 of MSP? There are numerous reasons, including the presence of legacy programs; funder requirements; the need to maintain credibility and promote ease of access in local neighborhoods; and the desirability of having a broad base of support when engaging in advocacy. Examples of a Service Population—youth attending a drop-in center, school-based or community-based informative workshops, a job “fair” or other collection of resources for a given topic such as health, or senior citizen services, a soup kitchen or food distribution where no follow up will be done.” National Association for State Community Services Programs

What Does the Data Do? Can we use the data to demonstrate: we focused on “Changing Lives”? we are following up to show the long term impact of our work? what needs to be done next year to expand on what works? can we tell what set of services give the best opportunity for results?

Management Based on Data You can’t “manage” performance if you can’t define and measure it. Our expectations for outcomes must be based on who is served and the needs in the community. first collect appropriate data organize the data into useful reports for management enable agencies to more effectively connect people and distressed communities with the services needed to achieve specific outcomes. Identify actions to take to improve or strengthen outcomes

Performance Mgmt and CSBG Report Areas to Assess Quality of the data -- Accuracy, completeness and timeliness of the data included in the report (4 items) Presentation of the data in the report – it is easy for the reader to find data that is important to him/her? (6 items) Usefulness of the data – is it relevant to the task of managing the program/service? Do I have access to the data when I want/need it? (7 items) Performance Focus – includes data on outcomes and performance (3 items) Depending on where you are in the agency’s hierarchy, you may have more or less access to the system. We have tested this set of statements in a limited way and have found that most middle managers in agencies agree or strongly agree with only about 40% of the items identified. How useful are your management reports? assessment tool (handout) is designed to cover several important areas to help you determine if you have a data collection and reporting system that supports both management and accountability efforts.   National Association for State Community Services Programs

Performance Measurement and CSBG Analysis of Data What skills, expertise, resources are needed to analyze your agency data? Brainstorm to answer the first question Possible responses: math, organization, ability to read a report, understanding of the community needs, time National Association for State Community Services Programs

Can You Compare Your Data? Program data from year to year Quantity of service, outcome, cost of program Your program with other similar ones What outcomes do other agencies who do this service achieve? Are our populations similar? How might differences in population impact differences in outcomes? Your program with national trends

Sample Questions for Analysis Performance Mgmt and CSBG Sample Questions for Analysis Was there something unexpected that influenced the outcomes? Are some populations achieving outcomes at different rates than others? Did we not recruit and enroll sufficient numbers to allow us to achieve our target outcomes? Do we need additional resources? Did the outcomes validate our assumptions? Did we make an impact on the identified needs? National Association for State Community Services Programs

Performance Measurement and CSBG Where We Are: ROMA Internal Challenges Many CAAs see ROMA merely as a reporting requirement CSBG Network struggles to see clearly what works for whom under what conditions Existing Community measures do not adequately demonstrate the depth of work CAAs do We understand that the NPIs track movement of families up the ladder and often multiple indicators are connected to one family, but this is not understood by people outside the CSBG Network   External Challenges Conditions of poverty demand different levels of results Political climate expects the CSBG Network to articulate meaningful results National Association for State Community Services Programs

Where We Want To Be: ROMA NG Performance Measurement and CSBG Where We Want To Be: ROMA NG OCS, States and CAAs use performance data to achieve greater stability and self-sufficiency for families and communities. Each level of the CSBG Network executes the full ROMA cycle Each level of the CSBG Network uses data on people, services, and outcomes for decision making The CSBG Performance Management Framework supports improvements in the quality of both individual and community level results achieved by CAAs The presentation of CSBG Network results provides a clearer story of the impact achieved by CAAs National Association for State Community Services Programs

Performance Measurement and CSBG How Do We Get To ROMA NG? Connect ROMA Next Generation with the organizational standards and state and federal accountability measures in a meaningful and clear way so the CSBG Network understands this is one CSBG Performance Management Framework Establish and adopt a standardized framework for ROMA Next Generation Provide leadership to invest in people, build organizational culture, and improve data systems National Association for State Community Services Programs

What is our Theory of Change? The National TOC will help all levels of the network to consider universal basic assumptions as they establish performance management protocols. However, it is a Local Theory of Change that will help local agencies as they implement the ROMA Cycle.

ROMA Next Generation Theory of Change

Local TOC + Needs Assessment The local agency identifies its assumptions about poverty in the community. Poverty is different in different communities. Use TOC assumptions to make meaning out of comprehensive community needs assessments.

Local TOC & Delivery of Services Local agencies identify specific strategies and services that they think will produce family, agency, and community outcomes directly related to the locally assessed needs and the mix of resources that are available both within the agency and in the broader community.

Local TOC and Planning Inform strategic planning efforts. Establish specific populations the agency serves (children/youth, senior citizens, veterans, unemployed wage earners, etc.) Identify specific services that the agency will provide, based on its mission and priorities.

Local TOC and Evaluation The TOC can be a yardstick to understand assumptions behind programs and expected outcomes, measure and analyze agency data. Allow the agency to test assumptions – did the service produce the expected outcome? Why or why not?

A Local Theory of Change Will: Guide analysis of data, informed by the TOC, which feeds into the Assessment phase of the ROMA cycle as the cycle starts again, so the agency can continuously improve strategies and increase Community Action's effectiveness and efficiency.

Take Aways Community Needs Assessment data should be used to develop the Strategic Plan and the Community Action Plan. These activities can be taken together to ensure agencies match their ROMA outcome measures with identified needs, and are able to make meaning of their data. The full ROMA Cycle can be implemented by using these data to support and validate the agency’s Theory of Change. Reports can be strengthened to ensure connection to both outcomes data, analysis of services, and the effectiveness on populations

Performance Measurement and CSBG Contact Us For more information please visit our website at http://www.nascsp.org/. National Association for State Community Services Programs