ROMA Next Generation Results-Oriented Management and Accountability Region 9 Webinar

Session Objectives Update on ROMA Next Generation Performance Management Principles Protocols needed Data Analysis at different levels Benefits of a local Theory of Change (TOC) Changes to the CSBG IS Use of CSBG $ Population Targeting What to expect from changes to National Performance Indicators Family and Community level outcomes Bundled Services JT

ROMA as a Performance Management System

Performance Mgmt and CSBG We have spent some time discussing the steps on the ROMA cycle that are related to Assessment and Planning. Today we will look more closely at the steps on the other side of the cycle: observing and reporting results and analysis of data. In the ROMA Cycle, we use the word Evaluation to mean “analysis of the data, understanding of the situation in which the data was collected, and consideration of what may need to change to improve services and outcomes.” Evaluation used in this way is an integral part of Performance Management National Association for State Community Services Programs

Evaluation as a part of Performance Management Evaluation = Analysis of Data Focus on short-term outcomes Identify incremental progress Identify the specific elements of the services that support progress What works? What needs changed? Identify how the short term outcomes lead to more enduring intermediate and ultimate outcomes Follow up to assure additional outcomes are achieved Research Based Evaluation Focus on long-term outcomes perhaps intermediate outcomes if they are clearly established pre-conditions Done “after the fact” the data may not be used to help improve the service May have a narrow focus May not take into account other services and outcomes that are interrelated to the target outcome that is being evaluated. We hear a lot about Evidence Based service. What does that mean? It means that there has been some formal evaluation that is done in a Research framework. I want to differentiate this kind of evaluation from the “evaluation” step in the ROMA Cycle. In the ROMA Cycle, we use the word evaluation to an integral part of Performance Management

Principles to Consider Performance Mgmt and CSBG Principles to Consider Here are a few principles, designed to identify some data elements and protocols that support collection and analysis of the elements. Include something like a Logic Model (a graphic representation of the elements to be collected) to connect the identified need to population to services to outcomes Acknowledge the need to establish different time frames for measuring progress (short term, during one program year, over multiple program years) Demonstrate how family and community successes are interrelated. JT Fall work… NASCSP

Note: the “Actual Results” column is missing from this graphic. ROMA Logic Model Organization/Program _________ Level: __ family __ agency __ community Need Service or Activity Include who will be served, how many people/units of service and time frame. Outcome Indicator Of those who will be served, how many will achieve the outcome in the time frame. Evidence The tool that will be used to measure and document success. Data collection processes and personnel Mission Statement: Note: the “Actual Results” column is missing from this graphic. Once service is delivered, actual results must be compared with projected results.

Implementing These Principles Performance Mgmt and CSBG Implementing These Principles Agency level issues State level issues Implement full ROMA Cycle on a continual basis. Use systems that allow data to be aggregated, analyzed and reported quickly and efficiently Achieve an unduplicated count without comprising program participant privacy Connect critical data from disparate systems to meet diverse program needs and reporting requirements Monitor and manage the quality and effectiveness of services Use data to make adjustments for improvement (what’s working, what’s not working, and WHY). Assess the capacity of the local agencies (eligible entities) Monitor agency performance management processes Compare plans with actuals Use data to compare agency performance, statewide, and replicate best practices BM National Association for State Community Services Programs

DATA ANALYSIS USING THE ROMA CYCLE Performance Measurement and CSBG October 2013 Focus on analysis DATA ANALYSIS USING THE ROMA CYCLE JT National Association for State Community Services Programs

Performance Measurement and CSBG October 2013 What difference did the Network make? NEEDS ASSESSMENT What Outcomes Do We Want to Achieve? PLANNING What Services Do We Need to Provide to Achieve the Outcomes? What Partnerships Do We Need to Make To Achieve the Outcomes? What Services Do We Need to Bundle to Better Achieve the Outcomes? What Data Do We Need to Collect/Use? JT IMPLEMENTATION Services Provided to Low-Income Individuals/Families Strategies Implemented to Change Conditions in the Community Individuals Served Are Tracked Services Provided Are Tracked Performance Data Is Collected National Association for State Community Services Programs

Performance Measurement and CSBG October 2013 DATA ANALYSIS/ EVALUATION DATA ANALYSIS/ EVALUATION at the local agency level Which individuals achieved particular outcomes? Which NPIs did the agency meet? Not meet? What services were most used by individuals achieving the outcomes? What specific mix of services was most used by individuals achieving the outcomes? What services were most used by individuals not achieving the outcomes? Did the individuals not achieving the expected outcomes use different services? Not use specific services? What can we surmise about what services are most effective for achieving the expected outcomes? How well did we deliver services? (Timeliness, Accuracy, Cost, Client Satisfaction) JT What changes should we make to our Service Model? What do more of, less of? Any services to stop delivering? Any services to deliver differently? Any services to add? Do we need to improve how we deliver services? National Association for State Community Services Programs

EVALUATION at the State Level DATA ANALYSIS/ EVALUATION DATA ANALYSIS/ EVALUATION at the State Level Is the data received from the local agencies accurate? Reliable? (Unduplicated counts of individuals served?) What Outcomes (NPIs) were met by each of the local agencies across the state? What Outcomes were not met? Taken together, what outcomes (NPIs) were achieved across the state? How many people were served across the state? What types of services were provided? How many of these types of services were provided across the state? What service delivery models appear most effective? What local agency best practices should be replicated in other parts of the state? What does the data tell us about any patterns and trends re: service delivery and outcomes achieved? In Summary: What did Community Action achieve in our State? “In this State, we achieved X Results (Outcomes) for this many low-income individuals/families/communities (Output) by delivering these services (Output), at this Cost.” What local agencies should receive enhanced training and technical assistance?

Theory of Change Explain WHAT we do, WHY we do it, and WHY we think what we do will make a difference. Identify what will change. Identify who will achieve the change. JT

Theory of Change Identify the assumptions behind the work of CAAs What do we believe about poverty? What are the causes and conditions of poverty in our community? What is the role of our CAA? Who does our CAA serve? Identify the changes we expect to achieve If we do not expect change to happen, we must identify what will happen to the people and communities that are served.

HOW does change happen? As a national network of local CAAs, there are a lot of different things happening across the country. The national Community Action TOC is stated to reflect the broad network reality. Community Action stabilizes families by meeting immediate needs. Community Action creates pathways to self-sufficiency by employing a continuum of services. Community Action joins with partners to create change in community conditions that foster the environment needed to cultivate and maintain self-sufficiency. However, local agencies identify specific strategies and services that produce family, agency, and community outcomes directly related to the locally assessed needs and the mix of resources that are available both within the agency and in the broader community. Local agencies may be focused on a narrow set of strategies and outcomes that do not address all of the things in the national TOC. This focus is what the local Theory of Change should identify.

A Local Theory of Change will help make meaning out of comprehensive community needs assessments. Organize the “needs” according to the agency’s assumptions and you will better understand the statistical data. The qualitative data (surveys, forums) will clarify the needs. The agency data will identify what you did last year

A Local Theory of Change will inform strategic planning efforts. Performance Mgmt and CSBG A Local Theory of Change will inform strategic planning efforts. Using the TOC will allow the agency to test assumptions – did the service provided last year produce the expected outcome? Why or why not? The TOC will help identify the population that you plan to serve next year. The agency can use the TOC to consider: What will be done differently next year? The TOC can be a yardstick to measure and analyze agency data. National Association for State Community Services Programs

NPIs and Other Information Annual reporting

Performance Mgmt and CSBG By Service Categories Use of CSBG Dollars BM NASCSP

$ assigned by different methods Direct tracking method: In this case the local eligible entity or discretionary grantee has created a fiscal accounting system in which the use of CSBG funding is tracked to specific activities, and those activities are characterized as one of the service categories. Allocation method: There are two ways that CSBG dollars are allocated to service categories. One uses the number of participants who received agency services and the other uses the total agency budget to identify the agency’s services by category.

Introductory Question Suggestion: Add a question to identify how CSBG$ are being reported. Provide choices: Actual tracking of dollars to expenses by category Assignment to categories by percent, based on overall agency budget also identified by categories Assignment to categories by percent, based on participants (number and service received) Assignment based on other system (describe)

Proposed Service Categories Performance Mgmt and CSBG Proposed Service Categories 1. Employment 2. Education 3. Income Management 4. Housing 5. Emergency and Stabilization Services 6. Nutrition 7. Community Linkages 8. Family Development/Case Management 9. Health 10. Disaster Relief 11. Organizational Capacity Building 12. Administration BM National Association for State Community Services Programs

Community Level Expenses Performance Mgmt and CSBG Community Level Expenses What expenses support the development of community level strategies? Community level work could be done in any of the domains (housing, employment, etc.) Linkages defined as community level Community level work also found within agency expenses: Some needs assessment work would be community level Support of community engagement and input Development of partnerships and coalitions How much of total CSBG funding is spent in this way? BM National Association for State Community Services Programs

Community Level Work Suggestion: Add Table 2: Identification of CSBG Funds that were used in the categories identified above for community outcomes. Of the funding listed in Table 1, the funds used for community level work / associated with community outcomes (reported in NPI Goals 2, 3 or 4)

Our expectations for outcomes must be based on who is served. Population served

Population Served - Characteristics Performance Mgmt and CSBG Population Served - Characteristics We agree that Community Action Agencies provide a wide range of services. But then all the customers are pooled, and characteristics are identified for the entire agency population served. This makes analysis of outcome data difficult. If we served 16.5 million people last year, why did we only report 100 k jobs? This is .6% - just about half of a percent. This is not the right question. National Association for State Community Services Programs

Performance Mgmt and CSBG This graph shows all the people that were served for whom we have age characteristics. This is 13.7 million people. But notice how many are children! The number of people who were seeking employment is much less. We have that information for goal 1, so that makes it easy for us to contextualize this outcome. National Association for State Community Services Programs

Comparing Apples to Apples Performance Mgmt and CSBG Comparing Apples to Apples This shows that 59% of the people who were enrolled in programs that were related to employment and employment supports actually obtained a job. And 54% who were employed maintained the job for 90 days. But who are these people? Are they teens? Senior citizens? Single mothers? All of the above? What kind of skills did they have coming into the agency? What barriers? And what services did they get? Single service? Multiple services? Bundled or integrated services? We don’t know in the national report. National Association for State Community Services Programs

Relating Outcomes to Populations Performance Mgmt and CSBG Relating Outcomes to Populations If a third of the population served by CAAs are under 17, can we connect what happened to them in an improved performance management system? Separate the population that received single or short term services from those that receive multiple/integrated services. (more about this coming later) Did some children benefit from services such as housing, food, other family services? What percent received direct services? Such as early childhood education or support for school success? National Association for State Community Services Programs

Performance Mgmt and CSBG Demonstrate ability to project successes. Targeting Considering the population will improve an agency’s ability to target. The prior example leads us to consider this task of setting targets for your work. National Association for State Community Services Programs

Improving the evidence of our abilities to identify: Number to be served Number to achieve outcome

Successful Targeting Is Expected 4/17/2017 The GPRA legislation (Government Performance and Results Act of 1993/ Modernization Act of 2010) refers to “performance plans” and “performance goals,” which both relate to the concept of planning for specific results. This references two kinds of targets: the number of people who are expected to be served and the number of people (of those served) who are expected to achieve an outcome. These targets are identified in planning so they can be assessed in the evaluation phase. National Association for State Community Services Programs

Success and Circumstances 4/17/2017 These two different target numbers produce several different ways of looking at “success.” The success calculations require data to be collected during the planning process and again at the end of program service. Setting targets is important in the world of performance management. If you don’t set targets and then evaluate how well you met them, you will never know how you are performing. If you only know how many you served, you are not being “results oriented” – so what happened? And if you only know how many achieved a result you cannot say if that is good or not. If 20 people were successful is this out of a possible 30? Or out of a possible 300? You need both numbers to calculate success. National Association for State Community Services Programs

We Have a Good Start! From NASCSP CSBG Annual Report 2014: 4/17/2017 From NASCSP CSBG Annual Report 2014: In addition to CSBG’s performance measurement initiative, the Executive Office of the President, Office of Management and Budget (OMB) has established a government-wide initiative to use performance targets and outcome measures to assess the program efficiency and effectiveness of all federally-funded domestic assistance programs. As a result, beginning in FY 2004 OCS began to develop and request a report on performance targets, or anticipated levels of result achievement, for four CSBG Performance Indicators. It is expected that targeting of outcomes will expand to other indicators. National Association for State Community Services Programs

Assessing Ability to Target 4/17/2017 As the data accrue, agencies find out how well they can predict performance outcomes. To find out the percent of successful targeting, the number of participants who actually achieved the goal (at the end of the program) is divided by the number expected to achieve the goal (set in planning). The resulting percentage assesses CAAs’ knowledge of how their programs work. National Association for State Community Services Programs

Performance Mgmt and CSBG One Targeting Percent We talked about National Association for State Community Services Programs

Another Targeting Percent The other way to evaluate the success of a program(which we saw earlier) is to compare the number achieving the outcome with the number who were provided service (the total number that could have achieved).

Understand the Circumstances Recognizing When Some Data Needs Special Attention

How to target for single or short term services? 4/17/2017 There are programs that an agency provides that are designed to meet an immediate need or crisis situation. There is no change in status expected. The recipient of service will still be facing all of the needs, but this one area will not be in-crisis any longer (or at least for a certain period). In this case you can only target the number to be served, unless you have a system to do follow up. National Association for State Community Services Programs

Dealing with the Impact of One-Time Events What happens when you have large events only once a year? Special programs at Thanksgiving/Christmas Or events that serve large numbers at specific times of year Fuel programs that peak in winter months summer programs for school children. The participants in these events are not going to be tracked for outcomes. In this case you will target only the number to be served. Refer to chart in manual.

Dealing with the Complexity of Anti-Poverty Programs Poverty is an enormously complex issue. Reducing poverty, whether for a household or for a community, is difficult and requires a long-term effort on many levels. “Long term” effort means that you have to acknowledge challenges in data collection practices that may cover several years (or reporting periods) of service. It is important to develop tracking systems that allow you to identify participants who have made progress but not yet achieved the outcome. (as different from those who have dropped out or are no longer progressing) Refer to charts in manual.

Dealing with the Complexity of Anti-Poverty Programs Acknowledge that some outcomes depend on the activities of many programs working together. Operationally distinct programs can contribute to meeting one or more poverty reduction targets. Each separate program has its own internal targets, but they also share targets related to the long-term goal of family self-sufficiency. So a participant may be making progress in one program, which is a prerequisite for progress in another program, but does not appear active in both programs. Agency tracking systems should have a way to identify these circumstances. Refer to charts in manual.

National performance indicators Better identification of the changes that happen National performance indicators

Performance Mgmt and CSBG The Six National Goals Goal 1. Low-income people become more self-sufficient. (Family) Goal 2. The conditions in which low-income people live are improved. (Community) Goal 3. Low-income people own a stake in their community. (Community) Goal 4. Partnerships among supporters and providers of services to low-income people are achieved. (Agency) Goal 5. Agencies increase their capacity to achieve results. (Agency) Goal 6. Low-income people, especially vulnerable populations, achieve their potential by strengthening family and other supportive systems. (Family) The current NPIs are based on the Six National Goals National Association for State Community Services Programs

The network’s long term goals: Performance Mgmt and CSBG The network’s long term goals: Low-income people become more self-sufficient. The conditions in which low-income people live are improved (producing a thriving or revitalized community). As we started the TOC work we asked: what are our long term goals. We had agreement that these are the long term goals. National Association for State Community Services Programs

These are actually pre-conditions (or intermediate goals) Performance Mgmt and CSBG These are actually pre-conditions (or intermediate goals) Agencies increase their capacity to achieve results – and be better able to document and analyze the changes they have supported. Low-income people own a stake in their community – to improve their own social capital and take action for community issues. Partnerships among supporters and providers of services to low-income people are achieved – to achieve a specific purpose. Low-income people, especially vulnerable populations, achieve their potential – to maintain or achieve stability. National Association for State Community Services Programs

How to Improve the National Performance Indicators Make connections between the indicators and goals. Why do we think these indicate success in the Six National Goals? Identify the numbers of individuals, families, communities that achieved the GOALs. Have they achieved multiple indicators? Connect the services provided with the indicators reported Is it one service for one outcome? Many services for one outcome? Many outcomes from one service? Clarify the population that is achieving the indicators Target population and service population Unduplicated count? Think of the logic model. We have identified a broad outcome without numbers. The indicators have the numbers and a greater detail and are connected to a measurement tools and processes. The NPIs are just a snapshot of the one column. We need to analyze data from the field to determine the amount and type of service needed to produce results that actually matter (are "good enough") in moving the families that receive “outcome producing services” toward the long term national goals. In other words, what produces change. To do that we need to have a new area of reporting that asks for information about services provided.   We need to be able to identify the number of customers who are receiving single services and those who are receiving multiple services. (Differentiate between the “target” population and the “service” population.) Of those receiving multiple services, identify those who have a comprehensive sequenced plan identifying how the services will achieve customer goals.

Performance Mgmt and CSBG Family Level NPIs BM National Association for State Community Services Programs

Examples of Family Indicators Performance Mgmt and CSBG Examples of Family Indicators Net family household income is increased. Families reduce reliance on public benefits. Families improve status in multiple life domains. Children meet developmentally appropriate milestones. Vulnerable populations achieve/maintain stability. Ability to track multiple services to the change they produce. National Association for State Community Services Programs

Performance Mgmt and CSBG Community Level NPIs JT National Association for State Community Services Programs

Community Indicators Community Action addresses complex issues that relate to creating thriving communities. Community Action joins with stakeholders (partners) to improve conditions in low income communities. Communities are improved by people with low incomes having network connections, and these community networks help improve the social capital of people with low incomes. JT

Reporting Community Work The agency can clearly articulate the purpose of each partnership and the expected changes that will be produced. The agency can identify the number of partnerships that both support the agency’s activities (agency outcome) and support community change. The agency can track its involvement in and value to community changes. The agency records involvement of their customers (individuals with low-income) in community projects.

Examples of Community Changes Change in personal behaviors of community residents – such as reduced litter or graffiti; or increase use of radon detectors Change in corporate practices – such as grocery stores moved to low income communities; or businesses hire low income community residents Change in policy or regulation - increase in minimum wage; changes to child care eligibility for working parents

Identify those who move to self sufficiency using Bundled services

Bundled, Integrated, Multiple? It is important to know if a participant has received multiple services, and which services were received. The question is: do participants receiving multiple services achieve more outcomes than their peers who receive single services? Are the “multiple” services achieving maximum power? (Which comes when the services are sequenced and are carefully selected in a strengths-based planning process.) It is the “comprehensive” and “integrated” nature of the services that adds the most value.

Hypothesis Community Action has the ability to determine which “bundle of services” are most effective for certain demographic groups and in certain locals to achieve “success.” Note: Success maybe be defined in many different ways – and may include the achievement of multiple outcomes (in multiple domains).

Putting data to work during analysis Learning from the Data Putting data to work during analysis

Management based on data You can’t “manage” performance if you can’t define and measure it. first collect appropriate data find out “what happened” (both what went well and what can be done differently) Identify actions to take to improve or strengthen outcomes

Questions for analysis Was there something unexpected that influenced the outcomes? Did we not recruit and enroll sufficient numbers to allow us to achieve our target outcomes? Do we need additional resources? Did the outcomes validate our assumptions? Did we make an impact on the identified needs?

Performance Mgmt and CSBG Comparing Data Program data from year to year Quantity of service, outcome, cost of program Your program with other similar ones What outcomes do other agencies who do this service achieve? Are our populations similar? How might differences in population impact differences in outcomes? Your program with national trends National Association for State Community Services Programs

more resources and information. Visit www.nascsp.org http://www.nascsp.org/ROMA/919/ROMA-Next-Generation.aspx?iHt=15 for more resources and information. 4/17/2017 National Association for State Community Services Programs 63