1. ROMA Next Generation Results-Oriented Management and Accountability
Region 9 Webinar

2. Session Objectives Update on ROMA Next Generation
Performance Management Principles
Protocols needed
Data Analysis at different levels
Benefits of a local Theory of Change (TOC)
Changes to the CSBG IS
Use of CSBG $
Population
Targeting
What to expect from changes to National Performance Indicators
Family and Community level outcomes
Bundled Services JT

3. ROMA as a Performance Management System

4. Performance Mgmt and CSBG
We have spent some time discussing the steps on the ROMA cycle that are related to Assessment and Planning.
Today we will look more closely at the steps on the other side of the cycle: observing and reporting results and analysis of data.
In the ROMA Cycle, we use the word Evaluation to mean “analysis of the data, understanding of the situation in which the data was collected, and consideration of what may need to change to improve services and outcomes.”
Evaluation used in this way is an integral part of Performance Management
National Association for State Community Services Programs

5. Evaluation as a part of Performance Management
Evaluation = Analysis of Data
Focus on short-term outcomes
Identify incremental progress
Identify the specific elements of the services that support progress
What works?
What needs changed?
Identify how the short term outcomes lead to more enduring intermediate and ultimate outcomes
Follow up to assure additional outcomes are achieved
Research Based Evaluation
Focus on long-term outcomes
perhaps intermediate outcomes if they are clearly established pre-conditions
Done “after the fact”
the data may not be used to help improve the service
May have a narrow focus
May not take into account other services and outcomes that are interrelated to the target outcome that is being evaluated.
We hear a lot about Evidence Based service. What does that mean?
It means that there has been some formal evaluation that is done in a Research framework.
I want to differentiate this kind of evaluation from the “evaluation” step in the ROMA Cycle.
In the ROMA Cycle, we use the word evaluation to an integral part of Performance Management

6. Principles to Consider
Performance Mgmt and CSBG
Principles to Consider
Here are a few principles, designed to identify some data elements and protocols that support collection and analysis of the elements.
Include something like a Logic Model (a graphic representation of the elements to be collected) to connect the identified need to population to services to outcomes
Acknowledge the need to establish different time frames for measuring progress (short term, during one program year, over multiple program years)
Demonstrate how family and community successes are interrelated. JT
Fall work…
NASCSP

7. Note: the “Actual Results” column is missing from this graphic.
ROMA Logic Model
Organization/Program _________
Level:
__ family __ agency __ community
Need
Service or Activity
Include who will be served, how many people/units of service and time frame.
Outcome
Indicator
Of those who will be served, how many will achieve the outcome in the time frame.
Evidence
The tool that will be used to measure and document success.
Data collection processes and personnel
Mission Statement:
Note: the “Actual Results” column is missing from this graphic.
Once service is delivered, actual results must be compared with projected results.

8. Implementing These Principles
Performance Mgmt and CSBG
Implementing These Principles
Agency level issues
State level issues
Implement full ROMA Cycle on a continual basis.
Use systems that allow data to be aggregated, analyzed and reported quickly and efficiently
Achieve an unduplicated count without comprising program participant privacy
Connect critical data from disparate systems to meet diverse program needs and reporting requirements
Monitor and manage the quality and effectiveness of services
Use data to make adjustments for improvement (what’s working, what’s not working, and WHY).
Assess the capacity of the local agencies (eligible entities)
Monitor agency performance management processes
Compare plans with actuals
Use data to compare agency performance, statewide, and replicate best practices BM
National Association for State Community Services Programs

9. DATA ANALYSIS USING THE ROMA CYCLE
Performance Measurement and CSBG
October 2013
Focus on analysis
DATA ANALYSIS USING THE ROMA CYCLE JT
National Association for State Community Services Programs

10. Performance Measurement and CSBG
October 2013
What difference did the Network make?
NEEDS
ASSESSMENT
What Outcomes Do We Want to Achieve?
PLANNING
What Services Do We Need to Provide to Achieve the Outcomes?
What Partnerships Do We Need to Make To Achieve the Outcomes?
What Services Do We Need to Bundle to Better Achieve the Outcomes?
What Data Do We Need to Collect/Use? JT
IMPLEMENTATION
Services Provided to Low-Income Individuals/Families
Strategies Implemented to Change Conditions in the Community
Individuals Served Are Tracked
Services Provided Are Tracked
Performance Data Is Collected
National Association for State Community Services Programs

11. Performance Measurement and CSBG
October 2013
DATA ANALYSIS/
EVALUATION
DATA ANALYSIS/
EVALUATION at the local agency level
Which individuals achieved particular outcomes?
Which NPIs did the agency meet? Not meet?
What services were most used by individuals achieving the outcomes? What specific mix of services was most used by individuals achieving the outcomes?
What services were most used by individuals not achieving the outcomes? Did the individuals not achieving the expected outcomes use different services? Not use specific services?
What can we surmise about what services are most effective for achieving the expected outcomes?
How well did we deliver services?
(Timeliness, Accuracy, Cost, Client Satisfaction) JT
What changes should we make to our Service Model? What do more of, less of? Any services to stop delivering? Any services to deliver differently? Any services to add? Do we need to improve how we deliver services?
National Association for State Community Services Programs

12. EVALUATION at the State Level
DATA ANALYSIS/
EVALUATION
DATA ANALYSIS/
EVALUATION
at the State Level
Is the data received from the local agencies accurate? Reliable? (Unduplicated counts of individuals served?)
What Outcomes (NPIs) were met by each of the local agencies across the state?
What Outcomes were not met?
Taken together, what outcomes (NPIs) were achieved across the state?
How many people were served across the state?
What types of services were provided? How many of these types of services were provided across the state?
What service delivery models appear most effective?
What local agency best practices should be replicated in other parts of the state?
What does the data tell us about any patterns and trends re: service delivery and outcomes achieved?
In Summary: What did Community Action achieve in our State? “In this State, we achieved X Results (Outcomes) for this many low-income individuals/families/communities (Output) by delivering these services (Output), at this Cost.”
What local agencies should receive enhanced training and technical assistance?

13. Theory of Change
Explain WHAT we do, WHY we do it, and WHY we think what we do will make a difference.
Identify what will change.
Identify who will achieve the change. JT

14. Theory of Change Identify the assumptions behind the work of CAAs
What do we believe about poverty? What are the causes and conditions of poverty in our community?
What is the role of our CAA?
Who does our CAA serve?
Identify the changes we expect to achieve
If we do not expect change to happen, we must identify what will happen to the people and communities that are served.

15. HOW does change happen?
As a national network of local CAAs, there are a lot of different things happening across the country. The national Community Action TOC is stated to reflect the broad network reality.
Community Action stabilizes families by meeting immediate needs.
Community Action creates pathways to self-sufficiency by employing a continuum of services.
Community Action joins with partners to create change in community conditions that foster the environment needed to cultivate and maintain self-sufficiency.
However, local agencies identify specific strategies and services that produce family, agency, and community outcomes directly related to the locally assessed needs and the mix of resources that are available both within the agency and in the broader community.
Local agencies may be focused on a narrow set of strategies and outcomes that do not address all of the things in the national TOC.
This focus is what the local Theory of Change should identify.

16. A Local Theory of Change will help make meaning out of comprehensive community needs assessments.
Organize the “needs” according to the agency’s assumptions and you will better understand the statistical data.
The qualitative data (surveys, forums) will clarify the needs.
The agency data will identify what you did last year

17. A Local Theory of Change will inform strategic planning efforts.
Performance Mgmt and CSBG
A Local Theory of Change will inform strategic planning efforts.
Using the TOC will allow the agency to test assumptions – did the service provided last year produce the expected outcome? Why or why not?
The TOC will help identify the population that you plan to serve next year.
The agency can use the TOC to consider: What will be done differently next year?
The TOC can be a yardstick to measure and analyze agency data.
National Association for State Community Services Programs

20. NPIs and Other Information
Annual reporting

21. Performance Mgmt and CSBG
By Service Categories
Use of CSBG Dollars BM
NASCSP

22. $ assigned by different methods
Direct tracking method: In this case the local eligible entity or discretionary grantee has created a fiscal accounting system in which the use of CSBG funding is tracked to specific activities, and those activities are characterized as one of the service categories.
Allocation method: There are two ways that CSBG dollars are allocated to service categories.
One uses the number of participants who received agency services and
the other uses the total agency budget to identify the agency’s services by category.

23. Introductory Question
Suggestion:
Add a question to identify how CSBG$ are being reported. Provide choices:
Actual tracking of dollars to expenses by category
Assignment to categories by percent, based on overall agency budget also identified by categories
Assignment to categories by percent, based on participants (number and service received)
Assignment based on other system (describe)

24. Proposed Service Categories
Performance Mgmt and CSBG
Proposed Service Categories
1. Employment 2. Education 3. Income Management 4. Housing 5. Emergency and Stabilization Services 6. Nutrition 7. Community Linkages 8. Family Development/Case Management 9. Health 10. Disaster Relief 11. Organizational Capacity Building 12. Administration BM
National Association for State Community Services Programs

25. Community Level Expenses
Performance Mgmt and CSBG
Community Level Expenses
What expenses support the development of community level strategies?
Community level work could be done in any of the domains (housing, employment, etc.)
Linkages defined as community level
Community level work also found within agency expenses:
Some needs assessment work would be community level
Support of community engagement and input
Development of partnerships and coalitions
How much of total CSBG funding is spent in this way? BM
National Association for State Community Services Programs

26. Community Level Work Suggestion:
Add Table 2: Identification of CSBG Funds that were used in the categories identified above for community outcomes.
Of the funding listed in Table 1, the funds used for community level work / associated with community outcomes (reported in NPI Goals 2, 3 or 4)

27. Our expectations for outcomes must be based on who is served.
Population served

28. Population Served - Characteristics
Performance Mgmt and CSBG
Population Served - Characteristics
We agree that Community Action Agencies provide a wide range of services.
But then all the customers are pooled, and characteristics are identified for the entire agency population served.
This makes analysis of outcome data difficult.
If we served 16.5 million people last year, why did we only report 100 k jobs? This is .6% - just about half of a percent.
This is not the right question.
National Association for State Community Services Programs

29. Performance Mgmt and CSBG
This graph shows all the people that were served for whom we have age characteristics.
This is 13.7 million people. But notice how many are children!
The number of people who were seeking employment is much less.
We have that information for goal 1, so that makes it easy for us to contextualize this outcome.
National Association for State Community Services Programs

30. Comparing Apples to Apples
Performance Mgmt and CSBG
Comparing Apples to Apples
This shows that 59% of the people who were enrolled in programs that were related to employment and employment supports actually obtained a job. And 54% who were employed maintained the job for 90 days.
But who are these people? Are they teens? Senior citizens? Single mothers? All of the above?
What kind of skills did they have coming into the agency? What barriers?
And what services did they get? Single service? Multiple services? Bundled or integrated services?
We don’t know in the national report.
National Association for State Community Services Programs

31. Relating Outcomes to Populations
Performance Mgmt and CSBG
Relating Outcomes to Populations
If a third of the population served by CAAs are under 17, can we connect what happened to them in an improved performance management system?
Separate the population that received single or short term services from those that receive multiple/integrated services. (more about this coming later)
Did some children benefit from services such as housing, food, other family services?
What percent received direct services? Such as early childhood education or support for school success?
National Association for State Community Services Programs

32. Performance Mgmt and CSBG
Demonstrate ability to project successes.
Targeting
Considering the population will improve an agency’s ability to target.
The prior example leads us to consider this task of setting targets for your work.
National Association for State Community Services Programs

33. Improving the evidence of our abilities to identify:
Number to be served
Number to achieve outcome

34. Successful Targeting Is Expected
4/17/2017
The GPRA legislation (Government Performance and Results Act of 1993/ Modernization Act of 2010) refers to “performance plans” and “performance goals,” which both relate to the concept of planning for specific results.
This references two kinds of targets: the number of people who are expected to be served and the number of people (of those served) who are expected to achieve an outcome.
These targets are identified in planning so they can be assessed in the evaluation phase.
National Association for State Community Services Programs

35. Success and Circumstances
4/17/2017
These two different target numbers produce several different ways of looking at “success.” The success calculations require data to be collected during the planning process and again at the end of program service.
Setting targets is important in the world of performance management.
If you don’t set targets and then evaluate how well you met them, you will never know how you are performing.
If you only know how many you served, you are not being “results oriented” – so what happened? And if you only know how many achieved a result you cannot say if that is good or not. If 20 people were successful is this out of a possible 30? Or out of a possible 300? You need both numbers to calculate success.
National Association for State Community Services Programs

36. We Have a Good Start! From NASCSP CSBG Annual Report 2014:
4/17/2017
From NASCSP CSBG Annual Report 2014:
In addition to CSBG’s performance measurement initiative, the Executive Office of the President, Office of Management and Budget (OMB) has established a government-wide initiative to use performance targets and outcome measures to assess the program efficiency and effectiveness of all federally-funded domestic assistance programs.
As a result, beginning in FY 2004 OCS began to develop and request a report on performance targets, or anticipated levels of result achievement, for four CSBG Performance Indicators.
It is expected that targeting of outcomes will expand to other indicators.
National Association for State Community Services Programs

37. Assessing Ability to Target
4/17/2017
As the data accrue, agencies find out how well they can predict performance outcomes.
To find out the percent of successful targeting, the number of participants who actually achieved the goal (at the end of the program) is divided by the number expected to achieve the goal (set in planning).
The resulting percentage assesses CAAs’ knowledge of how their programs work.
National Association for State Community Services Programs

38. Performance Mgmt and CSBG
One Targeting Percent
We talked about
National Association for State Community Services Programs

39. Another Targeting Percent
The other way to evaluate the success of a program(which we saw earlier) is to compare the number achieving the outcome with the number who were provided service (the total number that could have achieved).

40. Understand the Circumstances
Recognizing When Some Data Needs Special Attention

41. How to target for single or short term services?
4/17/2017
There are programs that an agency provides that are designed to meet an immediate need or crisis situation.
There is no change in status expected.
The recipient of service will still be facing all of the needs, but this one area will not be in-crisis any longer (or at least for a certain period).
In this case you can only target the number to be served, unless you have a system to do follow up.
National Association for State Community Services Programs

42. Dealing with the Impact of One-Time Events
What happens when you have large events only once a year?
Special programs at Thanksgiving/Christmas
Or events that serve large numbers at specific times of year
Fuel programs that peak in winter months
summer programs for school children.
The participants in these events are not going to be tracked for outcomes.
In this case you will target only the number to be served.
Refer to chart in manual.

43. Dealing with the Complexity of Anti-Poverty Programs
Poverty is an enormously complex issue. Reducing poverty, whether for a household or for a community, is difficult and requires a long-term effort on many levels.
“Long term” effort means that you have to acknowledge challenges in data collection practices that may cover several years (or reporting periods) of service.
It is important to develop tracking systems that allow you to identify participants who have made progress but not yet achieved the outcome. (as different from those who have dropped out or are no longer progressing)
Refer to charts in manual.

44. Dealing with the Complexity of Anti-Poverty Programs
Acknowledge that some outcomes depend on the activities of many programs working together.
Operationally distinct programs can contribute to meeting one or more poverty reduction targets.
Each separate program has its own internal targets, but they also share targets related to the long-term goal of family self-sufficiency.
So a participant may be making progress in one program, which is a prerequisite for progress in another program, but does not appear active in both programs. Agency tracking systems should have a way to identify these circumstances.
Refer to charts in manual.

45. National performance indicators
Better identification of the changes that happen
National performance indicators

46. Performance Mgmt and CSBG
The Six National Goals
Goal 1. Low-income people become more self-sufficient. (Family) Goal 2. The conditions in which low-income people live are improved. (Community) Goal 3. Low-income people own a stake in their community. (Community) Goal 4. Partnerships among supporters and providers of services to low-income people are achieved. (Agency) Goal 5. Agencies increase their capacity to achieve results. (Agency) Goal 6. Low-income people, especially vulnerable populations, achieve their potential by strengthening family and other supportive systems. (Family)
The current NPIs are based on the Six National Goals
National Association for State Community Services Programs

47. The network’s long term goals:
Performance Mgmt and CSBG
The network’s long term goals:
Low-income people become more self-sufficient.
The conditions in which low-income people live are improved (producing a thriving or revitalized community).
As we started the TOC work we asked: what are our long term goals. We had agreement that these are the long term goals.
National Association for State Community Services Programs

48. These are actually pre-conditions (or intermediate goals)
Performance Mgmt and CSBG
These are actually pre-conditions (or intermediate goals)
Agencies increase their capacity to achieve results – and be better able to document and analyze the changes they have supported.
Low-income people own a stake in their community – to improve their own social capital and take action for community issues.
Partnerships among supporters and providers of
services to low-income people are achieved – to achieve a specific purpose.
Low-income people, especially vulnerable populations, achieve their potential – to maintain or achieve stability.
National Association for State Community Services Programs

49. How to Improve the National Performance Indicators
Make connections between the indicators and goals.
Why do we think these indicate success in the Six National Goals?
Identify the numbers of individuals, families, communities that achieved the GOALs. Have they achieved multiple indicators?
Connect the services provided with the indicators reported
Is it one service for one outcome? Many services for one outcome? Many outcomes from one service?
Clarify the population that is achieving the indicators
Target population and service population
Unduplicated count?
Think of the logic model. We have identified a broad outcome without numbers. The indicators have the numbers and a greater detail and are connected to a measurement tools and processes. The NPIs are just a snapshot of the one column.
We need to analyze data from the field to determine the amount and type of service needed to produce results that actually matter (are "good enough") in moving the families that receive “outcome producing services” toward the long term national goals. In other words, what produces change. To do that we need to have a new area of reporting that asks for information about services provided.
We need to be able to identify the number of customers who are receiving single services and those who are receiving multiple services. (Differentiate between the “target” population and the “service” population.)
Of those receiving multiple services, identify those who have a comprehensive sequenced plan identifying how the services will achieve customer goals.

50. Performance Mgmt and CSBG
Family Level NPIs BM
National Association for State Community Services Programs

51. Examples of Family Indicators
Performance Mgmt and CSBG
Examples of Family Indicators
Net family household income is increased.
Families reduce reliance on public benefits.
Families improve status in multiple life domains.
Children meet developmentally appropriate milestones.
Vulnerable populations achieve/maintain stability.
Ability to track multiple services to the change they produce.
National Association for State Community Services Programs

52. Performance Mgmt and CSBG
Community Level NPIs JT
National Association for State Community Services Programs

53. Community Indicators
Community Action addresses complex issues that relate to creating thriving communities.
Community Action joins with stakeholders (partners) to improve conditions in low income communities.
Communities are improved by people with low incomes having network connections, and these community networks help improve the social capital of people with low incomes. JT

54. Reporting Community Work
The agency can clearly articulate the purpose of each partnership and the expected changes that will be produced.
The agency can identify the number of partnerships that both support the agency’s activities (agency outcome) and support community change.
The agency can track its involvement in and value to community changes.
The agency records involvement of their customers (individuals with low-income) in community projects.

55. Examples of Community Changes
Change in personal behaviors of community residents – such as reduced litter or graffiti; or increase use of radon detectors
Change in corporate practices – such as grocery stores moved to low income communities; or businesses hire low income community residents
Change in policy or regulation - increase in minimum wage; changes to child care eligibility for working parents

56. Identify those who move to self sufficiency using
Bundled services

57. Bundled, Integrated, Multiple?
It is important to know if a participant has received multiple services, and which services were received.
The question is: do participants receiving multiple services achieve more outcomes than their peers who receive single services?
Are the “multiple” services achieving maximum power? (Which comes when the services are sequenced and are carefully selected in a strengths-based planning process.)
It is the “comprehensive” and “integrated” nature of the services that adds the most value.

58. Hypothesis
Community Action has the ability to determine which “bundle of services” are most effective for certain demographic groups and in certain locals to achieve “success.”
Note: Success maybe be defined in many different ways – and may include the achievement of multiple outcomes (in multiple domains).

59. Putting data to work during analysis
Learning from the Data
Putting data to work during analysis

60. Management based on data
You can’t “manage” performance if you can’t define and measure it.
first collect appropriate data
find out “what happened” (both what went well and what can be done differently)
Identify actions to take to improve or strengthen outcomes

61. Questions for analysis
Was there something unexpected that influenced the outcomes?
Did we not recruit and enroll sufficient numbers to allow us to achieve our target outcomes?
Do we need additional resources?
Did the outcomes validate our assumptions?
Did we make an impact on the identified needs?

62. Performance Mgmt and CSBG
Comparing Data
Program data from year to year
Quantity of service, outcome, cost of program
Your program with other similar ones
What outcomes do other agencies who do this service achieve?
Are our populations similar?
How might differences in population impact differences in outcomes?
Your program with national trends
National Association for State Community Services Programs

63. more resources and information.
Visit
for
more resources and information.
4/17/2017
National Association for State Community Services Programs 63