1. Performance Management
Performance Mgmt and CSBG
Performance Management
Analysis & Use of Data JT
National Association for State Community Services Programs

2. Workshop Objectives
This session will focus on how a review and analysis of agency Community Needs Assessment data can be used to best develop the Strategic Plan and the Community Action Plan.
Assuring the implementation of the full ROMA Cycle can be done while using these data to support and validate the agency’s Theory of Change.
These activities taken together will ensure agencies are matching their ROMA outcome measures with identified needs, and are able to make meaning of their data.

3. Performance Management Framework
Performance Measurement and CSBG
Performance Management Framework
JENAE
National Association for State Community Services Programs

4. Performance Measurement and Management Principles
Performance Mgmt and CSBG
Performance Measurement and Management Principles
Identify the elements that come together to produce success.
Link data elements in a meaningful and useful way.
Assess poverty needs (causes and conditions) within the community.
Identify specific improvements, or results, to be achieved among low-income people and the community.
Specify the protocols that support collection and analysis of the elements. JT
Fall work…
NASCSP

5. Performance Measurement and CSBG
ROMA Logic Model
Organization/Program _________
Level:
__ family __ agency __ community
Need
Service or Activity
Include who will be served, how many people/units of service and time frame.
Outcome
Indicator
Of those who will be served, how many will achieve the outcome in the time frame.
Evidence
The tool that will be used to measure and document success.
Data collection processes and personnel
Mission Statement:
Barbara
Note: the “Actual Results” column is missing from this graphic.
Once service is delivered, actual results must be compared with projected results.
National Association for State Community Services Programs

6. Can Agencies Do This Now?
Performance Mgmt and CSBG
Can Agencies Do This Now?
Implement full ROMA Cycle on a continual basis.
Use systems that allow data to be aggregated, analyzed and reported quickly and efficiently.
Achieve an unduplicated count without comprising program participant privacy
Connect critical data from disparate systems to meet diverse program needs and reporting requirements.
Monitor and manage the quality and effectiveness of services.
Use data to make adjustments for improvement (what’s working, what’s not working, and WHY). BM
National Association for State Community Services Programs

7. Performance Mgmt and CSBG
Can States Do This Now?
Assess the capacity of the local agencies (eligible entities)
Ensure that agencies implement strategies to measure and record outcomes
Monitor agency performance management
Compare plans with actuals
Use data to compare agency performance, statewide, and replicate best practices BM
National Association for State Community Services Programs

8. How People, Services, and Outcomes Connect to Performance Measurement
Analysis

9. Population Served - Characteristics
Performance Mgmt and CSBG
Population Served - Characteristics
Community Action Agencies
Provide a wide range of services
Serve a wide range of people,
Produce a wide range of outcomes .
Analysis of outcome data must include clear identification of “who” received “what” services as well as the numbers served, and the outcomes achieved .
If we served 16.5 million people last year, why did we only report 130 k jobs? This is .6% - just about half of a percent.
This is not the right question, because the denominator of the fraction is only the set of those we served that were seeking jobs. This eliminates at least half of the population (previous slide).
National Association for State Community Services Programs

10. Example: Outcomes and Clients
Performance Mgmt and CSBG
Example: Outcomes and Clients
This graph shows all the people that were served for whom we have age characteristics.
This is 13.7 million people. But notice how many are children!
The number of people who were seeking employment is much less.
We have that information for goal 1, so that makes it easy for us to contextualize this outcome.
National Association for State Community Services Programs

11. Example: Outcomes and Clients
Performance Mgmt and CSBG
Example: Outcomes and Clients
How many of the clients can actually be reasonably expected to obtain specific outcomes?
If a third of the population served by CAAs are under 17, what happened to them?
(family service or direct service to the youth)
What percent of the total population received single or short term services? How many received multiple/integrated services?
Of course, some children benefit from services such as housing, food, other family services.
But others received direct services (Such as early childhood education or support for school success) What percent?
National Association for State Community Services Programs

12. Performance Mgmt and CSBG
Core Population
The individuals, families, groups, or communities an organization = its core population
The agency may not collect a full range of characteristics about everyone in this category, but some key characteristics will be collected.
At the minimum, this population should have a unique agency identifier that would enable an unduplicated count of individuals and/or families.
How does agency holds itself accountable for the achievement of specified outcomes for core populations?
I.E. There is a system for follow up to identify outcomes (change).
Agencies provide services in a variety of ways to the target population they serve.
In some cases individuals and families receive a single service from the agency. It produces an outcome in one life domain area.
In other cases, individuals and families may receive multiple services from the agency but these are not coordinated, integrated nor included in an overall service plan. The agency is able to identify the number of different services the individual/family has received but does not aggregate the information about the family.
It is believed that this number of unduplicated individuals and families will represent the majority of those receiving service from an agency.
However, it is also true that some individuals and families receive what has been characterized as “bundled services,” in which a plan for comprehensive and sequenced services has been created. In this case, the services are followed, assessed and the outcomes that result from the services are observed, documented and discussed as a part of future planning.
National Association for State Community Services Programs

13. Performance Mgmt and CSBG
Service Population
The individuals, families, groups or community members that an organization serves, but does so without expecting outcomes.
agency may collect no characteristics about this population, and may not be able to identify if individuals and families are duplicated in the overall service count.
Identified in 12.2 of MSP?
There are numerous reasons, including the presence of legacy programs; funder requirements; the need to maintain credibility and promote ease of access in local neighborhoods; and the desirability of having a broad base of support when engaging in advocacy.
Examples of a Service Population—youth attending a drop-in center, school-based or community-based informative workshops, a job “fair” or other collection of resources for a given topic such as health, or senior citizen services, a soup kitchen or food distribution where no follow up will be done.”
National Association for State Community Services Programs

14. What Does the Data Do? Can we use the data to demonstrate:
we focused on “Changing Lives”?
we are following up to show the long term impact of our work?
what needs to be done next year to expand on what works?
can we tell what set of services give the best opportunity for results?

15. Management Based on Data
You can’t “manage” performance if you can’t define and measure it.
Our expectations for outcomes must be based on who is served and the needs in the community.
first collect appropriate data
organize the data into useful reports for management
enable agencies to more effectively connect people and distressed communities with the services needed to achieve specific outcomes.
Identify actions to take to improve or strengthen outcomes

16. Performance Mgmt and CSBG
Report Areas to Assess
Quality of the data -- Accuracy, completeness and timeliness of the data included in the report (4 items)
Presentation of the data in the report – it is easy for the reader to find data that is important to him/her? (6 items)
Usefulness of the data – is it relevant to the task of managing the program/service? Do I have access to the data when I want/need it? (7 items)
Performance Focus – includes data on outcomes and performance (3 items)
Depending on where you are in the agency’s hierarchy, you may have more or less access to the system. We have tested this set of statements in a limited way and have found that most middle managers in agencies agree or strongly agree with only about 40% of the items identified.
How useful are your management reports?
assessment tool (handout) is designed to cover several important areas to help you determine if you have a data collection and reporting system that supports both management and accountability efforts.
National Association for State Community Services Programs

17. Performance Measurement and CSBG
Analysis of Data
What skills, expertise, resources are needed to analyze your agency data?
Brainstorm to answer the first question
Possible responses: math, organization, ability to read a report, understanding of the community needs, time
National Association for State Community Services Programs

18. Can You Compare Your Data?
Program data from year to year
Quantity of service, outcome, cost of program
Your program with other similar ones
What outcomes do other agencies who do this service achieve?
Are our populations similar?
How might differences in population impact differences in outcomes?
Your program with national trends

19. Sample Questions for Analysis
Performance Mgmt and CSBG
Sample Questions for Analysis
Was there something unexpected that influenced the outcomes?
Are some populations achieving outcomes at different rates than others?
Did we not recruit and enroll sufficient numbers to allow us to achieve our target outcomes?
Do we need additional resources?
Did the outcomes validate our assumptions?
Did we make an impact on the identified needs?
National Association for State Community Services Programs

24. Performance Measurement and CSBG
Where We Are: ROMA
Internal Challenges
Many CAAs see ROMA merely as a reporting requirement
CSBG Network struggles to see clearly what works for whom under what conditions
Existing Community measures do not adequately demonstrate the depth of work CAAs do
We understand that the NPIs track movement of families up the ladder and often multiple indicators are connected to one family, but this is not understood by people outside the CSBG Network
External Challenges
Conditions of poverty demand different levels of results
Political climate expects the CSBG Network to articulate meaningful results
National Association for State Community Services Programs

25. Where We Want To Be: ROMA NG
Performance Measurement and CSBG
Where We Want To Be: ROMA NG
OCS, States and CAAs use performance data to achieve greater stability and self-sufficiency for families and communities. Each level of the CSBG Network executes the full ROMA cycle
Each level of the CSBG Network uses data on people, services, and outcomes for decision making
The CSBG Performance Management Framework supports improvements in the quality of both individual and community level results achieved by CAAs
The presentation of CSBG Network results provides a clearer story of the impact achieved by CAAs
National Association for State Community Services Programs

26. Performance Measurement and CSBG
How Do We Get To ROMA NG?
Connect ROMA Next Generation with the organizational standards and state and federal accountability measures in a meaningful and clear way so the CSBG Network understands this is one CSBG Performance Management Framework
Establish and adopt a standardized framework for ROMA Next Generation
Provide leadership to invest in people, build organizational culture, and improve data systems
National Association for State Community Services Programs

27. What is our Theory of Change?
The National TOC will help all levels of the network to consider universal basic assumptions as they establish performance management protocols.
However, it is a Local Theory of Change that will help local agencies as they implement the ROMA Cycle.

28. ROMA Next Generation Theory of Change

33. Local TOC + Needs Assessment
The local agency identifies its assumptions about poverty in the community.
Poverty is different in different communities.
Use TOC assumptions to make meaning out of comprehensive community needs assessments.

34. Local TOC & Delivery of Services
Local agencies identify specific strategies and services that they think will produce family, agency, and community outcomes directly related to the locally assessed needs and the mix of resources that are available both within the agency and in the broader community.

35. Local TOC and Planning Inform strategic planning efforts.
Establish specific populations the agency serves (children/youth, senior citizens, veterans, unemployed wage earners, etc.)
Identify specific services that the agency will provide, based on its mission and priorities.

36. Local TOC and Evaluation
The TOC can be a yardstick to understand assumptions behind programs and expected outcomes, measure and analyze agency data.
Allow the agency to test assumptions – did the service produce the expected outcome? Why or why not?

37. A Local Theory of Change Will:
Guide analysis of data, informed by the TOC, which feeds into the Assessment phase of the ROMA cycle as the cycle starts again, so the agency can continuously improve strategies and increase Community Action's effectiveness and efficiency.

38. Take Aways
Community Needs Assessment data should be used to develop the Strategic Plan and the Community Action Plan.
These activities can be taken together to ensure agencies match their ROMA outcome measures with identified needs, and are able to make meaning of their data.
The full ROMA Cycle can be implemented by using these data to support and validate the agency’s Theory of Change.
Reports can be strengthened to ensure connection to both outcomes data, analysis of services, and the effectiveness on populations

39. Performance Measurement and CSBG
Contact Us
For more information please visit our website at
National Association for State Community Services Programs