1. Evaluation – Analysis of the Data for Boards
Reviewing “what happened” and comparing with plans
A Video Series for CAA Board Members
Objectives of this video will be to help the board identify:
agency performance -- what is working and what is not
the “why” behind the performance
how do they find out,
what does it all mean,

2. Implementing ROMA – A Video Series for CAA Boards
Introduction to Implementing ROMA for Boards
Community Needs Assessments
Creating a Local Theory of Change
Strategic
Planning
Implementation of Services and Strategies
Observing and Reporting Results
Analysis and Evaluation
This presentation is a part of a series that is intended for Community Action Agency Board Members interested in deeper understanding of
How to help your Community Action Agency increase its capacity and its results.
How to meet and exceed Organizational Standards
The Roma Next Generation Video Series consists of these modules.
With me today to present this particular concept is Dr. Barbara Mooney, the Director of the Association for Nationally Certified Trainers. She is co-author of Introduction to ROMA, Research Fellow for the National Association for State Community Service Programs, consultant to the national Community Action Partnership, regional coordinator for Region Three Performance and Innovation Consortium and contributing author to Temple University's Strengths-Based Family Worker Training curriculum.  Over the past five years, she has worked on the development of the OCS Performance Management Framework.
In addition we have …..

3. The next phase of the ROMA Cycle we will explore is that of Evaluation – Analyze data and compare with benchmarks.
The organizational standard related to this:
Standard 9.3
The organization has presented to the governing board for review or action, at least within the past 12 months, an analysis of the agency’s outcomes and any operational or strategic program adjustments and improvements identified as necessary.
Another standard that is a part of this process is
Standard 6.5
The governing board (or advisory body) has received an update(s) on progress meeting the goals of the strategic plan (or comparable planning document) within the past 12 months.
 These periodic updates provide opportunity for ongoing evaluation and analysis of the agency’s progress.

4. Board Roles
“At least each year, the board will receive the agency’s analysis of:
Outcomes
Operational or strategic performance
And will be presented with any suggestions for adjustments and improvements that staff have identified as necessary.”
Standard 9.5 says that the board gets a report that includes both the outcomes the agency achieved and the context in which these outcomes occurred.
If there are operational and strategic program adjustments/improvements that were identified, that means that those things were also analyzed -- as well as the outcomes. So the board should not just accept a report that says (for example) “300 individuals got jobs” – but they would also be wanting to know “how did we help those individuals get those jobs?” That would be part of the analysis.
About suggestions for adjustments and improvements –
Who is involved in the analysis process? The Board may have members who are interested in this kind of work, and may participate in a committee designed to aggregate, sort, suggest, etc.
It is not necessary for the Board to be involved in the actual analysis – but they must make sure that there is a process in place that includes staff or consultants with sufficient knowledge and skill to do the work.
Once the data is analyzed, the committee or team or lead staff person will create some suggestions that could lead to improvements (or to maintaining successful programs). The Board is responsible for considering these suggestions and making necessary decisions.

5. What Do We Mean By Evaluation and Analysis?
Evaluation is about looking back to see if we did what we thought we would do
Evaluation helps us consider the outcome or the value of what is being reviewed using established criteria
In the ROMA Cycle we say: Compare with benchmarks
Additionally, this includes Analysis of the data:
Analysis the process of breaking a complex topic into its parts to get a better understanding of it
It turns the raw data into information
Analysis is ongoing, throughout the entire ROMA Cycle
Evaluation happens at the end
after the program, service or strategy is completed
once the plan has been implemented
after the reports are compiled and submitted.
It is about comparing with “criteria” or “a set of standards” – and in our network we use the benchmarks we have set for ourselves to see how well we have done.
In other words, the plans you set for yourself have included the criteria that you will now use to judge how well you have done.
Analysis is on going throughout the entire ROMA Cycle.
It is a careful in-depth study of something to learn about its parts, what they do, and how they are related to each other;
It helps to form an explanation of the nature and meaning of something.

6. Why Analysis of Data Is Important
Analysis of the data is how we learn
It is how we understand:
What we are doing
Why we are doing it
How well we are doing it
And what is happening
Analysis of our data helps to identify the decisions you need to make and provides the information needed for decision making
Doing data analysis is bound to cause you to discover that at least some of what you thought just isn't so – which means you may have to change your approach or do some things differently
As a board member you are not going to be engaged in the actual data collection or measurement processes which we discussed in the last segment, or in the actual analysis process (this segment).
But as we have said previously, you must understand what the processes are and why they are important.
Remember: If your agency is to be “data centric” - as expected of Next Generation agencies -  the culture that fosters this approach has to start with the Board. You have to demonstrate an interest in and a value of the analysis of your agency data (beyond just Reporting) – and understand that you will use the analysis of the data for decision making going forward.
Because the Board is receiving regular updates about progress toward meeting the agency’s goals, this is the time to find out if they met them in the projected time frame. And what decisions are needed as you go forward: Are changes needed? Additional support needed for a successful program? ??
Once you identify the decisions you need to make, the data will help you make those decisions based on reality not on personal beliefs or feelings.

7. Data Analysis Useful Information Figures Facts Stories
The analysis process produces useful information -- suggesting conclusions and supporting decision-making.

8. Prepare for Data Analysis
Assure the data is complete, accurate, reliable, and timely
Identify questions you have that the analysis of the data can answer
Aggregated the data into meaningful categories so it is easier to understand the relationships
Make sure you are starting with “good” data.
If the data is flawed, you may come up with incorrect conclusions in your analysis. That is a waste of time and can produce errors in future actions that waste other resources.
Remember these
Standard 9.1 The organization has a system or systems in place to track and report client demographics and services customers receive.
Standard 9.2 The organization has a system or systems in place to track family, agency, and/or community outcomes.
“Aggregation” means to bring things together. In the case of agency data, you may bring together all the data related to one program (participant demographics, service units and outcomes for example) or bring all the data related to participants (age, location, family type, etc) from all the agency programs.
Aggregation can also bring together “facts” and observations (experiences, stories) to see the whole picture

9. Understand Data Specified by a Funding Source
Sometimes the source of your funding gives you an outcome you are to achieve with their funds AND a target that you are expected to hit (number to be served, and number to achieve the specified outcome)
This can be good news, because it is easy to determine what you are expected to do
It may be bad news if the imposed target is difficult to hit, or if there are other outcomes that you want to track
Yes it is important to understand the data related to this kind of outcome.
However, one of the limitations of having funding source targets is that it can create a mental barrier – preventing you from seeing what other outcomes could be tracked (in addition to those required).
You want to see how ALL your services fit together to produce outcomes that move individuals, families and communities toward agency identified outcomes. Remember in the Local TOC discussion the difference between looking at each program separately and bringing them together for common goals? You are encouraged to have a WHOLE AGENCY approach, which will include funder imposed targets within your larger understanding of the “targets” you want to achieve.
Be sure you understand the difference between outcomes that are specified by a funding source and the “big picture” agency outcomes you have identified in your own local Theory of Change.

10. Asking the Right Questions
The most important first step in making meaning from the data
Asking the Right Questions

11. Some Questions You Want To Answer With Your Data
Did we do what we thought we would do?
Did we serve the population we thought we would serve?
Did we make an impact on the identified needs?
Can we tell what services (or set of services) produced the best opportunity for results?
Are some populations achieving outcomes at different rates than others?
Did we recruit and enroll sufficient numbers to allow us to achieve our target outcomes?
Do we need additional resources?
Was there something unexpected that influenced the outcomes?
Who might have some insight into these questions?
Find ways to discuss the data with others who may have insights

12. Ability to Target How close did we come to our projections?
The relationship between the number of outcomes you projected and the number you actually achieved can be stated as a percent that shows your agency’s targeting ability.
We talked about the importance of setting targets during Implementation.
Now we want to consider what the percent of targeting success means.
Ability to Target

13. In the Reporting Section we made these comparisons
How well did we target?

14. Calculating Targeting Ability
In our after school program, 35 actually improved grades
- We calculate our targeting ability by dividing 35 by the number that we projected which was 40
35 ÷ 40 = 87.5% success rate for our ability to target performance
In the employment program, 20 actually got jobs
- Again, we calculate by dividing by the 15 that we targeted to get jobs
20 ÷ 15 = 133% success rate for our ability to target performance
In the housing program, 80 actually maintained housing
- Divide by the 90 we projected or targeted to maintain housing
80 ÷ 90 = 89% success in targeting
This percent will allow you to analyze how successfully you can project your performance.
This calculation is a part of the Annual Report – it is auto calculated in the report, using the information that is entered.
The success rate is something to consider as you are setting targets in your next Planning Phase.
It is expected that your success rate will be between 80% and 120% - giving a plus or minus 20% as a range within which you are expected to operate.
What does it mean to over or underestimate your performance?
If you are over estimating, you may need to follow up to see what additional barriers are preventing achievement of outcomes.
Or you may find that you have not served as many people as you thought you would and so that impacts your outcomes. (Note: be careful that you don’t include the number served in this calculation as it is just about the relationship between projected and actual outcomes)
It may look like the 133% success in the jobs program is better than the success in the other two programs. But notice that it is over the 120% guideline.
This means that you are under estimating your performance. Some say you are “low balling” your expectations so it will look like you are doing better than projected. This, however, also demonstrates that you do no know what your services will produce. That you have not paid attention to some aspect of the programming that would impact on your outcomes.
There is always a way to explain differences in target and actual performance. If you got more people jobs, maybe there was a new business in town that needed employees. If you got fewer jobs, maybe you lost a business and the job market was flooded with people who were displaced.

15. Using Information from Data Analysis to Make Decisions
The analysis of your data should lead to your agency maintaining or improving quality services and producing outcomes
Of course you will want to use the information you have developed to improve your knowledge about the issue.
What caused the trend to turn? What else do you need to know?
Using Information from Data Analysis to Make Decisions

17. Analysis Action
Your agency should be able to rely on effective analysis and interpretation of all the data
Once the agency has “made meaning” out of the data you will determine appropriate actions to take 
High performing agencies depend on using the information gained from the analysis to improve results for families and communities
The actions that result from the analysis of the data are found throughout the phases of the ROMA cycle
Data must be turned into information to support decision making -- every day in every decision at the operational level
For the Board, long term strategic decisions should be based on the analysis of facts that can be sourced back to reliable and accessible data.
It is the application of this analysis to the agency’s actions that will improve service delivery, the management of programs and services and produce better results for families and their communities

18. Find Out What services produced the outcomes/results?
One? More than one?
How can we track the multiple services connected to each customer?
Who achieved the results?
What was the age, income level and income source of those who achieved? What about education level? etc.
Did the individuals who achieved one outcome achieved any others? How many individuals achieved multiple outcomes?
Connecting the people to the services to the outcomes is what ROMA Next Generation is all about.
We have to know if what we are doing is addressing the identified needs in the community and if the people we serve are better off!

19. (To help answer questions you found in the analysis process)
Continuing the Cycle
Think about your next Community Needs Assessment
(To help answer questions you found in the analysis process)
What data do you want to collect that you haven’t collected before?
Was anything missing from your previous data collection efforts?
Do you want to gather data from additional people?
Other thoughts……
Once the data is analyzed and some decisions are made, this should lead back into the Assessment phase of the cycle.
The information you have identified should help you look at the way you do assessment of needs.
What else do you want to know?
Are you getting enough of the right data? (maybe there are data elements you previously included in the CN A that you do not need?)

20. Taking Action Get reports that aggregate information in useful ways
Assure there is a system for analysis of the data and a way to “make meaning” about what is working
Compare the projected indicators with the actual indicators to understand what happened
Make decisions base on the analysis of the data
Put decisions into action
Decide what else you need to know in the next Community Needs Assessment process