1. Using Data to Inform Policies and Programs
Day 1 - Session 2
In this session we will discuss the importance of using data to improve programs.

2. MEASURE Evaluation is funded by the U. S
MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) and the U.S. President's Emergency Plan for AIDS Relief (PEPFAR). Views expressed in this presentation do not necessarily represent the views of USAID, PEPFAR or the U.S. government.
MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University.

3. Session Objectives
Understand the importance of using data to make programmatic decisions
Apply data use principles through case study discussion
By the end of this session, the learner will be able to:

4. Demand for Data
Demand – the value stakeholders place on data, whether or not the data are used
Demand exists if
Specific questions need to be answered and data are sought to answer them
A decision needs to be made and data are sought to inform it
The term data demand is related to the value stakeholders place on data, whether or not the data are used. Demand exists if:
Specific questions need to be answered and data are sought to answer them
And/or if a decision needs to be made and data are sought to inform it 4

5. Data Use Use refers to the decision-making process
A decision maker uses information if he/she
Is explicitly aware of the decision to be made or question to be answered
Considers relevant information in making the decision, even if the information is outweighed by other factors
The term data use refers to the decision-making process. A decision maker uses information if he/she:
Is aware of the decision to be made or question to be answered
Considers relevant information in making the decision, even if the information is outweighed by other factors
Data use is NOT dissemination, it needs to include the review of information to help make a decision or answer a question. 5

6. Context
Pressing need to develop health policies, strategies, and interventions
Within the context of a high disease burden, a growing population, and insufficient health services, it is extremely important for governments/donors/programs to make the best use of their limited resources. The need to develop strategies, policies, and interventions that are based on data is urgent. As a result, many programs have established or are establishing data collection systems to track critical information. 6

7. Why address data demand & use?
Let’s first review why we are addressing the topic of using data in decision making … the picture above may look familiar to many of you. In today’s environment, there is emphasis on accountability and tracking of the services provided at the community and facility levels. Data requirements from government and donors have grown exponentially to the point where some providers have pages and pages of forms to fill in daily. Frequently, after the data are collected, the provider summarizes them in summary reports and sends them to the required supervisor. After that, the data may be left clogging workspaces (like above) or spilling out of filing cabinets and closets.
Very rarely are data used to monitor programs and make decisions beyond individual patient care. This is a huge lost opportunity, for data are critical to the program improvement and decision-making process.

8. Barriers to Data Use Complex data systems
Inadequate flow of information
Too much data
Lack of process for identifying which data to use
Do not have the data you need to make an informed decision
We have talked about the fact that the use of data is important for program improvement and that often data, while available, are not be used for decision making. Let’s discuss some of the reasons why data are not used as often as they could be.
(Read slide.)
There are many more barriers to data use such as the technical capacity of staff and behavior and attitudes around data use, but in this training we are going to focus on the two barriers highlighted in yellow. First we’ll talk about the data you currently have available to you.

9. Data Sources for Project Decision Making
Types of data or information
Anecdotal evidence
Project documents
M&E systems
Other data sources (HMIS, surveillance)
Special studies (e.g., HIV/AIDS studies, malaria)
Additional data collection
As mentioned in the last slide, we are going to discuss the data and information that are currently available to you in your job. In your work, where do you get the data that helps you understand what is going on with your program? And how do you know whether you need to make changes to it?
Note to Facilitator: Give time for participants to respond and jot down their responses on a flipchart. Then click on animation to reveal some data sources that are often available. You’ll note that project documents and M&E systems are highlighted in yellow since this is often available at the project level.
Program managers, policy makers, and others rely on different types of data/evidence to make decisions. Some evidence is more rigorous than others – for example, there is anecdotal evidence such as what people say they have heard – e.g., clients really like the new approach we are using for condom distribution. A decision maker might say, “Well, if the clients like the new approach, let’s keep using it.” The converse of that is if they hear that some people don’t like the new approach, the manager may decide to cancel or change the activity.
Other times, people may rely on data from different types of data sources (M&E systems, Health Management Information Systems (HMIS), or other special studies) and even collect additional data. We’ll talk more about different types of data in some of the later sessions.
The type and breadth of evidence one uses to make decisions is based on several factors such as data availability, data quality, funding, timeframe for decision making, and other factors, but ideally decisions should be made based on some type of information. For many of you, the indicators reported on in the M&E system is often a good place to start.

10. M&E Systems Include Indicators
An indicator is a quantitative or qualitative variable (something that changes) that provides a simple and reliable measurement of one aspect of performance, achievement, or change in a program or project
M&E systems include the indicators that the program has identified to measure achievements. Let’s review what an indicator is.
(Read slide.)

11. An Indicator can be a: ratio percentage average classification number
rate
ratio
percentage
average
classification
number
index (composite of indicators)
Indicators can be presented in a variety of ways, and there are often standard indicators for different types of programs and from different donors. These are examples of the different ways indicators can be expressed. What is important is that the type of indicator and way it is calculated be clearly articulated through indicator reference sheets and/or other documentation. Once the data have been collected, data are analyzed and presented either in tables or charts.

12. Target Definition
A specified level of performance for a measure (indicator), at a predetermined point in time (i.e., achieve “x” by “y” date)
Overall target
Annual targets
Indicators are most helpful for decision making when they are reviewed against targets.
Let’s take a look at this definition of a target. Essentially, a target is a number that you set (specified level of performance) for a given indicator by a certain period of time. There are two types of targets.
An overall target measures expected performance for the life of the program – whether it is one year, three years, or five years. It defines what you want to accomplish for an indicator by the end of the program. For example, the government of Namibia may want the percentage of males who have comprehensive knowledge of HIV to reach 76% in three years – by 2017.
Annual targets measure the expected performance for each year of the program. For example, in the above scenario, let’s say the indicator was at 62% in 2014 at the start of the program – getting to 76% in three years would be a large increase (14 percentage points). Setting annual targets will help to break it down. The program might set the annual targets at 66% by 2015, up to 71% by 2016, and up to 76% by 2017.

13. Why Set Targets? Targets help program staff with: Planning
Staffing and service delivery
Commodities
Monitoring progress
Break long-term goals into manageable pieces
Check progress on indicators
(Read slide.)
Targets play an important role in ensuring you’re on track with your program.

14. What is Monitoring?
Continuous systematic process of collecting, analyzing, and using information to track the efficiency of achieving program goals and objectives
Provides regular feedback that measures change over time
An unexpected change in monitoring data may trigger the need for a more formal investigation of activities
As you know, monitoring measures how well program activities are being performed. This information is sometimes collected on a routine basis, such as through staff reports, but may also be collected periodically in a larger scale process evaluation effort.
Demonstrating changes or improvements through continuous monitoring will allow the team to track progress in implementing…(read slide).

15. Has the program met its goal?
% of New Enrollees Tested for HIV at Each Site, by Quarter
Here is an example of program monitoring for an indicator coming from an M&E system. It is a result of some preliminary data analysis from the AIDS Relief program in Nigeria. It gives the percentage of new enrollees tested for HIV at each site by quarter. In the green line across the top we can see the target is to test 50% of new enrollees at each site in each quarter. In interpreting this, we see that sites 1 & 3 have met their targets, but that site 2 has not; it is at 30% new enrollees tested.
Now that the analysis has helped us determine whether the program met the indicator target for percent of new enrollees tested for each site, it’s time to interpret why site 2 did not achieve the target of 50%. Now we’ll walk through some steps to follow for interpretation.
Target
Data Source: Program records, AIDS Relief, January 2009 – December quarterly Country Summary: Nigeria, 2008

16. Interpreting Data
Adding meaning to information by making connections and comparisons and exploring causes and consequences
Facilitator note: Click animation to reveal “Relevance of finding.”
This step involves adding meaning to information by making connections and comparisons and exploring causes and consequences – it helps you put the data in the context of your program. At this point, you might want to ask some of these questions:
How far from the target is the indicator? How does it compare (to other time periods, other facilities)? Are there any extreme highs and lows in the data? Is there anything that surprises you in the data?
Facilitator note: Click animation to reveal “Reasons for finding.”
When seeking potential reasons for the finding, we often will need additional information that will put our findings into the context of the program. Supplementing the findings with expert opinion is one way to do this. For example, talk to others with knowledge of the program or target population and who have in-depth knowledge about the subject matter, and get their opinions about possible causes. Or look at what else might be going on in the environment (e.g., political or security-related issues) that might help explain the gap. There might be an obvious reason (e.g., the project director had not hired enough staff). When seeking potential reasons for the finding, we often will need additional information that will put our findings into the context of the program.
For example, if your data show that you have not met your targets, you may want to know if:
--The community or target population is aware of the service
--A sufficient number of awareness campaigns have been implemented
To answer this, you could talk to community leaders, program managers, subject matter experts, or providers to get their opinions.
Sometimes ad hoc conversations with experts are insufficient. To get a more accurate explanation of your findings, you often will have to consider other data sources.
Relevance of finding
Reasons for finding

17. Reasons for Finding Why are we seeing this trend?
How do these data compare with data from other facilities?
Are there external factors contributing to the findings? Examples include seasonal, political, environmental, cultural, or socioeconomic factors
Could the trend be the result of improved data collection?
What other data should be reviewed to understand the finding?
These are some of the questions you might want to ask when reviewing the data you have to help you understand the reasons for your finding, in this case, why Site 2 is not meeting the target.

18. Interpreting Data
Adding meaning to information by making connections and comparisons and exploring causes and consequences
If there is not an obvious reason for the insufficient performance, you may need to consider other data (either existing indicators from the M&E system or data from other sources) to help us explain why Site 2 did not meet the 50% target.
Relevance of finding
Reasons for finding
Consider other data

19. Considering Other Data
What are other data sources that might help explain why the percent of new enrollees might be missing the target in Site 2?
Commodities data
Patient/provider ratio
Distance from health facility
Health facility operating hours
Other surveys or data reports (DHS, AIDS surveys)
Technical assistance/supervision reports
Note to Facilitator: Read first bullet and give participants time to respond. You may need to go back to slide 14 to show the chart. Jot down participants’ responses on flip chart paper. After they indicate other information that might be useful, click to reveal animation and a list of other information that could be reviewed to help interpret the performance of Site 2.

20. Interpretation – conduct further research
Data gap conduct further research
Methodology depends on questions being asked and resources available
Relevance of finding
Reasons for finding
Consider other data
Collect additional data
Once you review additional data, it may become apparent that these data are not sufficient to explain the reasons for your findings – that a data gap exists. In these instances, it may be necessary to conduct further research.
The types of research designs that are applied will depend on the questions that need to be answered, and of course will be tempered by the feasibility and expense involved with obtaining the new data.

21. Case Example – Information Use for Vulnerable Children Programs
Volunteers in a rural district in Mozambique provide psychological, social, and economic services to vulnerable children. The volunteers told the program manager that they provide more services to girls than boys and that they are concerned they are not meeting boys needs.
Let’s now review a case example that illustrates the way an orphans and vulnerable children (OVC) program in Mozambique used data in different ways to make decisions about the program.
Note to facilitator: Ask a participant to read the slide.
We will present three different scenarios for what the program manager did with this information and how she ended up making different types of decisions based on the information.

22. Scenario 1
Action – The program manager called a meeting of all the volunteers in the district and asked them to discuss reasons why they thought more females were served than males. The team met and volunteers determined that the males did not feel comfortable talking to the them and avoided them when they came to visit the home.
Decision – The program manager decided to mandate gender sensitivity training for all volunteers.
Note to facilitator: Ask a participant to read the slide. After the participant reads it, ask what type of evidence the program manager and volunteers based their decision on. Record responses on a flipchart. Answers could be anecdotal, perceptions of key informants.

23. Scenario 2 – Vulnerable Children Served
Male
Female
Total N %
District A 98 54 82 46
180
District B 8 21 30 79 38
District C 65 41 94 59
159
District D 13 62
District E 55 35
100
155
District F 60 33
120 67
First the program manager requested additional gender disaggregated data from the district where the problem was first identified (the row in blue). Then she requested gender-disaggregated data from all of the districts in the province and noticed a trend – that in addition to District B, Districts C, D, E, and F also had a higher proportion of female children served by the program.

24. Scenario 2 – % of Vulnerable Children Served by Program Area by Gender
Psychological
Social
Economic
Male
Female
District A
22%
78%
45%
55%
54%
46%
District B
32%
68%
47%
53%
District C
42%
58%
51%
49%
50%
District D
12%
88%
48%
52%
District E
15%
85%
District F
Next, the program manager asked for gender disaggregation by service area. From this review, she noted that the gender balance in service delivery was relatively similar for males and females with the social and economic services. He noted, however, that the service delivery for psychological services was skewed, with many more females than males receiving services in all of the districts.

25. Scenario 2
Action 1 – The program manager requested gender-disaggregated data from the M&E officer to confirm what she heard.
Action 2 – She then requested this information from M&E officers in other districts and noticed a trend.
Action 3 – She also requested gender-disaggregated data for each of the service delivery areas.
Finding – Gender discrepancy was primarily for psychological services.
Decision – Develop 2-day training curriculum aimed at improving volunteer interactions with boys during counseling.
Let’s summarize what actions the program manager took in Scenario 2.
Note to facilitator: Have another participant read the slide.
In this scenario, what type of evidence did the program manager use? (wait for participants to respond) – Responses could be M&E indicators, disaggregated by service delivery type, gender, and district.

26. Scenario 3
Action 1 - The program manager did the same things as in Scenario 2, but before making a decision, she decided to collect additional information.
Action 2 - She conducted two sets of focus groups in a sample of two districts. One set was with volunteers and the other was with male vulnerable children. The FGDs asked questions about how services are carried out, and males perspective of the services.
Now, let’s look at a final scenario.
Note to facilitator: Have a different participant read the slide.

27. Scenario 3, cont.
Findings: Volunteers primarily visit homes in the morning when males are working in the field and are unavailable.
Decision: Volunteers will schedule visits for psychosocial counseling in the late afternoon or on rest day.
Now, let’s look at a final scenario.
Note to facilitator: Have a different participant read the slide. Ask participants what evidence the program manager used in Scenario 3. A response for this could be M&E data supplemented with qualitative data collection (FGDs).

28. Discussion
1. Which of the three scenarios do you think is best? Why? 2. What was the difference in the types of decisions made based on the findings? 3. What implications do the different types of decisions have (cost, time, effect)?
Facilitator note: The answers to the questions posed on the slide are as follows:
1) The third scenario is best because the program manager looked at all the data available to her to identify the specific problem. She then collected additional data to understand why the problem was occurring.
2) The decisions in scenario 1 and 2 were to 1) conduct gender sensitivity training for all volunteers and 2) to develop 2-day training curriculum aimed at improving volunteer interactions with boys during counseling. Neither of these decisions would have actually solved the problem because the problem was not caused by the volunteers inability to interact with boys. It was actually caused by a scheduling problem.
3) If scenario 1 & 2 solutions had been implemented, time and money would have been spent creating curriculum and implementing training that would have had no effect on reaching more boys.

29. Group activity 7 people come to front of the room
Arrange each person in order of height
Are they ordered right?
Would anyone else have done it differently?
Any other ways to judge height more systematically?
Now let’s talk about how we collect additional information.
-Take off shoes hats
-Doctors records
-Bring tape measure
Need to set up a systematic way to measure – standardized protocols