Module 1: Key concepts in data demand & use

Module 1: Learning objectives Understand the value of data collection and use in program monitoring and improvement key concepts in data demand and use decision-making process At the end of this session the learner will… Read slide.

Why address data demand & use? Let’s first review why we are addressing the topic of using data in decision making … the picture above may look familiar to many of you. In today’s environment, nurses have become overwhelmed with collecting data on the services they deliver. Data requirements from government and donors have grown exponentially to the point where some providers have pages and pages of forms to fill in daily. Frequently, after the data are collected, the provider summarizes them in summary reports and sends them to the required supervisor. After that, the data may be left clogging workspaces (like above) or spilling out of filing cabinets and closets. Very rarely are data used to monitor programs and make decisions beyond individual patient care. This is a huge lost opportunity, for data are critical to the program improvement and decision-making process.

National-level Policymaker, Nigeria “… without information, things are done arbitrarily and one becomes unsure of whether a policy or program will fail or succeed. If we allow our programs to be guided by empirical facts and data, there will be a noticeable change in the impact of what we do.” National-level Policymaker, Nigeria In 2007, MEASURE Evaluation conducted a data use assessment in Nigeria, which involved interviews with a range of professionals at the national, regional, and facility levels. One of the national policymakers interviewed, stated… (Read slide.)

What are data? Definition Data sources Service delivery statistics Census Surveys, evaluations, research studies Sentinel surveillance Budget information Let’s start at the beginning and talk about what we mean by ‘data.’ The definition of data is – factual information in raw un-synthesized form. Data can be either numeric or narrative. Common data sources include: Service delivery data – This is a data source you will all become very familiar with, as you are the primary data collectors. Service delivery data reflect the services you provide in your clinical settings. During this module, most of our discussions will rely on service delivery data as a data source. Census – A census is a counting of the people in a specific geographic area. You are all probably familiar with your national census that enumerates the numbers of people in your country. This is also an important data source in the health field because it allows us to calculate the numbers of people in need of specific services. Surveys, evaluations, research studies – These capture information on specific topics and populations. These data sources help us to answer specific questions and frequently give us information that can help us to improve our programs. Surveillance – Tracks the prevalence of specific diseases in a target population over time. Prevalence refers to the total number of cases of a disease in a given population at a specific time. These data help us to estimate the burden of specific diseases. Budget – Last, there are other sources of data that we don’t often consider. For example, budget information can help us track our expenditures and illuminate what specific program elements cost.

Data Infrastructure Routine data Non-routine data National Census Regional Evaluations District Surveillance Facility Surveys Regarding data, this slide represents what we refer to as the data pyramid. On the left, you see the data that originate in the health facility – the service statistics. These data are referred to as ‘routine data’ because they are collected – by you, the provider – on a regular basis each time a service is delivered. These data usually are summarized monthly at the facility level and then are sent up to the district, followed by the regional and then national levels. At each level, the data are summarized and hopefully are used to inform decision making. You will note that it is the facility-level data that eventually make their way to the national level to form the basis of national policy and programs. The provider’s role as the primary data collector for the state is critical. On the right hand of the slide, you see what are referred to as ‘non-routine data.’ These data sources help us to answer specific questions and frequently give us information that can help us to improve our programs. As you move up the pyramid, the amount of data available about a specific individual or case is less and less. However, these summarized and aggregated data become more useful to decision makers outside of the facility.

Why do we collect data? Track changes in program performance over time Monitoring Attribute program outcomes to their causes Evaluation As the previous slide showed us, the data we collect are used at all levels of the health system to track progress in the delivery of health services and to evaluate the outcome and impact on the health status of our communities. Specifically we collect data to: ….. Note to facilitator: Read slide. Monitoring and evaluation is very important in the improvement of health services. In this module we will be addressing only the concepts of monitoring.

Purposes of monitoring and evaluation Determine whether a plan or program is on schedule with planned activities Assess whether a policy, plan, or program has produced desired impacts Generate knowledge: Identify factors that influence health outcomes Inform policy, planning, or program decisions Monitoring and evaluation (M&E) is an essential process in providing effective and efficient services and ensuring that programs are relevant and successful. For example, it helps us to make informed decisions about such questions as appropriate staffing and other necessary resources. M&E helps us know whether a program is being true to its stated goals and objectives. For instance, … M&E helps us evaluate whether our programs are having their desired impact. If we want to know how a program is performing, we might assess it against targets that have been set for specific indicators by the program or funding agency or government. For instance, we might assess if a breastfeeding program is reaching its goals in providing counseling to pregnant women during ANC and by the percentage of children under six months who are exclusively breastfed. However, for M&E to have this desired impact, M&E data and information must be used strategically by programs, service delivery organizations, policymakers, and other stakeholders. 8

M&E is not an enemy Policymakers, program managers, and M&E/strategic information specialists can be partners Strong decision making and management rely on high-quality M&E / strategic information Data quality is linked to data use Oftentimes, people see M&E as policing or a process meant to criticize and undermine their work. While an M&E system may at times uncover program weaknesses, the purpose of this discovery is to address the weakness, not to criticize its existence. It must always be remembered that an M&E system’s primary purpose is to tell us what is happening in our programs so that we can praise our progress or improve on our weaknesses. Policymakers, program managers, and M&E specialists can be partners in progress – designing new programs, making improvements to plans and programs, policymaking and, at the facility level – identifying gaps and opportunities. Strong decision making and management rely on high-quality M&E or strategic information. Without information, it is difficult to make an effective and successful decision or manage shifts in a program. Finally, data quality is linked to data use. As increased attention is being paid to data quality, especially at the lower levels, it’s important to know that data quality naturally improves as individuals and organizations understand how useful the data and information can be to them.

Why are data important to a provider? Understand program performance Strengthen programs and improve services delivered Advocate for additional resources, policies, and programs Ensure accountability to civil society Now that we have discussed the importance of data and the M&E process, let’s talk about why data are important to you – the provider. As we mentioned in the previous slide, data help the provider to understand how their everyday work contributes to the goals of the facility. By regularly monitoring data, providers can identify programmatic concerns and/or problems and develop solutions to overcome them. Regularly monitoring data can also help a provider to identify areas that need more funding or new program approaches. And last, regularly monitoring the progress of the health facility provider ensures that the needs of the communities they serve are being met. 10

Using data helps providers to… Ask critical questions Open lines of communication Note to facilitator: Read slide. Improve services 18 30 21 29 22 30

Monitoring and Evaluation allows…. data-based decisions which lead to… better health programs and better health outcomes The use of monitoring and evaluation data allows providers to make data-informed decisions to design and manage health programs, which results in better health outcomes. 12

Role of the provider in M&E and data use Ensure timely and quality data Facilitate facility-level data analysis, interpretation, and display Identify problems /concerns and develop/suggest solutions to address them Request feedback from management on decisions made Now that we have discussed the importance of M&E and data use to the provider, let’s talk about the role of the provider in these two activities. First, and very important, is the provider’s role in ensuring timely and quality data. In an earlier slide on the data pyramid, we talked about the provider’s role as the front-line data collector. In this role, it is the provider’s responsibility to ensure timely and quality data. Facilitating facility-level data analysis and use is another key role of the provider – and what we are here to talk about today. Providers need to monitor and track services to diagnose problems, develop solutions, and advocate for change. Providers are also critical in the feedback process. By requesting feedback from management, they are encouraging the demand for data. Providers have the right to know how the data they collect are used by higher levels in government. Before we move on to the next topic, I’d like to talk a bit more about the provider’s role in collecting timely and quality data. Note to facilitator: Click the mouse to highlight the first bullet.

Data Quality Accurate Complete Timely When we talk about collecting data that are accurate, complete, and timely, we are talking about data quality. Data quality refers to: Accurate data – meaning that the data collected are true and without errors Complete data – meaning that all data requested on a data collection form are present – there is nothing missing Timely – meaning that the data are recorded and reported by the time they are requested. We don’t want to delay for months to report on our services because by then the information we are reporting on is old and does not reflect the current situation. As the provider in a clinic, you are the primary person collecting data. You collect data during your daily activities when you fill in a client chart, a client health card, a clinic register, a commodities log, or any other data capture form – on the services you are delivering. Ensuring the quality of those data is one of your primary responsibilities. Data quality is important because if you are making decisions based on data, you want your data to be as accurate as possible.

Monthly summary of child health & nutrition services – October 2010 Child health and nutrition Children needing follow-up Male Female Total Marasmus 4 9 13 Kwashiorkor 6 5 10 Anemia 22 17 93 Faltering weight 24 55 Other (specify):___________ 1 Let’s look at an example of poor data quality. Here we see a section of a monthly clinic report on child health and nutrition. This table shows the total number of children that received follow-up services for the listed conditions in October 2010. In the rows, each condition is listed, as is the number of male and female children followed up for those conditions. So, the 3rd row shows marasmus, a type of malnutrition, and that there were 4 male and 9 female children followed up for that condition in the month of October – totaling 13 children for marasmus. Let’s look at the accuracy and completeness of the data in this report. Note to facilitator: Go to next slide to reveal animation.

Monthly summary of child health & nutrition services – October 2010 Child health and nutrition Children needing follow-up Male Female Total Marasmus 4 9 13 Kwashiorkor 6 5 20 Anemia 22 17 39 Faltering weight 24 Other (specify):___________ 1 The first error I see is that data were left out for follow-up on faltering weight. Here, we don’t know if there were no male children treated – which would be represented by a zero – or if the provider forgot to fill in this section – which would result in an underreporting of this condition. Note to facilitator: Click mouse to reveal animation. Second, I see that one girl was treated for a condition not listed on the table. The condition should be identified in the blank space. Clinic management will need to know if she was treated for Vitamin A deficiency or some other issue. Note to facilitator: Click mouse to reveal animation. Last, we see that there is an error is addition. 6+5=11, not 20; this also will result in an overreporting of the specific condition. Paying attention to issues of data quality at the time data are collected is critical for accurate data-informed decision making.

Data Demand & Use Now let’s take a minute and step back to look at a conceptual framework that will help us to understand some of the underlying concepts that affect how we access and use data. The cycle above illustrates that the use of data is linked to demand for quality information. A demand for information leads us to collect and analyze data related to the specific demand or data need. Once data are collected, the data are made available in a format that is useful to the decision maker. This information is then used to make decisions and eventually contributes to the improvement in health systems. The cycle supports the assumption that the more positive experiences a decision maker has in using information to support a decision, the stronger the commitment will be to improving data collection systems and continuing to use the information they generate. 17

Demand for Data Demand – the value stakeholders place on data, whether or not the data are used Demand exists if Specific questions need to be answered and data are sought to answer them A decision needs to be made and data are sought to inform it The term data demand is related to the value stakeholders place on data, whether or not the data are used. Demand exists if: Specific questions need to be answered and data are sought to answer them And/or if a decision needs to be made and data are sought to inform it 18

Data Use Use refers to the decision-making process A decision maker uses information if he/she Is explicitly aware of the decision to be made or question to be answered Considers relevant information in making the decision, even if the information is outweighed by other factors The term data use refers to the decision-making process. A decision maker uses information if he/she: Is aware of the decision to be made or question to be answered Considers relevant information in making the decision, even if the information is outweighed by other factors 19

What Are Decisions? Choices that lead to action All decisions are informed by questions All questions should be based on data When we talk about data use, we usually refer to the term ‘decisions’ – such as data-informed decision making. Let’s clarify what we mean by ‘decisions’ …. Choices that lead to action All decisions are informed by questions All questions should be based on data For example: Every day you need to make a decision about what to wear out of the house. To make this decision, you may ask yourself some questions that will inform the decision: What is the temperature? Is it raining? What events do I have planned for the day? To answer these questions, you may consult the thermometer, the weather report on the radio, or your daily calendar. In this instance, you are informing a decision with data.

Group participation What are decisions you make in your daily life? What are examples of decisions made by others – that affect you in your daily life? Note to facilitator: Read each bullet and solicit the group to provide answers to the questions listed.

Decision Areas Program design Program management and improvement Strategic planning Advocacy and policy development There are four main types of decision areas in the health sector: Program design Management & improvement - Example – Determine if the program is meeting its objectives & if not develop new strategies to increase coverage Strategic Planning - Example – Identify geographic areas of highest need Advocacy & Policy Development - Example – Identifying and quantifying underserved populations As providers, the types of decisions we mostly deal with are the second type – program management and improvement.

Who makes decisions? Traditional decision makers Other stakeholders Program manager, policymaker, funder Other stakeholders Any person, group, or organization with a particular interest in a policy or program Providers, implementers, beneficiaries, civil society As we talk about decision making, let’s talk about who makes decisions. We are all familiar with the traditional decision maker – the program manager, policymaker, or funder. But there are other types of decision makers. They include important stakeholders – a person, group, or organization with a particular interest in a policy or program. These could include: providers, implementers, beneficiaries, or even civil society.

Role of the provider in decision making Data collection – filling out the forms Data compilation Data analysis, interpretation, & reporting Program monitoring Problem identification & solution development Best practice identification Communication Now that we have defined the provider as a decision maker – let’s discus your role in the decision-making process. The provider’s role is to: Note to facilitator: Read slide.

What Determines Data Demand & Use? ORGANIZATIONAL TECHNICAL BEHAVIORAL There are many factors that affect the decision-making process. Here you see the three main determinants of data use: Organizational determinants – these determinants relate to the organizational context that supports data collection, availability, and use. Examples are: the identified procedures and the roles and responsibilities of those who collect, analyze, disseminate, and use data. Technical determinants – refer to the technical aspects of the data collection process, as well as tools, such as the data collection processes, methods, and forms. Last, Behavioral determinants – refer to the behavior of individuals who produce and use data. This would cover their skills, attitudes, values, and motivation. It is important to consider what affects the data demand and use process so that we can identify barriers to data use and develop strategies to overcome them.

Data are often underutilized because of… Technical determinants Poor data quality Insufficient data analysis skills The determinants of data use frequently contribute to the underutilization of data. For example, let’s discuss some examples of what we mean by the categories of determinants of data use. Technical – Data are often underutilized because there may be: Poor data quality, so providers and program managers don’t trust the data to represent the current situation in their clinics Or there may be a lack of technical skills among providers and other data users, such as the ability to analyze data * Based on PRISM analytical framework (LaFond, Fields et al. 2005 The PRISM: An Analytical Framework for Understanding Performance of Health Information Systems in Developing Countries. MEASURE Evaluation).

Data are often underutilized because of… Organizational determinants Roles and responsibilities are unclear Inadequate support for data use by management Behavioral determinants Staff motivation Organizational determinants include: The fact that roles and responsibilities may be unclear. Providers, data clerks, and program managers may not be clear on whose job it is to compile clinic data, report the data, and use them in decision making. Also, low support for data use by facility managers can inhibit data use. Last, data can be underutilized because of individual behaviors. For instance: Staff motivation to collect quality data, analyze the data, and use them may be low – because they don’t understand how data can be useful.

What Determines Data Demand & Use? CULTURE ORGANIZATIONAL TECHNICAL BEHAVIORAL POLITICS SOCIETY In addition to the three determinants of decision making – organizational, technical, and behavioral – there are other factors that can negatively affect the decision-making process that often are beyond our control to affect. They include: political, cultural, and social factors. We need to remember that these factors exist and affect decision making so that we can develop strategies to overcome them. We will discuss some of these strategies in the following sessions.

Key Messages Data are needed to improve the delivery of services Providers play a critical role in data collection, data monitoring, problem identification, & solution development Decisions based on evidence lead to better health outcomes Decision making is affected by three major determinants We have now come to the end of Module 1 on the key concepts in data demand and use. The key messages of this module include: Note to facilitator: Read slide.

Group work 1 – Instructions Select a reporter. Read the case study and answer the following questions (20 min): What prompted the data use undertaking? What was the decision taken? What types of data were used to make the decision? What was the outcome of the decision? Report back (5 min per group). Now we will spend some time discussing data use concepts in small groups. Within your small groups, please select a reporter, then read the case study and answer the following questions: Note to facilitator: Read questions. You will have 20 minutes to read the case study and answer the questions, and then we will ask each group to report back in plenary.