Indicators and Calculating Coverage indicators. M&E Indicators For Malaria Programs.

Slides:



Advertisements
Similar presentations
Donald T. Simeon Caribbean Health Research Council
Advertisements

Celebrating Achievements
Goal 5: Improve maternal health Target 5b: Achieve, by 2015, universal access to reproductive health 5.3 Contraceptive prevalence rate 5.4 Adolescent birth.
Measuring access to diagnosis and treatment RBM-CMWG July 9, 2009 Richard Steketee, MACEPA-PATH RBM-MERG Co-Chair.
The state of the art: DHS and MICS
APPMG World Malaria Day Event, 2013 Invest in The Future: Defeat Malaria Kolawole Maxwell, Malaria Consortium Nigeria Country Director.
Monitoring and Evaluation: Indicators
Compendium of Indicators for Monitoring and Evaluating National Tuberculosis Programs Using the Compendium to Plan for Monitoring and Evaluation of NTPs.
Module 2: Data analysis key concepts. Module 2: Learning objectives  Understand the definition and purpose of data analysis  Define statistical and.
Learning Objectives By the end of the session, participants will be able to: Identify sources of data for calculating coverage indicators Estimate denominators.
DETERMINANTS OF DATA USE Session 2. Session Objectives  Explain the data-use conceptual framework  Highlight the determinants of data use  List potential.
Context of Decision Making
Compendium of Indicators for Monitoring and Evaluating National Tuberculosis Programs.
Indicators, Data Sources, and Data Quality for TB M&E
Unit 3: Sample Size, Sampling Methods, Duration and Frequency of Sampling #3-3-1.
Unit 4: Monitoring Data Quality For HIV Case Surveillance Systems #6-0-1.
Lecture 3: Data sources Health inequality monitoring: with a special focus on low- and middle-income countries.
1 Interpretation and use. 2 The walls inside are plastered with laboriously made graphs…
Evaluating impact of OVC programs: standardizing our methods Jenifer Chapman, PhD Futures Group/MEASURE Evaluation.
1 Influence of PBF Indicators on Health Coverage Kathy Kantengwa M.D, MPA; PBF advisor, MSH Montreux, November 2010 Rwanda IHSS Project.
Technical Approach to and Experiences from Strengthening National Monitoring and Evaluation System for Most Vulnerable Children Program in Tanzania Prisca.
District Health Information Systems
Indicators Dr Murali Krishna Public Health Foundation of India.
Antenatal care MDG 5, Target 5b, Indicator 5.5
1 What are Monitoring and Evaluation? How do we think about M&E in the context of the LAM Project?
Multiple Indicator Cluster Surveys Survey Design Workshop Sampling: Overview MICS Survey Design Workshop.
Unit 10. Monitoring and evaluation
Roadmap Progress Report 2011 Zambia SARN-RBM PARTNERS ANNUAL CONSULTATIVE MEETING, JULY 2011.
1 Data analysis. 2 Turning data into information.
Zambia National Malaria Indicator Survey (MIS) 2010 Welcome!! The Barn Motel Lusaka.
Primer on Monitoring and Evaluation. The 3 Pillars of Monitoring and Evaluation  Identifying the Performance Indicators  Collecting information using.
Indicators Regional Workshop on the
Indicators for ACSM.
Case management versus M&E in the context of OVC programs: What have we learned? Jenifer Chapman, PhD Futures Group/MEASURE Evaluation.
21/4/2008 Evaluation of control measures 1. 21/4/2008 Evaluation of control measures 2 Family and Community Medicine Department.
Data Triangulation Applications in Population and Health Programs- -The Global Experience.
PLACE Method Priorities for Local AIDS Control Efforts 1 1 MEASURE Evaluation, A Manual for implementing the PLACE Method.
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
Linking Data with Action Part 1: Seven Steps of Using Information for Decision Making.
Linking Data with Action Part 2: Understanding Data Discrepancies.
9-1 Copyright © 2016 McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Session 2: Developing a Comprehensive M&E Work Plan.
Introduction to Monitoring and Evaluation. Learning Objectives By the end of the session, participants will be able to: Define program components Define.
WHY IS THIS HAPPENING IN THE PROGRAM? Session 5 Options for Further Investigation & Information Flow.
Session 5: Selecting and Operationalizing Indicators.
Introduction to Group Work. Learning Objectives The goal of the group project is to provide workshop participants with an opportunity to further develop.
Definition of indicators Facilitators’ Workshop on District Health Performance Improvement Lilongwe, 25 th – 27 th March 2015.
2014 Kenya Demographic and Health Survey (KDHS) Key Indicators.
Unmet Need Exercise  Review the trends in CPR. What do you conclude about the performance of the FP program in each country?  Review trends in unmet.
Knowledge Practice & Coverage Survey (KPC) Overview & resources Moving from research objectives, questions, hypothesis to questionnaire design Day 3: Session.
Developing Program Indicators Measuring Results MEASURE Evaluation.
Compendium of Indicators for Monitoring and Evaluating National Tuberculosis Programs.
MEASURE Evaluation Data Quality Assurance Workshop Session 3 Introduction to Routine Data Quality Assessment.
Developing a Monitoring & Evaluation Plan MEASURE Evaluation.
Monitoring & Evaluation Capacity Strengthening Workshop WORKSHOP INTRODUCTION AND OVERVIEW.
iCCM Recommended Indicators
Data Quality Assurance Workshop
Data Quality Assurance
Session: 5 Using the RDQA tool for System Assessment
Community Health Information System in Action in SNNPR/Ethiopia
Fundamentals of Monitoring and Evaluation
Monitoring and Evaluation of HIV/AIDS Programs Workshop Overview
Monitoring and Evaluation Systems for NARS Organisations in Papua New Guinea Day 2. Session 6. Developing indicators.
Overview of the RHIS Rapid Assessment Tool
Session: 4 Using the RDQA tool for Data Verification
Community Health Information System in Action in SNNPR/Ethiopia
Assessment Training Session 9: Assessment Analysis
Session: 6 Understanding & Using the RDQA Tool Output
Measuring Data Quality
Willis Odek, PhD Chief of Party/Senior Technical Advisor,
Presentation transcript:

Indicators and Calculating Coverage indicators

M&E Indicators For Malaria Programs

M&E Indicators: Module Objectives At the end of the session, participants will be able to: 1.Critique indicators 2.Identify criteria for selection of sound indicators 3.Understand how indicators are linked to the frameworks covered in the Frameworks Module 4.Select indicators and complete an Indicator Reference Sheet

What is an Indicator? An Indicator is… –a variable –that measures –one aspect of a program/project or health outcome An appropriate set of indicators includes at least one indicator for each significant aspect of the program or project (i.e. at least one per box in an M&E framework)

Anatomy of an Indicator Metric  Proportion of Households with at Least One ITN* –Numerator: Number of households surveyed with at least one ITN. –Denominator: Total number of households surveyed. * An ITN is 1) a factory treated net that does not require any treatment, 2) a pretreated net obtained within the past 12 months, or 3) a net that has been soaked with insecticide within the past 12 months.

Common Indicator Metrics Counts –Number of providers trained –Number of ITNs distributed Calculations: percentages, rates, ratios –% of facilities with trained provider –Under 5 mortality rate, case fatality rate, annual blood examination rate (ABER) Index, composite measures –Quality index comprising the sum of scores on six quality outcome indicators –DALY –Wealth index Thresholds –Presence, absence –Pre-determined level or standard –Cut-off point

Characteristics of Good Indicators Valid: accurate measure of a behavior, practice or task Reliable: consistently measurable in the same way by different observers Precise: operationally defined in clear terms Measurable: quantifiable using available tools and methods Timely: provides a measurement at time intervals relevant and appropriate in terms of program goals and activities Programmatically important: linked to a public health impact or to achieving the objectives that are needed for impact

Characteristics of Good Indicators: Valid Accurate measure of a behavior, practice or task Indicator measures what it is supposed to measure –Direct measures –Indirect/Proxy measures –Straightforward interpretation: change in value signals a change in focal concept or behavior

Validity: Class Activity 1.Is parasitemia a valid measure of morbidity? 2.Is fever a valid measure for malaria? 3.Is parasite testing a valid measure for parasite prevalence? 4.Is the number of people reached by BCC campaigns a valid measure of malaria knowledge?

Characteristics of Good Indicators: Reliable Consistently measurable in the same way by different observers Types of measurement error –Sampling Error: over-representation of urban populations because access is easier –Non-Sampling Error: survey estimates of bed net use, due to response bias –Subjective Measurement: indicators that ask for personal judgment such as “quality,” “environment” and “progress”

Characteristics of Good Indicators: Precise Operationally defined in clear terms Activity: Develop definitions for: Effective treatment Population at-risk Suspected cases of malaria

Characteristics of Good Indicators: Measurable Quantifiable using available tools and methods Are the following indicators measurable? 1.Number of ITNs distributed 2.Compliance to antimalarial treatment 3.Anemia 4.Parasitemia

Characteristics of Good Indicators: Timely Provides a measurement over periods of time of interest with data available for all appropriate intervals Timeliness Considerations –Reporting schedules –Recall periods –Survey schedules –Length of time over which change can be detected

Characteristics of Good Indicators: Programmatically Important Linked to a public health impact or to achieving the objectives needed for impact  Are the following indicators programmatically important? Example 1: ITN distribution program –Indicator: # ITNs distributed in past quarter Example 2: Program to increase access to ACTs through community-based health workers –Indicator: Number of ACT sales points with antimalarial drugs

Factors to Consider When Selecting Indicators Link to framework Programmatic needs/information for decision making Resources (Time) External requirements (government, health partner, headquarters) Data availability Standardized indicators

Operationalizing Indicators Establish exactly how a given concept / behavior will be measured –Precise definition and metric –How the value will be reliably calculated –Anyone using the same data will arrive at exactly the same indicator value –Challenges Subjective judgment Local conditions Unclear yardsticks Skills of the users

Sources of Indicators: Using Pre-Defined Indicators What are some sources for pre-defined indicators? –Past years of the program –Related or similar programs –Lists of global or recommended indicators Roll Back Malaria. Guidelines for Core Population-based Indicators, 2009 Global Fund Indicator Guide Global Fund performance framework

Indicator Pyramid District or Facility Identify progress, problems, and challenges National/Sub-national Assess effectiveness of response Reflect goals/objectives of national/ sub-national response Number of Indicators Decreases Increases Global Compare countries Overview world-wide situation

Indicator Matrix Data SourceFrequencyLevelDecision points/comments Output Number of health personnel and community health care agents trained in case management Program Records QuarterlyFacilityDisaggregate by district Outcome Proportion of children under 5 years old who slept under an ITN the previous night Representative household survey (ex. DHS, MICS, MIS) Periodic (Every 1-5 years) NationalTo be used to determine where to target ITN distribution and BCC activities

Indicator Reference Sheet Compile detailed documentation for each indicator: Basic information Description Plans for data collection Plans for data analysis, reporting, and review Data quality issues Performance data table (baseline and targets)

Indicator Reference Sheet: Example Name of Indicator DESCRIPTION Rationale Definition of the Indicator:  Numerator:  Denominator: Measurement: Frequency: Interpretation: Data Source(s): Strengths: Limitations: THIS SHEET LAST UPDATED ON: 07/18/2011

Indicator Strengths & Limitations All indicators have limitations, even those commonly used: Household spraying, net impregnation: Recall bias this can result in considerable ‘heaping’ of dates Net Use: self-reporting bias, seasonality of survey may affect net use

Indicators: How they link to frameworks

INPUT Human and financial resources ITNs OUTPUT ITNs sold and distributed OUTCOME Use of ITNs IMPACT Prevalence of malaria Indicator: Number of ITNs sold and distributed Indicator: Proportion of household members who slept under an ITN the previous night Indicator: Prevalence of malaria parasite infection PROCESS Establish distribution points for ITNs Logic Model Indicators

Results Framework Indicators Number of malaria cases Proportion of households that received spraying through an IRS campaign within the last 12 months Proportion of women who received IPT during antenatal care visits during their last pregnancy Proportion of household members who slept under an ITN the previous night IR-1.2: Improved coverage of IPTs IR-1: Improved malaria prevention IR-1.3: IRS coverage increased IR-1.1: Access to and coverage by ITNs increased

Setting Indicator Targets: Useful Information Sources Past trends Client expectations Donor expectations Expert opinion Research findings What has been accomplished elsewhere International conventions

Common Pitfalls in Indicator Selection Indicators not linked to program activities Poorly defined indicators Indicators that do not currently exist and cannot realistically be collected Process indicators to measure outcomes & impacts Indicators that are not very sensitive to change Too many indicators

Pitfalls with Selecting Indicators Indicator not linked to program activities IR: Expanded access to malaria treatment services Activities: train providers in current clinical protocols Inappropriate Indicator: % of facilities with adequate conditions to provide care Better indicators: # of clinicians trained, % of facilities with a trained provider  The program is not aiming to affect facility conditions, only provider skills.

Pitfalls with Selecting Indicators Data needed for indicator not available Inappropriate Indicator: % of days per quarter that service delivery points have stock-out of drugs Data issue: Information on stock-outs may not be collected daily Better indicators: % of service delivery points that had a stock out of drugs at some time during the last quarter  If relying on routine data, indicator definition must depend on how data are collected

Pitfalls with Selecting Indicators Indicator does not accurately represent desired outcome IR: Access to effective treatment among children <5 years old with malaria Inappropriate Indicators: % of children <5 years old who received ACTs; % of people who received ACTs for malaria infection who are children<5 Better indicator: % of children <5 years old who were diagnosed with malaria in the past 2 weeks who received ACTs  What does it mean if inappropriate indicators increase? Decrease? Do they reflect the desired program effect?

Indicator systems -- How much is enough? Rule of thumb At least one or two indicators per key activity or result (ideally, from different data sources) At least one indicator for every core activity (e.g., ITN distribution, IRS, training, BCC) No more than 8-10 indicators per area of significant program focus Use a mix of data collection strategies/source

Choosing the right number of indicators

Good indicators: Provide information useful for program decision-making Are consistent with international standards and other reporting requirements, as appropriate Are defined in clear and unambiguous terms Are non-directional, “independent” Have values that are: –Easy to interpret and explain –Precise, valid and reliable measures –Comparable across relevant population groups, geography, other program factors, as needed

NOT EVERYTHING THAT CAN BE COUNTED COUNTS, AND NOT EVERYTHING THAT COUNTS CAN BE COUNTED. Albert Einstein

Summary: Guiding principles to selecting indicators Ensure that the indicators are linked to the program goals and are able to measure change Ensure that standard indicators are used to the extent possible Consider the cost and feasibility of data collection and analysis Keep the number of indicators to the minimum and include only those needed for program and management decisions or for reporting

Calculating and Interpreting Coverage Indicators For Malaria Programs

Learning Objectives By the end of the session, participants will be able to: Identify sources of data for calculating coverage indicators Estimate denominators for routine coverage estimates Calculate and interpret coverage indicators from routine data Use online resources for estimating coverage indicators Assess the quality of relevant data sources Reconcile coverage estimates from different data sources

ITN Coverage Indicators Proportion of households with at least one ITN/LLIN Proportion of population with access to an ITN within their household Proportion of households with at least one ITN for every two people Proportion of the population/children under 5 years old/pregnant women who slept under an ITN/LLIN the previous night

IPTp Coverage Indicator Proportion of women who received two, three and four or more doses of intermittent preventive treatment for malaria during their last pregnancy in the last two years

Diagnostics and Treatment Coverage Indicator Proportion of children under 5 years old with fever in the last 2 weeks who had a finger or heel stick Proportion receiving ACTs (or other first line treatment), among children under five years old with fever in the last two weeks who received any antimalarial drugs

IRS Coverage Indicators Proportion of Households which Received Spraying through an IRS Campaign within the Last 12 Months Coverage of vector control: Proportion of Households with at least one ITN/LLIN and/or sprayed by IRS in the last 12 months

Why Coverage Indicators Are Important Understand how effective program is See if one target group is reached more effectively than another Identify underserved areas/regions

Estimating Coverage From Routine Data

Indicators for Program: Numerators HMIS and routine reports give information on numerators Numerators: number of houses sprayed with IRS, number of LLIN distributed through antenatal care, number of women receiving at least two doses of SP during antenatal care Denominators: ?

Example: Importance of denominator IPTp Provided to –Town A= 200 women –Town B= 400 women –Town C= 600 women Number of pregnant women: –Town A= 10,000 –Town B= 30,000 –Town C= 60,000 NumeratorDenominator Question: Can we say that Town C has the highest coverage? Please justify your response. Answer: No. We need the denominator for each town Question: What will be the denominator? Response: Number of women that need IPTp who visited ANC clinics in each town

Indicators for program: Denominators Population that are targeted by given intervention –District population –Women of childbearing age –Pregnant women visiting ANC –Children under the age of five –Children under 5 years old who had a fever

How Do We Get Denominators? Population registers Censuses Population projections Population growth rate (r) Rate of natural increase = crude birth rate (CBR) minus the crude death rate (CDR) Net migration rate: inmigration - outmigrants per 1000 population Population growth = rate of natural increase + net migration rate

Estimating population size Where: –P (t) is the population size after t years –P (0) is the population size at the last census –r the annual population growth rate Example:  300,000 people at census  Growth rate = 3% (0.03),  What is the population after 10 years? 404,958 people Use the national statistics office project national and sub-national level Use UN population, World Bank estimates for national level Use the official figures and only make projections if they are not available

Defining Population at Risk A group of people who share a characteristic that causes each member to be susceptible to a particular event, such as people living in an endemic area who are exposed to malaria Mid- term population (Mid-year) Expresses the population at the middle of the year Person-time Estimate of the actual time-at-risk in years, months, or days that all persons contributed to the period/under a particular intervention. Only possible if individuals are follow-up

Jan 01 Mid-Year Population vs. Person-Years Dec 31 Individual 1 Individual 2 Individual 3 Individual 4 Individual 5 Mid year pop: 1+1+1=3 Person time 3/12=0.25 5/12=0.42 7/12= /12= /12= Person year: =3.08

Estimating Target Population A District has 10,000 inhabitants in 2009, and 3% are children under 1 year of age –What is the annual target population for ITN distribution for infants? –What is the monthly population for ITN distribution for infants? Answers:  Annual target population= 10,000 x 0.03 = 300  Monthly target population= 1,800/12 = 25

Challenges in Estimating Coverage from Routine Data Limited knowledge of target pop/denominators Low timeliness & completeness of reporting Poor data quality –Lack of written standard reporting procedures –No systematic supervision on data management Dual reporting systems (EPI, HMIS) Data from private sector not often included

Assessing Reliability of Routine Coverage Indicators Understand how denominators are derived Understand the process of collecting the information Look for inconsistencies and surprises Look for reliable data from other sources to use as a basis for comparison Cross-check

Estimating Coverage From Survey Data

Tools for Coverage Estimation Large-scale population-based surveys Malaria Indicator Surveys - MIS Demographic and Health Surveys- DHS Multiple Indicator Cluster Survey- MICS Post-campaign coverage Surveys Other local surveys Lot quality sample coverage surveys

Routine Data vs. Survey Data

Reconciling Coverage Estimates from Different Data Sources Age group & geographic scope Health cards versus recall Different sources for different purposes Not all coverage data can be compared in constructive way Differences in inclusion of private sector Selectivity

On-line Resource: STATcompiler Innovative online database tool Allows users to select numerous countries and hundreds of indicators to create customized tables that serve specific needs Accesses nearly all malaria and population and health indicators published in MIS/DHS final reports

STATcompiler

Challenges with Routine-based Coverage Advantages: Provides information on more timely basis Makes use of data routinely collected Can be used to detect and correct problems in service delivery Disadvantages: Denominator errors Poor quality reporting

Challenges with Survey-based Coverage Advantages Avoids problems with denominators Includes community based information Disadvantages Larger standard errors at sub-national levels Irregular and expensive Survey timing may affect coverage rates

Group Project Form country groups For your project: Identify 4-6 indicators based on your framework and define metrics Create an indicator matrix for your indicators If frameworks are not finished, continue working on frameworks For two indicators, complete indicator reference sheet

References Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project. Bertrand, Jane T. and Escudero Gabriela, Compendium of Indicators for Evaluating Reproductive Health Programs, vols. 1 and 2, Chapel Hill, N.C.: MEASURE Evaluation. Roll Back Malaria Guidelines for core population-based indicators. January MEASURE Evaluation: Calverton, MD. Tsui, Amy Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. Tsui, Amy Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. UNICEF State of the World’s Children. USAID/Tanzania Country Strategic Plan, WHO, The Evolution of Diarrhoeal and Acute Respiratory Disease Control at WHO: Achievement Research, Development and Implementation (WHO/CHS/CAH/99.12).

MEASURE Evaluation is a MEASURE program project funded by the U.S. Agency for International Development (USAID) Through Cooperative Agreement GHA-A and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, John Snow, INC., ICF Macro, Management Sciences for Health, and Tulane University. VISIT US ONLINE AT