Selecting performance indicators and setting targets Joseph Naimoli The World Bank October 2008 Presented by Kate Gilroy Institute for International Programs, Johns Hopkins University Health Results-Based Innovation Trust Fund (HRBF)

Acknowledgements Jennifer Bryce, Institute for International Programs, JHSPH Susan Stout, Independent Consultant Rena Eichler, Broad Branch Associates Target Setting Guide (January 2004), Audit Commission, U.K. Government

Outline 1Results management in HRBF 2Performance indicators 3Targets 4Task for country teams

Results Management Section 1:

Results management The ‘art’ of results management is defining and achieving outcomes that are meaningful to both provider and client/consumer, are measurable in a credible way, and are used in decision making. --Susan Stout The ‘art’ of results management is defining and achieving outcomes that are meaningful to both provider and client/consumer, are measurable in a credible way, and are used in decision making. --Susan Stout

Outcomes of interest for RBF? Inputs Funding Planning & policies Harmonization & efficiency Inputs Funding Planning & policies Harmonization & efficiency Outputs Health services delivery Quality Behavioural Interventions & knowledge Outputs Health services delivery Quality Behavioural Interventions & knowledge Outcomes Service utilization and intervention coverage Behavior change Outcomes Service utilization and intervention coverage Behavior change Impact Mortality Morbidity Nutrition Impact Mortality Morbidity Nutrition Process Training & Capacity building Procurement and supply Guidelines IEC Community mobilization Process Training & Capacity building Procurement and supply Guidelines IEC Community mobilization Reduced inequity Source: A common evaluation framework for the scale up to the health MDGs. Institute for International Programs, The Johns Hopkins University. Common evaluation framework for the scale-up for the health MDGs Results

A step-wise approach to achieving effective coverage Source: A common evaluation framework for the scale up to the health MDGs. Institute for International Programs, The Johns Hopkins University.

PERFORMANCE INDICATORS Section 2:

Reflect project objectives and priorities Are “true” (valid) measures of progress toward achieving MDGs 4, 5 and 1b, i.e.: Accurately reflect changes in service quality, utilization, intervention coverage, or behavior change Are feasible for use in low-income settings Can be repeated at frequent intervals as triggers for releasing incentive payments Can be influenced by recipients’ actions (attributable) Can be replicated and confirmed by others (reliable) Measurements that: Performance indicators for RBF

Other considerations for RBF indicators Feasibility of measurement Country’s existing HMIS Availability of data at different levels, i.e., region, district, facility, community Straightforward Can be easily understood by key actors and stakeholders Stakeholder consensus on importance Iterative: start with a few key indicators…..

Performance indicators for RBF “Measure aspects of service delivery that are important, rather than seeking to make things that are easily measured important” --Audit commission, U.K. Government “Measure aspects of service delivery that are important, rather than seeking to make things that are easily measured important” --Audit commission, U.K. Government

Most reflect years of research and “learning by doing” Standards for definition and measurement are available Allows comparisons across RBF projects and with national and international results Already being collected in DHS/MICS and other country-level surveys Considering the use of standard MNCH consensus indicators makes sense for RBF

What are the standard indicators for MNC outcomes? 2008 Countdown indicators for proven high-impact interventions addressing maternal, newborn and child mortality and nutrition Definitions available at and at the workshop An ongoing technical review and field testing process to improve the measurement and reporting of Countdown coverage indicators is being established with support from the Gates Foundation.

How many indicators? Your task: To select a few (3-5) key indicators of utilization, coverage, or behavior change that are directly related to the aims of your RBF project. Your task: To select a few (3-5) key indicators of utilization, coverage, or behavior change that are directly related to the aims of your RBF project. Too many can be counter-productive: difficult to communicate and manage, and can induce “indicator overload and fatigue” Too few may focus attention and new resources on selected indicators at the expense of other important areas No perfect answer

TARGETS Section 3:

Targets: What are they? Commitments, expressed in quantitative terms, to achieve a certain level and/or quality of service within a well-defined time period

Targets: Examples Absolute Increase the total number of children under 1 immunized from 800 (baseline) to 1000 Proportional or percentage achievement Option 1: Increase immunization coverage of children under 1 by 25 percentage points (from 50% to 75%) Option 2: Increase immunization coverage of children under 1 by 50 percent (from 50% to 75%) Additional service provided $5 US for each additional child vaccinated

Targets: Absolute or relative? Absolute: Increase immunization coverage of children under 1 to 80% at all health facilities Relative to providers’ existing baseline: Increase immunization coverage of children under 1 by 30 percent in each facility FacilityBaseline coverage Relative target for improvement Absolute Target for facility 120%30%26% 240%30%52% 350%30%65%

Targets: Absolute or relative? Absolute: Increase immunization coverage of children under 1 to 80% at all health facilities Relative to providers’ existing baseline: Increase immunization coverage of children under 1 by 15 percentage points in each facility FacilityBaseline coverage Target for improvement Absolute Target for facility 120%+15pp35% 240%+15pp55% 350%+15pp65%

Targets: Guiding Principles Use targets to encourage improved performance and motivate people when there is a probability of success Targets should be realistic but challenging Too easy  complacency Too difficult  discouragement, demotivation

Be “SMART”: specific, measurable, attributable, realistic, and time-scaled Be honest and unambiguous Review them periodically (not set in stone) Get buy-in from all stakeholders Ensure targets at different levels are linked and define who is responsible for achievements RBF experience to date: multiple targets at multiple levels (central, provincial, district, facility, community) in accordance with cascading performance contracts Tips for setting targets

Are there any recent trends for this indicator? What has similar activity achieved elsewhere? Does the target represent value for money? (what will it cost to collect the data?) What issues are likely to affect performance towards the target? Are there any barriers to achieving the target? Tips for setting targets Consider the following questions for each target:

Larger increases are possible when starting from a low baseline; higher baselines often provide little room for growth or change Targets should be attainable within a contract period Targets are tied to performance payments Tips for setting targets Remember:

TASK FOR COUNTRY TEAMS Section 4:

1. Identify indicators most relevant to your country’s national priorities and objectives and the aims of your project; review indicator definitions 2. Review available data for each indicator in your country, using Countdown and other data sources 3. Assess each indicator relative to criteria presented here 4. Define priority indicators for relevant levels, the source of data, and how often it will be reported Select potential indicators

Define targets, taking into account the guiding principles and “tips” presented here Define targets For each indicator, at each level:

Prepare a plan IndicatorTargets Process for measurement Project objective 1: 1 2 Project objective 2: 3 4 Project objective 3 5

Prepare a plan: Example IndicatorTargets Process for measurement Project objective 1: 1% of children U1 fully vaccinated 80%Immunization records with random HH checks for validity 2 Project objective 2: 3 4 Project objective 3 5

DISCUSSION