M&E in HNP Operations: Lessons Learned in South Asia Benjamin Loevinsohn and Aakanksha Pande October 26th, 2006

Review of M&E in South Asia HNP Operations A. Background & Methods B. Results C. Recommendations

Three Aspects of M&E Measuring the results of World Bank financed operations – fiduciary obligation Strengthening the capacity of governments to monitor the performance of their HNP sectors – the long-term solution Impact evaluation – rigorously testing whether innovations actually make a difference

Sampling 12 South Asia HNP operations were selected using stratified random sampling. Since India represents 60% of HNP operations, 7 of the 12 projects were from India Indian sample was also stratified to ensure a balance between disease specific projects and state health systems projects

Design of Study 12 projects/ 60 indicators Interview with TTLs PAD Aide Memoire PSR/ISR ICR Sources of Data AP BPL SR Reviewers Kappa Statistic

Review of M&E in South Asia HNP Operations A. Background & Methods B. Results C. Recommendations

Summary of Results Indicators could be better selected (too many & too focused on inputs) Data collection plans ok but could be improved Data collection plans rarely implemented Even when data is collected, not often analyzed. Even when analyzed, not often used Few controlled evaluations, even though many opportunities Little attention to building M&E capacity

Indicator Selection Indicators are usually defined in a measurable way – 80% of the time. Uncertainty about the selection of indicators, reasonable grounds for disagreement Task teams need to keep a lot of people happy Many indicators are “upstream” more than would make sense for attribution Many indicators are chosen for each project avg. = 26, range 5 to 47 Having more indicators leads to less data

Evidence of “Indicator Inflation”

Data Collection Plans 33% of indicators had 5 attributes: defined in a measurable way (80%) clear method of collection (90%) clear target (53%) clear schedule (53%) clear responsibility (73%) Individually these attributes were clearly important to people

Implementation of Data Collection Plans Characteristic % of indicators or projects Kappa p-value % of indicators with baseline data 39 0.50 0.0000 % of projects with 1st ISR had “satisfactory” baseline data 25 0.51 0.0002 % of indicators with follow-on data 47 0.32 % of indicators where data collection schedule as in PAD 18 0.37 % of projects where data collection plan implemented 0.48

Data Collection Baseline data rarely collected Overall data collection plans were not implemented as designed Inconsistent approach to data collection – sad examples But, good examples of data collection do exist

Data Analysis Even when there was data, evidence of analysis existed only half the time For only 27% of indicators was there any evidence of data being analyzed Even when there was analysis, there was evidence of action 27% of the time

M&E Capacity Analysis of M&E capacity appears in PADs a third of the time Little agreement among observers about presence of M&E capacity building plans (kappa = 0.18) Lack of clarity on plans, implied not specified Much of the effort focused on HMIS Capacity building on M&E seen as important by World Bank staff

Evaluation of Innovations Half of projects contained at least 1 innovation Innovations not clearly defined, sometime implicit Explicit mechanism for evaluating innovation in 8% of projects Only one project had a controlled design

% of Children under 5 who were moderately or severely malnourished - BINP

% of Children under 5 who were moderately or severely malnourished - BINP

Little Progress of M&E Over Time

Review of M&E in South Asia HNP Operations A. Background & Methods B. Results C. Recommendations

Recommendations Use of an M&E checklist during preparation and implementation Select a set of indicators that TTLs can choose from Provide technical assistance to task teams where needed Dedicate staff to implement M&E Build the M&E capacity of task teams through training

Recommendations Deal with all aspects of M&E: (i) M&E of the project; (ii) capacity building of Government; (iii) impact evaluation Clear & consistent messages from management about importance of M&E i.e. Results-Based Management

M&E Checklist (page 22 of Report) Meant to be a guide for task teams List of actions that should be considered during preparation and implementation Part of QER package Has been adapted for water supply, rural development, & education.

Life is short. The art is long. --Hippocrates