1. M&E in HNP Operations: Lessons Learned in South Asia
Benjamin Loevinsohn and Aakanksha Pande
October 26th, 2006

2. Review of M&E in South Asia HNP Operations
A. Background &
Methods
B. Results
C. Recommendations

3. Three Aspects of M&E
Measuring the results of World Bank financed operations – fiduciary obligation
Strengthening the capacity of governments to monitor the performance of their HNP sectors – the long-term solution
Impact evaluation – rigorously testing whether innovations actually make a difference

4. Sampling
12 South Asia HNP operations were selected using stratified random sampling.
Since India represents 60% of HNP operations, 7 of the 12 projects were from India
Indian sample was also stratified to ensure a balance between disease specific projects and state health systems projects

5. Design of Study 12 projects/ 60 indicators Interview with TTLs PAD
Aide Memoire
PSR/ISR
ICR
Sources of Data AP
BPL SR
Reviewers
Kappa Statistic

6. Review of M&E in South Asia HNP Operations
A. Background &
Methods
B. Results
C. Recommendations

7. Summary of Results
Indicators could be better selected (too many & too focused on inputs)
Data collection plans ok but could be improved
Data collection plans rarely implemented
Even when data is collected, not often analyzed.
Even when analyzed, not often used
Few controlled evaluations, even though many opportunities
Little attention to building M&E capacity

8. Indicator Selection
Indicators are usually defined in a measurable way – 80% of the time.
Uncertainty about the selection of indicators, reasonable grounds for disagreement
Task teams need to keep a lot of people happy
Many indicators are “upstream” more than would make sense for attribution
Many indicators are chosen for each project avg. = 26, range 5 to 47
Having more indicators leads to less data

9. Evidence of “Indicator Inflation”

10. Data Collection Plans 33% of indicators had 5 attributes:
defined in a measurable way (80%)
clear method of collection (90%)
clear target (53%)
clear schedule (53%)
clear responsibility (73%)
Individually these attributes were clearly important to people

11. Implementation of Data Collection Plans
Characteristic
% of indicators or projects
Kappa
p-value
% of indicators with baseline data 39
0.50
0.0000
% of projects with 1st ISR had “satisfactory” baseline data 25
0.51
0.0002
% of indicators with follow-on data 47
0.32
% of indicators where data collection schedule as in PAD 18
0.37
% of projects where data collection plan implemented
0.48

12. Data Collection Baseline data rarely collected
Overall data collection plans were not implemented as designed
Inconsistent approach to data collection – sad examples
But, good examples of data collection do exist

13. Data Analysis
Even when there was data, evidence of analysis existed only half the time
For only 27% of indicators was there any evidence of data being analyzed
Even when there was analysis, there was evidence of action 27% of the time

14. M&E Capacity
Analysis of M&E capacity appears in PADs a third of the time
Little agreement among observers about presence of M&E capacity building plans (kappa = 0.18)
Lack of clarity on plans, implied not specified
Much of the effort focused on HMIS
Capacity building on M&E seen as important by World Bank staff

15. Evaluation of Innovations
Half of projects contained at least 1 innovation
Innovations not clearly defined, sometime implicit
Explicit mechanism for evaluating innovation in 8% of projects
Only one project had a controlled design

16. % of Children under 5 who were moderately or severely malnourished - BINP

17. % of Children under 5 who were moderately or severely malnourished - BINP

18. Little Progress of M&E Over Time

19. Review of M&E in South Asia HNP Operations
A. Background &
Methods
B. Results
C. Recommendations

20. Recommendations
Use of an M&E checklist during preparation and implementation
Select a set of indicators that TTLs can choose from
Provide technical assistance to task teams where needed
Dedicate staff to implement M&E
Build the M&E capacity of task teams through training

21. Recommendations
Deal with all aspects of M&E: (i) M&E of the project; (ii) capacity building of Government; (iii) impact evaluation
Clear & consistent messages from management about importance of M&E i.e. Results-Based Management

22. M&E Checklist (page 22 of Report)
Meant to be a guide for task teams
List of actions that should be considered during preparation and implementation
Part of QER package
Has been adapted for water supply, rural development, & education.

23. Life is short.
The art is long.
--Hippocrates