1. Improving Lives through Impact Evaluation Dr. Jyotsna (Jo) Puri Head of Evaluation Deputy Executive Director, 3ie Presentation at the IEO Inauguration 26 th of February 2014

2. Outline State of evidence Quality of evidence What role for IEO?

3. STATE OF EVIDENCE

4. Global Evidence: Where are we?

5. Where are most IEs happening

6. In what sectors are IEs happening Large number of IEs: Health Education Social Protection Agriculture Missing sectors: Economic Policy Private Sector Development Water and sanitation Energy & Environment Transportation and Urban development

7. Sectoral distribution of 3ie funded studies in India Pattern of missing evidence is similar to the global database. Missing evidence: Governance and delivery (PDS!) Financial and rural development Environment

8. INFLUENCING POLICY

9. Statistical Meta-Analytical Reviews - An Example Source: David B. Wilson, 2006, A systematic review of drug court effects on recidivism Systematic reviews (& meta-analyses) can help understand build consolidate this evidence. Leapfrog learning!

10. Systematic Reviews

11. Regional distribution of 3ie funded studies in India Several states (Chhatisgarh, Punjab, Uttar Pradesh) North-east ignored completely. India-wide impact evaluations? Systematic reviews?

12. QUALITY OF EVIDENCE

13. Criteria for quality Internal validity – Process monitoring – Theory of change – Identification design – Power analysis – Spillovers – Findings based on research External validity – Heterogeneity (sub-group analysis) Humanitarian assistance Since 2005 - $90 Billion spent. Only 39 impact evaluations Quality of impact evaluations: -29 HAD a theory of change. But -23 did not have any balance tests (bw comparison and treatment groups) -29 did not have any power analysis (for sample size) -All used experimental or quasi- experimental methods. -Only 5 discuss ethical concerns.

14. INGO and Govt. 11.2% RESEARCH ORGANIZATION (76%) NGO (14%) Evidence-based policy makers – where art thou?? Who demands evaluations?

15. WHAT CAN IEO DO?

16. 1. Data India quite good Geographic information systems?

17. 2. Process evaluations Implementation research and operational research…. What works! Use experimental methods to understand the science of delivery. Maldives Community health workers vs. Nurses vs. Doctors to achieve primary health care objectives. India: Business correspondents vs. electronic delivery

18. 3. Standards for evidence Overall framework – how can we move from IEs being ad hoc to them being part of the system? Internal validity standards Overall no impact on maternal mortality of community based interventions (power) Significant impact of community based interventions on neo-natal mortality.

19. 4. Cost effectiveness studies 5. Collating evidence Cost effectiveness Systematic reviews

20. Thank you. www.3ieimpact.org jpuri@3ieimpact.org 20