1. #ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 ARIANNA LEGOVINI DEVELOPMENT IMPACT EVALUATION IMPACT EVALUATION FOR REAL TIME POLICY MAKING

2. WHAT IS THE VALUE OF A POLICY? “measuring” HOW TO GET THE MOST OUT OF A POLICY? “reaching” 2

3. Policy objective Invest in what gets us results 3 Invest in the cause to get an effect

4. Evaluation objective Identify cause - effect 4 enable good policy decisions

5. 55 WHY IS UNDERSTANDING CAUSALITY SO IMPORTANT? I THINK I WILL STOP USING UMBRELLAS WHEN I USE MY UMBRELLA MY SHOES GET WET It is easy to confuse correlation with causation

6. 6 Invest in the cause to get results Mentorship interacts with quality of mentor to change attitudes and value for youth to act upon their life-skill training and change their behavior (Bushway and Reuter, 2002, 2007) Charismatic mentor + behavioral therapy Attitudes and values Vocational training EMPLOYMENT More motivated youth

7. Evaluation Narrow down causality by identifying a counterfactual and compare 7 WHAT HAPPENED WHAT WOULD HAVE HAPPENED TO

8. 8 What is counterfactual analysis? Compare same individual –with & without intervention –at the same point in time Missing data Compare statistically identical groups of individuals –with & without intervention –at the same point in time Comparable data

9. 9 Counterfactual criteria Treated & control groups Have identical initial average characteristics (observed and unobserved) So that, the only difference is the treatment Therefore the only reason for the difference in outcomes is due to the treatment

10. 10 USE SCIENTIFIC EXPERIMENTS TO GUIDE POLICY DECISIONS

11. 1111 Get the delivery right Subsidy for malaria treatment $350 million Sealing packages saved 16,600 under five + 2,200 adult lives in 5 years Problem: poor public sector distribution of medicines Solution: found system that reduced stockouts by 83% Source: Vledder, M.; Friedman, J; Yadav, P., and Sjoblom M. RANDOM ASSIGNMENT OF HEALTH DISTRICTS TO 3 DIFFERENT DISTRIBUTIO N SYSTEMS

12. 12 Price a CURE $6

13. 1313 Get people to do it Source: Goldstein, Thirumurthy, and Zivin HIV treatment $ 2.1 billion Reminders increase adherence to treatment by 35% Problem IE scalable solution RANDOM ASSIGNMENT OF DAILY AND WEEKLY REMINDERS TO HIV PATIENTS

14. 14 Price Faith’s LIFE $7

15. 15 TRANSFORMATIONAL (not marginal) IMPACT

16. 16 Public recognition DOUBLES performance [Ashraf et al., 2011]

17. What does dime do? 17 Transform development policy 1.Run experiments to inform decisions 2.Build agencies’ capacity to do this systematically 3.Draw lessons and share them face-to-face to global audiences

18. 175 IEs in 47 Countries Distribution of dime ’s IEs by GP and CCSA (number of IEs) 18

19. 1919 dime in action Inform policy design Guide mid- course corrections Inform adoption and scale-up Systematic use of evidence Train & apply Learn by doing Apply knowledge Capacity building IE Product IE DESIGN IE IMPLEMENTATION IE DISSEMINATION

20. Clear roadmap 20 Clearer roadmap WhatTo whomWhenWhere

21. 2121 UNDERSTAND THE THEORY OF CHANGE Inputs Financial and human resources Activities Actions to convert inputs into outputs Outputs Tangible products Intermediate outcomes Effective access Use Compliance Performance Service quality Final Outcomes Health outcomes Lives saved Productivity WHAT (PROGRAM)HOW TO (RESPONSE) & RESULTS

22. awareness disposition Take-up ADOPTION SKILLS WILLINGNESS motivation participation Accountability quality sustainability 22 BREAK IT DOWN

23. Policy questions in dime ’s portfolio (% of questions) 23

24. Better capacity 24 Better capacity Research team Full time field coordinator Training More and better data

25. dime results indicators 25

26. Workshops increase participants’knowledge (pre/post test)

27. Impact of Workshop on (External Double Blind) Technical Ratings of IE proposals 27

28. dime results indicators 28

29. More effort into implementation 29 Motivation to perform Goal: this is what we are trying to achieve Control: this is how we want to do it …and BTW somebody measures (and cares)

30. Better decisions 30

31. dime results indicators 31

32. IE increases disbursements IE helps projects stick to plans 32 Does IE help project implementation? 1300 World Bank projects - 100 impact evaluations (2005-2011) Legovini, Di Maro, Piza (2015) It does!

33. IE increases disbursements & closes implementation gap statistically significant at 1% level 33

34. AT WHAT COST? SOME SIMPLE COST-BENEFIT OF IMPACT EVALUATION 34

35. Is the benefit worth the cost? Zambia best supply chain –IE solution: 83% reduction in stock out 21-25% reduction in malaria mortality –Pilot: Benefit ($7.4M) / Cost ($3.8M) = 2 –Scaled up: Benefit ($60.7M) / Cost ($12.4M) = 4.9 Malawi commitment saving account –IE solution: 22% increase in farmer production value –Pilot: Benefit ($0.7M)/Cost ($0.5M) = 1.4 –If scaled: Benefit ($15.6M) / Cost ($1.7M) = 9.2 35 IE PAYS FOR ITSELF A PUBLIC GOOD WORTH FINANCING

36. dime & i2i IE has real impacts on developmental effectiveness and project delivery High return on investment Spillover effects i2i opportunity to increase scale and scope Open invitation to donors and governments 36 partners CRIME LAB

37. Human Development Education Health Social Protection i2i pillars and thematic areas 37 Shared Prosperity Finance, Private Sector and Jobs Agriculture Infrastructure Governance Public Sector Governance Justice Local Development Governance Public Sector Governance Justice Local Development Climate Change Energy Environment Natural Resource Management Agriculture Transport Fragility & Conflict Reintegration Governance Gender-based Violence Urban crime & violence Gender Human Capital Economic Opportunities Voice/Agency 5 123 64

38. Discussion Which development interventions are effective and worth scaling up? Which promising mechanisms are worth testing across different sectors? No one size fits all Importance of building evidence-based institutions 38

39. #ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 Thank You! facebook.com/ieKnow #impacteval blogs.worldbank.org/impactevaluations microdata.worldbank.org/index.php/catalog/ impact_evaluation http://dime.worldbank.org WEB