1. Randomized Control Trials for Agriculture Pace Phillips, Innovations for Poverty Action www.poverty-action.org

2. What we do IPA Ghana IPA’s Mission and Approach Why evaluate? How to evaluate impact? Examples of evaluations Outline

3. What does IPA do? IPA was created in 2002 to determine what works and what does not work in poverty alleviation through rigorous evaluation. Estimate magnitude of effects – Most cost-effective solutions; can be surprising Learn how to improve programs through testing operational questions Determine where to spend limited resources

4. IPA Ghana In Ghana since 2005 16 projects nationwide Permanent offices in Accra and Tamale Current Agric Partners: – Ghana Insurers Association – GIZ – IFPRI – Presbyterian Agricultural Service – MoFA – SARI – ISSER Trainings /Conferences

5. Research Areas JPAL-CMF Financial services How to encourage people to save more? Health How to encourage people to use health products such as bed nets? Agriculture What constrains farmers to make more productive investments? Education How can we increase learning outcomes in schools? Governance and Democracy Does community-based monitoring affect service providers’ incentives? SME and entrepreneur- ship How to identify successful entrepreneurs?

6. Surprisingly little hard evidence on what works Can do more with given budget with better evidence If people knew money was going to programs that worked, could help increase pot for anti-poverty programs Instead of asking “do aid/development programs work?” should be asking: – Which work best, why and when? – How can we scale up what works? Why focus on impact evaluation? 6

7. Process and impact evaluation InputsOutputsOutcomes Process evaluations Monitoring In the control of the organization Impact Impact evaluation: how, and how much, does the organization affect the welfare of beneficiaries (and non beneficiaries) Monitoring/Process evaluation: tracking development and outputs of a program as compared to stated objectives, targets and timelines

8. In Northern Ghana, a development organization undertakes a program to promote fertilizer use to improve yields. How do we know if the program was successful? How do you evaluate a program?

9. Impact: What is it? Time Crop Yields Impact Counterfactual Intervention

10. What is the impact? Time Crop Yields Impact Counterfactual Intervention

11. What is the impact? Time Crop Yields Impact Counterfactual Intervention

12. Counterfactual: What would have happened in the absence of the program? Problem: Counterfactual is not observable  the key goal of all impact evaluation methods is to construct or “mimic” the counterfactual. Solution: Counterfactual is often constructed by selecting a group not affected by the program. Counterfactual

13. Without Random Selection, the control group will have intrinsic differences that can bias the study. Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment Any difference that subsequently arises between them can be attributed to the program rather than to other factors If properly designed and conducted, randomized experiments provide the most credible method to estimate the impact of a program Why randomize? 13

14. Randomly sample from area of interest Random sampling and random assignment

15. Randomly sample from area of interest Randomly assign to treatment and control Random sampling and random assignment Randomly sample from both treatment and control

16. Which method we use matters! MethodImpact Estimate (1) Pre-post26.42* (2) Simple Difference-5.05* (3) Difference-in-Difference6.82* (4) Regression1.92 (5)Randomized Experiment5.87* *: Statistically significant at the 5% level

17. Underinvestment in Agriculture A study in Northern Ghana to understand if farmers do not invest in their farms because either capital constraint or risk aversion. Year 2: 1377 farmers in 74 villages Year 2 Randomization Insurance: 729 farmersBoth: 108 farmersCapital: 363 farmers Rainfall index insurance offered at varying prices GHS 350 provided Control: 177 farmers Rainfall index insurance offered at varying prices, AND GHS 350 provided

18. Results Capital only farmers – Used more inorganic fertilizer only Capital and insurance farmers – Increased farm investment by 20 percent Insurance farmers – Increased total farm expenditure by 13 percent Increased inorganic fertilizer use by 25 percent Increased cultivation area by 8 percent Increased expenditures on land preparation by 12 percent Increased total labor use on plots by 13 percent Increased harvest output by 8 percent Increase in investment but no increase in profitability

19. Demand for Agriculture Insurance Uptake

20. A Well Timed Nudge A study in Western Kenya tested whether farmers purchases of fertilizer was dependent on the time of the year it was offered. Baseline: 1,125 Farmers in Suri Kenya Randomization Fertilizer offered AFTER HARVEST with free delivery 50% Subsidy Fertilizer offered just BEFORE GROWING SEASON with free delivery CHOICE of when they want the fertilizer offer with free delivery. Control – No Offer Fertilizer offered just BEFORE GROWING SEASON with free delivery

21. Results Farmers had high demand and ability to purchase fertilizer in advance. Impact of the After Harvest offers were comparable to that of a 50% subsidy at fertilizer application time. Increase in Fertilizer Use Control Group 28% After Harvest 39% Choice of Time 47% Before Growing Season 33% 50% Subsidy Before Growing Season 41% Once the program stopped, fertilizer usage went back to what it had been.

22. Thank You www.poverty-action.org

23. IPA Ghana Why do farmers under-invest in farms? What is the impact of financial savings programs in schools? What is the willingness to pay for clean water? What is the impact of remedial education programs, smaller class size, teacher training and tracking on literacy and numeracy? What is the impact of health insurance education on enrollment and reenrollment in NHIS? Are mobile reminders an effective way to improve adherence to ACT regimens?

24. Problems Selecting a Control Group Difference between Groups Intrinsic DifferencesImpact of the program ObservablesNon Observables Selection Bias Without Random Selection, the control group will have intrinsic differences that can bias the study.

25. Our approach InnovateEvaluateReplicate Communi- cate Scale - Understand market failures -Develop innovative solutions to poverty - Use frontier knowledge from economics, and psychology Randomized Controlled Trials -Impact evaluations - Comparing variations of an intervention - Experiment with product designs Replicate evaluations in various settings to : - Generalize research findings - Tell practitioners what works (and not), when Effectively communicate to practitioners: -Conferences -Workshops with policy makers and practitioners -Policy memos and focus notes Facilitate scale- up of effective solutions : - Active policy outreach -Practitioners’ toolkits - Hands-on technical assistance We generate insights on what works and what does not through randomized evaluations, and ensure that those findings will be useful to, and used by practitioners and policy makers