1. Expanding the body of evidence Marc Shotland Training Director and Senior Research Manager

2. J-PAL What is…. 2

3. What is J-PAL? Research center within the Economics Department of MIT A network of 51 affiliated professors 3

4. A Research Network 4 Esther Duflo Abhijit Banerjee Rachel Glennerster

5. J-PAL Offices 5

6. J-PAL Mission Evaluate: Conduct Randomized evaluations to learn how to improve the lives of the poor Build Capacity: Train others to conduct randomized evaluations Impact Policy: Advocate for evidence-based policy 6

7. EVALUATION 7

8. J-PAL Projects J-PAL has 250 completed and ongoing evaluations in 43 countries 48 complete evaluations in Health

9. EVALUATE Why do we….

10. Puzzles in delivery and take-up Vaccinations or treatments exist People aren’t getting vaccinated or treated Why? – Supply chain is breaking down – “Demand chain” is breaking down

11. Example: Immunization in rural Rajasthan Vaccinations were free Provided by public health facilities As of baseline, only 2% of children received the full course of immunizations Health care center is “always” closed

12. Gov/NGO procures sufficient vaccines & distributes to facilities Facility has appropriate (non-expired) dose in stock Immunizations provided to patient (at minimal cost) Staff provide patients w/ guidance on requirements to complete full course Patient receives full course of immunizations Start with a mapping of successful program from supply- side (usually from mixed methods research)… Generating hypotheses to test with RCTs Functional supply chain

13. And then demand-side… Functional demand chain Patient willing/able to pay for vaccination Patient promptly visits quality provider Patient seeks immunizations Patient accepts vaccination & completes full course Patient receives full course of immunizations

14. Opportunity cost high due to distance & queue for service User fees high Supply chain irregular & insufficient Poor inventory management Then consider how this might break down… Gov/NGO procures sufficient vaccines & distributes to facilities Facility has appropriate (non-expired) dose in stock Immunizations provided to patient at minimal cost Staff provide patients w/ guidance on requirements to complete full course Patient receives full course of immunizations

15. Unaware of vaccination options Unaware of why they need immunizations Don’t believe that immunizations are effective Quality treatment is uncertain/ inaccessible due to: geographic proximity stock outs poor provider attendance, quality & training Vaccination is unaffordable Unable to afford full course Do not complete full course due to: procrastination inconvenience forgetfulness Breakdown in Demand Chain Patient willing/able to pay for vaccination Patient promptly visits quality provider Patient seeks immunizations Patient accepts vaccination & completes full course Patient receives full course of immunizations

16. Immunization in rural Rajasthan Uncertain Supply – Seva Mandir introduces “immunization camps” – Open from 11am-2pm on a fixed date Demand (incentives) – Offered 1kg dal per immunization administered – a set of thalis (metal meal plates) upon completion of a child’s full immunization course

17. Immunization Supply and Demand 17 Full Course Immunizations

18. Why rigorous impact evaluation? 1.Determine the impact of your policy 2.Ensure your impact estimate is valid 18 Poor inventory management Gov/NGO procures sufficient vaccines & distributes to facilities Facility has appropriate (non-expired) dose in stock Immunizations provided to patient at minimal cost Staff provide patients w/ guidance on requirements to complete full course Patient receives full course of immunizations

19. What is Impact? 19 Time Primary Outcome Counterfactual Intervention

20. What is Impact? 20 Time Primary Outcome Counterfactual Intervention

21. What is Impact? 21 Time Primary Outcome Counterfactual Intervention

22. How should we measure impact? 22 Impact is defined as a comparison between: 1.the outcome some time after the program has been introduced --and-- 2.the outcome at that same point in time had the program not been introduced (the “counterfactual”) Problem: Counterfactual cannot be observed Solution: We need to “mimic” or construct the counterfactual

23. Why Experiment? 23

24. Why Experiment? 24

25. Randomization allows meaningful comparisons Determine different treatment group randomly By design, program beneficiaries are not more motivated, richer, more educated than others Gives nice clean result everyone can understand 25

26. J-PAL’s Mission 26