Expanding the body of evidence Marc Shotland Training Director and Senior Research Manager

J-PAL What is…. 2

What is J-PAL? Research center within the Economics Department of MIT A network of 51 affiliated professors 3

A Research Network 4 Esther Duflo Abhijit Banerjee Rachel Glennerster

J-PAL Offices 5

J-PAL Mission Evaluate: Conduct Randomized evaluations to learn how to improve the lives of the poor Build Capacity: Train others to conduct randomized evaluations Impact Policy: Advocate for evidence-based policy 6

EVALUATION 7

J-PAL Projects J-PAL has 250 completed and ongoing evaluations in 43 countries 48 complete evaluations in Health

EVALUATE Why do we….

Puzzles in delivery and take-up Vaccinations or treatments exist People aren’t getting vaccinated or treated Why? – Supply chain is breaking down – “Demand chain” is breaking down

Example: Immunization in rural Rajasthan Vaccinations were free Provided by public health facilities As of baseline, only 2% of children received the full course of immunizations Health care center is “always” closed

Gov/NGO procures sufficient vaccines & distributes to facilities Facility has appropriate (non-expired) dose in stock Immunizations provided to patient (at minimal cost) Staff provide patients w/ guidance on requirements to complete full course Patient receives full course of immunizations Start with a mapping of successful program from supply- side (usually from mixed methods research)… Generating hypotheses to test with RCTs Functional supply chain

And then demand-side… Functional demand chain Patient willing/able to pay for vaccination Patient promptly visits quality provider Patient seeks immunizations Patient accepts vaccination & completes full course Patient receives full course of immunizations

Opportunity cost high due to distance & queue for service User fees high Supply chain irregular & insufficient Poor inventory management Then consider how this might break down… Gov/NGO procures sufficient vaccines & distributes to facilities Facility has appropriate (non-expired) dose in stock Immunizations provided to patient at minimal cost Staff provide patients w/ guidance on requirements to complete full course Patient receives full course of immunizations

Unaware of vaccination options Unaware of why they need immunizations Don’t believe that immunizations are effective Quality treatment is uncertain/ inaccessible due to: geographic proximity stock outs poor provider attendance, quality & training Vaccination is unaffordable Unable to afford full course Do not complete full course due to: procrastination inconvenience forgetfulness Breakdown in Demand Chain Patient willing/able to pay for vaccination Patient promptly visits quality provider Patient seeks immunizations Patient accepts vaccination & completes full course Patient receives full course of immunizations

Immunization in rural Rajasthan Uncertain Supply – Seva Mandir introduces “immunization camps” – Open from 11am-2pm on a fixed date Demand (incentives) – Offered 1kg dal per immunization administered – a set of thalis (metal meal plates) upon completion of a child’s full immunization course

Immunization Supply and Demand 17 Full Course Immunizations

Why rigorous impact evaluation? 1.Determine the impact of your policy 2.Ensure your impact estimate is valid 18 Poor inventory management Gov/NGO procures sufficient vaccines & distributes to facilities Facility has appropriate (non-expired) dose in stock Immunizations provided to patient at minimal cost Staff provide patients w/ guidance on requirements to complete full course Patient receives full course of immunizations

What is Impact? 19 Time Primary Outcome Counterfactual Intervention

What is Impact? 20 Time Primary Outcome Counterfactual Intervention

What is Impact? 21 Time Primary Outcome Counterfactual Intervention

How should we measure impact? 22 Impact is defined as a comparison between: 1.the outcome some time after the program has been introduced --and-- 2.the outcome at that same point in time had the program not been introduced (the “counterfactual”) Problem: Counterfactual cannot be observed Solution: We need to “mimic” or construct the counterfactual

Why Experiment? 23

Why Experiment? 24

Randomization allows meaningful comparisons Determine different treatment group randomly By design, program beneficiaries are not more motivated, richer, more educated than others Gives nice clean result everyone can understand 25

J-PAL’s Mission 26