povertyactionlab.org Managing Threats to Evaluations and Data Analysis Michael Kremer Harvard University

2 Outline Background: Worms and the Primary School Deworming Project Threats to Estimating the Impact of Deworming –Externalities –Compliance Deworming Treatment Effects on Health, Nutrition, and School Participation Cost Effectiveness Conclusion

3 Background on Worms Worms infect ¼ of people worldwide: Hookworm 1.3 billion Roundworm 1.3 billion Whipworm 900 million Schistosomiasis 200 million Spread through poor hygiene and sanitation Prevalent among school-age children in developing countries

4 Impacts of Worms Health –listlessness, diarrhea, abdominal pain, anemia, malnutrition –Light infections often asymptomatic Education: –Seen as a key issue in whether deworming should be a priority in poor countries –Hypothesized to negatively affect cognitive ability and academic performance –Little previous evidence on impacts of deworming on school attendance

5 Prevention and Treatment Treatment: –Low-cost single-dose oral therapies (50 cents) –Every 6 months –Virtually no side effects. –Girls over 12 typically excluded Although there is effective treatment… –Worms often seen as ubiquitous, non-life threatening –Rapid re-infection rates implies treatment must be regular Likely substantial externalities to treatment

6 Research Question What are the impacts of deworming on health and education? A few randomized studies, but overall evidence on impact of deworming on cognition is inconclusive. Dickson et al., 2000 conclude that they cannot recommend massive school-based programs.

7 Previous Studies Previous studies randomize deworming treatment within schools –Suppose that deworming prevents the transmission of disease, what problems does this create? –If external benefit of 10% less infection after 6 months, how much underestimate benefit? –Suppose externalities are local? How can we measure total impact?

8 Externalities: Other contexts Programs may create spillover effects on untreated individuals –Physical spillovers De-worming program: disease reduction externalities –Prices Vermeersch and Kremer (2004): school meals provision led other schools to reduce school fees –Learning and imitation effects Duflo and Saez (2004)

9 Meauring Treatment Effect with Local Externalities Form two villages Randomly assign $1 in economic development assistance within village 25 cent sharing What is measured treatment effect?

10 Measuring Externalities If we expect externalities that are all within school: –Randomization at the level of the school allows for estimation of the overall effect of deworming on a school

11 Measuring Externalities If we expect externalities across schools or outside of schools –Need to use random variation in density of treatment

12 Kenyan Deworming Program Primary School Deworming Project (PSDP) –75 primary schools and 30,000 children –Single region of Kenya (Busia) Randomize deworming across schools: –Schools stratified by geographic zone and other assistance –Listed alphabetically –Every 3rd school assigned to groups 1,2 & 3 Group 1: treatment starting in 1998 and 1999 Group 2: treatment starting in 1999 Group 3: treatment starting in 2001 –Balanced Control and Treatment schools

13 Treatment Program Education on worm prevention, hygiene and sanitation. Treatment delivered by Kenyan MoH officials and ICS officers: –Deworming drugs at 6-month intervals –All schools with geohelminth prevalence over 50% mass treated with albendazole. –All schools with schistosomiasis prevalence over 30% mass treated with praziquantel. –Treatment schools which did not meet these prevalence requirements, infected pupils were treated individually. Not everyone treated in treatment schools –Girls 13 years and older not treated –Consent procedures in treatment schools 1998: ICS obtained community consent 1999: Kenyan MoH required signed parental consent to receive treatment

14 Estimating Impacts of the Program Randomization implies program participation not correlated in expectation with individual characteristics or infection status. Controls include school average 1996 exam results, average prevalence in zone, indicators for school involvement in other NGO assistance, and grade cohort dummies. School-level random effect

15 Health impact after 1st round of treatment: Prevalence of worms significantly lower in Group 1 27% versus 52% moderate-to-heavy infection Fewer Group 1 report having been sick Group 1 students have better height- for-age Worm prevention techniques education failed

16 Health Externalities (Two types: Within-school and between school) Localized, within school externalities –Rates of moderate-to-heavy infections substantially lower in Group 1 pupils who did not receive medical treatment than among comparison students in Group 2 schools –Incoming pre-school students had only half infection rate –Moderate-to-heavy infection rates 10% lower for girls>12

17 Deworming treatment effects on health: Moderate-to-Heavy Helminth Infection Rates

18 Cross-school externalities –Can you look at number of treatment schools nearby? –What is potential problem? –What need to control for?

19 Y ij is health outcome of individual i in village j, T i is the indicator variable for individual assignment to the program, X ij is a vector of village and individual characteristics, N it is the total number of individuals at distance d from i is the number of individuals at distance d from i randomly assigned to the program Empirical Specification

20 Empirical Specification Treatment externality: In words: Change in outcome with each additional treated person living d distance away Average effect of de-worming on treatment group in year 1: Average effect of de-worming on treatment group in year 2:

21 Results, controlling for density –Infection rates 26 percentage points lower per 1000 pupils in Group 1 schools within 3 km –Infection rates 14 percentage points lower per 1000 pupils between 3-6 km away

22 Compliance: Treatment Some students in treatment schools not treated 1998: 78% of pupils assigned to receive treatment received at least some treatment 1999: around 72% Absence from school the major cause of non- compliance 19% of girls over 13 received treatment in 1998 –Confusion over age –Nurses Some students in comparison schools treated 5% received treatment outside of program

23 Compliance: Transfers

24 Potential Comparisons What if drop non-compliers from analysis? What if analyze based on actual treatment? What do you measure if compare based on assigned treatment?

25 Intention to Treat (ITT) Y(T) = The average number of days children in a deworming school came to school Y(C) = The average number of days children in comparison schools came Y(T)-Y(C) = the difference between the two is the “intention to treat” effect of the program. This is the difference between the people you intended to treat and those who you did not intend to treat.

26 Intention to treat: “What happened to the average child who is in a treated school in this population?” What does “intention to treat” measure? Is this the right number to look for? Remember: In the deworming case many children in treatment schools were not treated and many children in the control treatment were. Intention to treat (ITT)

27 When ITT is useful?  May relate more to actual programs  For example, we may not be interested in the medical effect of deworming treatment, but what would happen under an actual deworming program.  If students often miss school and therefore don't get the deworming medicine, the intention to treat estimate may actually be most relevant.

28 What ITT does not measure Why might ITT not give good sense of what would happen with real program? Why might take up be lower? Higher? Transfers?

29 What if you wanted to know effect of the medicine on the children who took the medicine?”

30 From ITT to Effect of Treatment of Treated (ITT) Initial assignment an instrumental variable for treatment –Correlated with actual treatment, uncorrelated with other factors affecting outcomes

31 “Treatment on the Treated” The effect of the treatment on those who got the treatment: Suppose children who got the treatment attended school A days, irrespective of whether they were in a treatment or a control school Suppose children who got no treatment attended school B days, again in both kinds of schools

32 “Treatment on the Treated” Then… Y(T)=A*Prob[treated|T]+B(1-Prob[treated|T]) Y(C)=A*Prob[treated|C]+B(1-Prob[treated|C]) A-B= (Y(T)-Y(C)) (Prob[treated|T] – Prob[treated|C]) =The “treatment on the treated” effect.

33 How do we calculate this? We need to observe the number of people getting treated Prob[treated|T] – Prob[treated|C]. This may not always be easy. Why? We don’t always need the entire sample: –It may be enough to get it from some representative sub-sample. General technique for when compliance is not perfect, for example, in an encouragement design.

34 Impacts on Education: Treatment associated with dramatic gains in school participation, large externalities Baseline absence rate (30%) reduced by ¼ Total effect: Increase in school participation of ~0.14 years per child Cost-effectiveness: $3.50/additional year of schooling

35 Figure 2: School participation rate May 1998 to November 1999 for girls under 13 and all boys, difference between Group 1 and Group 3 (diamonds), and difference between Group 2 and Group 3 (squares)

36 Estimating the Overall Effect Comparison schools: 1.5 percentage point increase in school participation, three Group 2 & 3 pupils for every treated child in Group 1. Treatment schools: 7 percentage point increase in school participation for all children, one untreated child for every 2 treated children. (.015 * 3) + (.07 *.5) + (.07 * 1) =.15 years For each child treated, school participation increased by.15 years.

37 Cost Effectiveness and Welfare Analysis Health cost effectiveness Educational cost effectiveness Human capital investment Externalities

38 Health Cost Effectiveness One of many worthy health interventions competing for scarce resources—annual government per capita expenditure on health is $5. DALY’s PCD cost per pupil treated is $0.49

39 Health Cost Effectiveness Treatment of schistosomiasis is very cost effective, but only meets the strictest standard when take into account the externalities present. $12 per DALY saved The externality benefits account for 61.1% of the DALY reduction Treatment of geohelminths would not meet standard cost –Effectiveness criteria in the poorest countries based on health impact alone: $155 per DALY saved.

40 Educational Cost Effectiveness What is the cost of an additional year of school participation? School participation increased by.15 years per treated child Cost is $.49 per child in a large-scale program. Costs only $3.27 per additional year of school participation. Much cheaper than other ways of increasing schooling.

41 Human Capital Investment What is the rate of return to deworming in future earnings? Labor market benefits alone may far outweigh cost. Excludes health and nutritional gains, thus understating benefits. Knight and Sabot (1987) estimate the avg. labor market return to an additional year of primary schooling in Kenya is 17%. Years of schooling alone account for approximately 40% of the total return to education controlling for scores. If deworming increases participation but has minimal cognitive gains, the direct return to an additional year of primary education would be 7%. Output per worker is $570 Discount rate of 10% Assume opportunity cost is half adult wage Deworming increases net present value of wages by more than $20 per treated child at a cost of only $0.49.

42 Externalities What subsidy would lead people to fully internalize the externalities present? Total net externality gain per child treated is at least $11.70, when cost of treating that child was $0.49. Government should heavily subsidize. Hard to know the optimal level of a subsidy without an epidemiological model.

43 Deworming program: follow-up “Sustainable” financing of deworming? –Health education, commitment ineffective –cost recovery leads to much less take up –Suggests a need for subsidies for de-worming Networks and social learning –Children with (randomly) more social links to early treatment schools are themselves significantly less likely to take deworming drugs

44 References Edward Miguel and Michael Kremer (2004) “Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities,” Econometrica, 72(1): –Kremer and Miguel (2004) “The Illusion of Sustainability,” mimeo. –Miguel and Kremer (2004) “Networks and Social Learning about Health,” mimeo.