1. Randomized controlled trials and the evaluation of development programs Chris Elbers VU University and AIID 11 November 2015

2. Joint work with Jan Willem Gunning Ideas developed when evaluating Dutch development programs (commissioned work) Related work by White (2006), Elbers, Gunning and de Hoop (WD 2009), De Janvry, Finan and Sadoulet (REStat 2012)

3. RCTs under fire Great successes trigger criticism RCTs’ claim to Gold Standard status has been attacked more or less aggressively – External validity questioned – Black box approach not scientific – Cannot answer ‘big questions’ (e.g. on economic development) – “experiments have no special ability to produce more credible knowledge than other methods” (Deaton, 2010, JEL)

4. Practical considerations External validity not an issue if the goal is to evaluate a particular project RCTs are great for providing proof of concept But… Actual programs not always amenable to evaluation by RCT – Of course, the program could be changed… Salvage old-fashioned regression using observational (i.e. non-experimental) data?

5. Outline Internal validity or RCTs not automatic Programs vs projects What do we want to estimate when evaluating a program? – The total program effect Application to health insurance in Vietnam Conclusion

6. When internal validity of RCTs could fail Example: program implemented at arm’s length: – Program officers select (intended) participants based on information specific to them Evaluation of the program must follow this design – Direct random assignment of (intended) treatment to ultimate participants misses the effect of POs’ selection activity → internal validity violated – Randomization must be over POs … – … killing statistical power precision is of order of number of POs in sample

7. Regression alternative (simplest case) Take random sample from potential beneficiaries of program Observe (intended) treatment status T and outcome y Regress y on T – Regression coefficient on T is ATET (assuming absence of confounders) – ATET times treatment fraction is per capita effect of program (‘total program effect’) – Precision is of order of number of sampled individuals

8. Projects and programs RCTs best suited for evaluating simple interventions in homogeneous group of subjects with strong supervision of implementation Real-life interventions are messier – They are a change of already existing policies – They implement different policies simultaneously, with different intensity, involving officers with varying degrees of enthusiasm, … – Selective participation is part and parcel of a typical program Should we not also try to quantify the impact of such programs? Can we?

9. A regression approach for evaluation of programs

10. The total program effect

11. Formally:

12. Example: health insurance in Vietnam Using data from a study by Wagstaff and Pradhan (WB, 2005) Health insurance introduced in 1990s Wagstaff and Pradhan try to avoid bias from treatment heterogeneity by matching insured and uninsured households on the likelihood of being insured (propensity score matching) This technique not suitable for TPE – Sample with matched T/C individuals is no longer representative of population

13. Table 1: Data for the Vietnam Insurance Example Variable: change in (average)MeanStd. Dev MinMax Arm circumference (cm)1.1542.013-7.39.4 Height (cm)5.17511.35-49.5739.84 Body weight (kg)2.9836.544-27.7526.25 Health expenditure (‘000 Dong)1,0815,519-8808233,965 Total consumption expenditure (‘000 Dong)6,5138,009-22,988116,826 Insurance (binary at individual level)0.1700.26801 School attended-0.0170.683-3.53 Currently attending school (binary at individual level)0.0820.388-22 Gender0.0020.138-0.751 Age3.5228.299-48.4348.6 Farm dummy-0.0790.4211 Household size-0.2671.696-1811 The number of observations varies between 4299 and 4305. Source: authors’ calculations using the Vietnam Living Standard Surveys 1992- 3, 1997-8.

14. Estimation of TPE

15. Table 2: Total Program Effects Dependent variableNaïve program effect † (I) (s.e.) Total program effect †† (II) (s.e.) R-squared of underlying regressions Remarks III Arm circumference.022 (.029) 0.090*** (0.027) 0.220.23 Height-0.190 (0.154).095 ( 0.139) 0.340.36 Body weight0.167* (0.083) 0.384*** (0.074) 0.310.33 Health expenditure-28.08 (60.59) -52.79 (51.01) 0.030.04Total consumption included in controls Health expenditure55.41 (66.42) 64.32 (52.87) 0.00 Total consumption expenditure not included Total consumption expenditure 626.7*** (110.9) 888.8*** (105.7) 0.100.12Total consumption expenditure not included

16. Conclusions RCTs ill suited for evaluation of ‘programs’ Programs involve strategic participation by potential participants and implementers – Evaluation must take that into account Regression techniques combined with proper sampling can identify combined impact of program elements – under nontrivial assumptions Simplest approximation of treatment heterogeneity suggests extensive use of interactions