The brass standard? The scope for RCTs in impact evaluation Jan Willem Gunning VU University Amsterdam University of Sussex, December 5, 2014

Assessing policy impact in development economics growing concern about endogeneity (growth regressions) scepticism about IV-estimates the example of medical research (testing drugs): RCTs But drugs testing is fundamentally different! extraordinary claims: development economis is RCTs

The randomista´s gold standard: only RCTs can identify causal effects don’t waste time on questions not suitable for RCTs RCT results have (unlimited) external validity Abhijit Banerjee en Esther Duflo

RCTs the gold standard? “ experiments have no special ability to produce more credible knowledge than other methods” Angus Deaton, JEL, 2010

Ignore the big questions? “Instead of discussing how best to fight diarrhea or dengue, many of the most vocal experts tend to be fixated on the “big questions”: What is the ultimate cause of poverty? How much faith should we place in free markets? Is democracy good for the poor? Does foreign aid have a role to play? And so on.” Banerjee en Duflo, Poor Economics, 2011

RCTs under fire.. not suitable for important questions Rodrik (2008), Ravallion (2012) doubts about external validity Banerjee and He (2008) vs. Banerjee and Duflo (2011) RCT may affect the control group Deaton (2010) upscaling changes the intervention Pritchett and Sandefur (2013), Bold et al. (2013) RCT misses selection on the gain Elbers and Gunning (2013)

Who benefits? selection on the gain assignment and treatment effect correlated program officer bases assignment in part on private information on treatment effect examples: NGO drinking water and sanitation program: local staff select program villages credit program: loan officer selects clients

RCTs in trouble..

Can RCTs deal with this? no, the problem is fundamental RCT (randomisation over beneficiaries) does not mimic the actual assignment process hence the RCT gives a correct answer to an irrelevant question randomising instead over program officers: loss of statistical power internal validity undermined: characteristics of program officers correlated with controls rehabilitation for observational data? such data became suspect because of endogeneity concerns.. but in this case they can do the trick.. if data from a reprentative sample: external validity assured

Estimating the total program effect (TPE) regression with observational data

Does it matter? example: impact evaluation of a health insurance program in Vietnam accounting for treatment heterogeneity doubles the health effect simple test: joint significance of the additional regressors in the TPE regression

Conclusion in development programs assignment often neither universal nor random, but correlated with treatment effects IV-regressions and RCTs then both wrong the impact evaluation literature focuses on an irrelevant parameter: Eβ the relevant parameter EβP can be estimated using observational rather than experimental data

Complication: P and X correlated examples: educational policies induce changes in parents’ contribution (Dercon); local governments reduce activities in response to central government programs (Deaton) unidirectional P caused by X: no adjustment unidirectional X caused by P: estimate TPE without the relevant X terms – this gives direct and indirect effect of P bidirectional case: if effect of P on X can be identified then indirect effect can be estimated and added to the direct effect 13