Randomized Control Trials Difficulties to Consider

Costs Cost of intervention itself often not difficult to justify Providing goods/services Only including promising possibilities New data are expensive Quality of evaluation dependent on quality of data More money spent on data is less money spent on providing the intervention to more people

Creating the Control Group Is it politically feasible to deny treatment to some people? How important is it to measure how well the intervention works? Issue of trade-offs Ethics less contested if: Budget constraints would have prevented everyone from receiving the intervention anyway Everyone eventually receives the intervention and the control group is only denied it initially (phased-in rollout)

Does Everyone Benefit? Necessary to deny control group intervention But don’t want to actively hurt them Can’t deceive Can’t make them worse off than they’d otherwise be Some sort of small gift/compensation typical – careful not to make this into a second treatment Must honor promises (phased-in rollout)

Internal Validity Was the control group valid? Randomization worked Intended treatment and control groups balanced Actual treatment and control groups same as intended Contamination from spillovers Was the intervention consistent in all treatment areas? Easier to guarantee in some cases than in others Do data exist to rule out alternate hypotheses?

External Validity Will the subjects in the experiment be representative of the entire population who will eventually receive the intervention? Logistically, much easier to do data collection in restricted area Less likely that experiment will generalize to entire country

Data Quality Sensitive questions Subjective questions How can we encourage subjects to give honest and complete answers? Subjective questions Self-reported vs quantitative measures “recall error” Hawthorne effect - People behave differently when they know they’re being watched Might be desirable to follow them closely for more data But that might make biases worse

Cost-Effectiveness Comparisons Resources are scarce – need to pick most effective programs Need to be able to convert impacts from various projects into one set of units How to compare improvement in nutrition to reduction in malaria?

Scaling Up Can intervention be implemented identically at scale? If not, is RCT still informative? Will the economy at large respond to the intervention at scale? (“general equilibrium effects”) Prices might go down – economies of scale Prices might go up – insufficient supply Spillover effects could set in

Final Thoughts RCT is gold-standard in terms of identifying causality But many complications arise during implementation Need to weigh theoretical advantages against practicalities – is it really the best method?