1. Randomized Control Trials
Difficulties to Consider

2. Costs Cost of intervention itself often not difficult to justify
Providing goods/services
Only including promising possibilities
New data are expensive
Quality of evaluation dependent on quality of data
More money spent on data is less money spent on providing the intervention to more people

3. Creating the Control Group
Is it politically feasible to deny treatment to some people?
How important is it to measure how well the intervention works?
Issue of trade-offs
Ethics less contested if:
Budget constraints would have prevented everyone from receiving the intervention anyway
Everyone eventually receives the intervention and the control group is only denied it initially (phased-in rollout)

4. Does Everyone Benefit? Necessary to deny control group intervention
But don’t want to actively hurt them
Can’t deceive
Can’t make them worse off than they’d otherwise be
Some sort of small gift/compensation typical –
careful not to make this into a second treatment
Must honor promises (phased-in rollout)

5. Internal Validity Was the control group valid?
Randomization worked
Intended treatment and control groups balanced
Actual treatment and control groups same as intended
Contamination from spillovers
Was the intervention consistent in all treatment areas?
Easier to guarantee in some cases than in others
Do data exist to rule out alternate hypotheses?

6. External Validity
Will the subjects in the experiment be representative of the entire population who will eventually receive the intervention?
Logistically, much easier to do data collection in restricted area
Less likely that experiment will generalize to entire country

7. Data Quality Sensitive questions Subjective questions
How can we encourage subjects to give honest and complete answers?
Subjective questions
Self-reported vs quantitative measures
“recall error”
Hawthorne effect - People behave differently when they know they’re being watched
Might be desirable to follow them closely for more data
But that might make biases worse

8. Cost-Effectiveness Comparisons
Resources are scarce – need to pick most effective programs
Need to be able to convert impacts from various projects into one set of units
How to compare improvement in nutrition to reduction in malaria?

9. Scaling Up Can intervention be implemented identically at scale?
If not, is RCT still informative?
Will the economy at large respond to the intervention at scale? (“general equilibrium effects”)
Prices might go down – economies of scale
Prices might go up – insufficient supply
Spillover effects could set in

10. Final Thoughts RCT is gold-standard in terms of identifying causality
But many complications arise during implementation
Need to weigh theoretical advantages against practicalities – is it really the best method?