Natural Experiments Versus RCTs Steven Gortmaker, Ph.D. Harvard School of Public Health /Harvard Prevention Research Center

Typical Criticisms of RCTs RCT interventions are often unrealistic as to what can be implemented and sustained in natural settings. The long lead time and demands of random assignment required for RCTs can preclude evaluations of innovative, expensive or complex interventions or policy changes in natural settings

Summary of Trade-Offs Natural Experiment –Worse internal validity; but more data points can help –Often limited generalizability –Potential to study more innovative, expensive, difficult to implement programs/policies Group Randomized Trial –Better internal validity –Often limited generalizabilty: but potential to improve –More limited programs/policies to study with RCT due to costs and difficulty of randomizing

Internal Validity: Control of Selection/Confounding Natural Experiment –Success dependent on selection of control sample (e.g. propensity matching) –Multiple pre-intervention and post-intervention data points can strengthen design Group Randomized Trial –The major strength of RCT’s –Multiple pre-intervention and post-intervention data points can strengthen design

Hypothetical Natural Experimental Design to Evaluate Impact of School Food Service Change, With Single Data Point Pre and Post-Intervention Year Mean BMI Intervention Begins Control Intervention

Hypothetical Natural Experimental Design to Evaluate Impact of School Food Service Change, With Multiple Data Points Pre and Post-Intervention Year Mean BMI Intervention Begins Control Intervention

A Suggestion An underutilized design in requests for proposals are natural experimental designs that include time-series of pre and post intervention data - the interrupted time-series design These designs can be effectively used to evaluate innovative, expensive or complex interventions or policy changes in natural settings

Implications for Funders Require assessment of implementation and sustainability in natural settings in proposals that utilize RCTs Encourage proposals for evaluations of innovative, expensive or complex real world interventions and policy changes, using natural experimental designs that include time-series of pre and post intervention data