2. Childhood Obesity Scenario: Quasi- Experiments and Natural Experiments Versus RCTs Steven Gortmaker, Ph.D. Harvard School of Public Health /Harvard Prevention Research Center

3. Definitions Quasi-experiments: can have all the attributes of randomized controlled trial with pretest and posttest data; key difference: no random assignment to intervention versus control Natural experiment: can have all the attributes of randomized controlled trial with pretest and posttest data; key differences: –no random assignment –experimenter does not control intervention

4. The Research Question Can be same in Quasi-experiment and Natural experiment as in Randomized Controlled Trial However, random assignment can set limits: –Long lead time for the study/results; may preclude rapid evaluations of innovations – Funding agency often pays for the intervention - may preclude evaluations of expensive or complex interventions or policy changes

5. Evaluation Data Quasi/Natural Experiment –Quasi-experiment has similar data requirements to RCT –Natural experiments may need to rely on surveillance data Group Randomized Trial –Want pre-post data on key measures

6. Internal Validity: Control of Selection/Confounding Quasi/Natural Experiment –Success dependent on selection of control sample (e.g. propensity matching) –Multiple pre-intervention and post-intervention data points can strengthen design Group Randomized Trial –The major strength of RCT’s –Multiple pre-intervention and post-intervention data points can strengthen design

7. Hypothetical Quasi Experimental Design to Evaluate Impact of School Food Service Change, With Single Data Point Pre and Post-Intervention Year Mean BMI Intervention Begins Control Intervention

8. Hypothetical Quasi Experimental Design to Evaluate Impact of School Food Service Change, With Multiple Data Points Pre and Post-Intervention Year Mean BMI Intervention Begins Control Intervention

9. Loss to Follow-up Quasi/Natural Experiment –Similar concerns to RCTS –Can study predictors of loss, model loss –Natural experiment: a concern is lack of detailed baseline data Group Randomized Trial –As with Quasi-experiment, can study predictors of loss, model loss

10. External Validity: Generalizability Quasi/Natural Experiment –Quasi-experiment: similar to RCT but may have broader sample of participant sites –Natural experiment: concerns re selection of intervention sites Group Randomized Trial –Participating sites may differ substantially from target population –As with Quasi-experiment, can study participant sites versus non-participants

11. Intervention Costs Quasi/Natural Experiment –Quasi-experiment: can be similar to RCT, or can capitalize on other funding –Natural experiment: intervention costs generally not borne by funding agency Group Randomized Trial –Intervention costs often borne by funding agency; can limit cost of intervention program –If costs borne by outside agency, difficulty with randomization (but not always!)

12. Evaluation Costs Quasi/Natural Experiment –Quasi-experiment: can be similar to RCT re data collection –Natural experiment: can be less expensive by using extant data Group Randomized Trial –Similar data collection costs re Quasi-experiment. –A hypothesis: typically data collection costs in RCTs are much larger than intervention costs

13. Summary of Trade-Offs Quasi/Natural Experiment –Worse internal validity; but more data points can help –Some limited generalizability - particularly in natural experiments –Potential to study more innovative, expensive, difficult to implement programs/policies Group Randomized Trial –Better internal validity –Often limited generalizabilty; but potential to improve –More limited programs/policies to study with RCT due to costs and difficulty of randomizing