Poor Research Designs in Policy Impact Studies: “Lies, Damn Lies, and Statistics” AHRQ 2007 Annual Conference: Improving Healthcare, Improving Lives September 26, 2007 Stephen Soumerai, ScD Department of Ambulatory Care and Prevention Harvard Medical School and Harvard Pilgrim Health Care

Pretest-Posttest Design Treatment Group Time O 1 X O 2 Posttest-Only Design Treatment Group Time X O 1 Weak Designs: No Causal Inference What Threats to Validity? Posttest-Only Design with Nonequivalent Groups (cross-sectional) Treatment Group Comparison Group Time X O 1 O 1

Non-equivalent Control Group Design Treatment Group Comparison Group R O 1 X O 2 Time O1O1 O2O2 Time Series Design Treatment Group Comparison Series possible Time O 1 O 2 O 3 X O 4 O 5 O 6 O 1 O 2 O 3 O 4 O 5 O 6 Strong Quasi-Experimental Designs

Soumerai et al, Milbank Q 1989; 67: Reported Effectiveness of Printed Education Materials Alone in Well-Designed vs. Inadequately Controlled Studies Adequately Controlled Studies Inadequately Controlled Studies

Effect Attributed by MCOP Example of an Inadequate Cross-sectional Design: Problems with Small Samples and Outliers

Post-only Evaluations of Several Drug Cost-Containment Policies on Medication Use and Health Outcomes Conclusions: Mixed effects on outcomes and costs Design: Post-policy comparisons of several groups (e.g., with and without employer insurance)  No data on baseline comparability  Statistical adjustment for group differences Problems:  Study groups were already different with respect to SES and health status  Instrumental variables, propensity scores, etc. can’t fully control for bias

Use Longitudinal Models Increases statistical power in quasi- experimental studies  Uses information on trends Multiple pre- and post-measurements of outcomes  Provide graphical evidence: visible versus statistical

Pearson et al, Arch Intern Med 2006; 166:572-9 Triplicate Prescription Policy Figure 1: Reductions in benzodiazepine use after Triplicate Prescription Policy among patients living in neighborhoods with different racial compositions

Benzodiazepine Use and Incidence of Hip Fracture among Women in Medicaid Before and After NY Regulatory Surveillance Cumulative Incidence of Hip Fracture per Female Users before Policy Bz Use among Female Users before Policy,%

Summary Points Longitudinal data allow for strong quasi- experimental designs  Provide more valid results  Visible effects almost always significant Creative use of comparison series  Unexposed comparison population  High risk subgroups  Unintended outcomes