Causal inference for health system effectiveness: hard but essential Alan Zaslavsky Harvard Medical School

Challenges to system-level analysis “Health system" ≠ “Health care system“ Difficulty of experiments Political, institutional and ethical barriers, time, etc. Multilevel structure of systems Each level dependent on others Interests, incentives differ across levels Greatest variation often at lowest levels Interventions are often multilevel (directly & indirectly) Challenges to data collection & dissemination “Second Law" problem: “Successes" cannot be replicated (because dependent on special leadership or circumstances). Who is responsible? … for which piece of the problem?

Health system researcher Thanks to R. Crumb

Nonetheless, careful inference about causal effect is essential Demonstration vs evaluation Might be for intermediate outcome If well validated! “Third Law” problem: “The best way to look good is always selection.” “Hawthorne effects” Better performance because you know you are innovative, observed Effects of time, maturation, concurrent events

Valid inferences without randomization? Multiple "outcomes" across levels Intermediate outcomes, processes, transmission Descriptive (partition of variance) Inference from quasi-observational data Design improves even nonrandomized study Identify and measure variables associated with adopting intervention Foster controlled variation in roll-out Baseline assessment of controls Population rather than purely clinical focus