1. Causal inference for health system effectiveness: hard but essential
Alan Zaslavsky
Harvard Medical School

2. Challenges to system-level analysis
“Health system" ≠ “Health care system“
Difficulty of experiments
Political, institutional and ethical barriers, time, etc.
Multilevel structure of systems
Each level dependent on others
Interests, incentives differ across levels
Greatest variation often at lowest levels
Interventions are often multilevel (directly & indirectly)
Challenges to data collection & dissemination
“Second Law" problem:
“Successes" cannot be replicated (because dependent on special leadership or circumstances).
Who is responsible? … for which piece of the problem?

7. Health system researcher
Thanks to R. Crumb

8. Nonetheless, careful inference about causal effect is essential
Demonstration vs evaluation
Might be for intermediate outcome
If well validated!
“Third Law” problem:
“The best way to look good is always selection.”
“Hawthorne effects”
Better performance because you know you are innovative, observed
Effects of time, maturation, concurrent events

9. Valid inferences without randomization?
Multiple "outcomes" across levels
Intermediate outcomes, processes, transmission
Descriptive (partition of variance)
Inference from quasi-observational data
Design improves even nonrandomized study
Identify and measure variables associated with adopting intervention
Foster controlled variation in roll-out
Baseline assessment of controls
Population rather than purely clinical focus