The Use and Evaluation of Experiments in Health Care Delivery Amanda Kowalski Associate Professor of Economics Department of Economics, Yale University September 26, 2015

What are the questions that YOU want to answer?

How can I help you to answer them?

A Call to Action 47/6223/720.full

Heard in the Trenches: Barriers to Randomization

“Why randomize?”  Way to mitigate influence of confounding factors  If you just compare treated group to a non-treated group, there could be other factors that produce different outcomes  If you just compare treated group after intervention to treated group before intervention, there could be other factors that changed over time

“Can’t you just analyze data collected after the intervention?”  It IS important to examine existing administrative data  Hard to learn about causality if you only have data on the treatment group after the intervention

“Is randomization fair?”  We only randomize if we don’t know if the intervention will work – “equipoise”  Randomization standard in clinical trials for medical interventions  When resources are scarce, randomization can be a fair way to allocate them

“Is randomization fair?” Example: Oregon Health Insurance experiment  Some funds to expand Medicaid coverage, but not enough to expand coverage to all interested parties  Held a lottery in 2008

“Is randomization fair?” Example: Oregon Health Insurance experiment

 Key findings:  Increased health care utilization  Emergency room utilization increased  Decrease in depression  No changes to physical health  Reduced financial strain  No discernable impact on labor market outcomes

“But some large-scale questions can’t be studied with randomization!”  For example, some initiatives like bundled payments might have larger impacts if they are implemented more broadly  Could also randomize fraction of population affected across different sites

“I’m too busy implementing this initiative to think about anything else!”  Randomization can be a seamless part of implementation  After implementation is too late

“I’m afraid to find out that what I have done does not work!”  Health care industry full of altruistic people who want to improve quality, increase access, and decrease cost  If your intervention does not work, can try something else next  If your intervention does work, evidence indicating so can be useful to others, broadening the impact of your work

“Will the implementation be costly?”  Clinical trials often costly  Subject recruitment  Informed consent  Randomized experiments in health care delivery  Subjects already in system  Consent waived  Costs of designing experiment  Costs of implementing randomization  Costs of collecting data – could focus on existing  Costs of analyzing data

“How will results be disseminated?”  Results will be published regardless of outcome  Institutional partner can opt for anonymity before publication

“When is a good time to get started?”  Before implementing a new intervention  Baseline data can be collected  Enriches comparison of treatment to control  Ensures that outcomes can be measured  Program probably not rolled out to everyone at the same time anyway

“Which interventions are best studied with randomization?”  Potential for large, detectable outcomes  But outcomes are not known  Large potential number of subjects  Increases statistical power  Interventions that would be implemented anyway  Increases real-world applicability  Successful implementation paves the way for future randomization of other initiatives

“What are the advantages of partnering with an economist?”  Statistical techniques  More subtle than comparing treatment group to control group  Dissemination  Potential to reach different audience  Results prepared by an independent entity potentially more impartial

“I’m on board with randomization. What’s next?”  What is the problem to be addressed?  What administrative data are available and how can they be accessed?  Who will be the implementing partners?  C-level advocate to push project through  Administrative contact for day-to-day

Let’s talk further!

  Slides on my website, video will be posted  …Tell your friends to examine slides, watch this presentation, propose an idea, and contact me!

What are the questions that YOU want to answer?