The search for internal validity in improvement Frank Davidoff Learning Lab – 2013 IHI Forum

Improvement has a two-part mantra Part 1: – All improvement involves change Part 2: – Not all change is improvement

Local project, part 1: make change “Here’s how we made (system-level) change happen…” – Identified a dysfunction in the system – Came up with an innovation (better process; change strategy for getting there) – Implemented our strategy in local context – Used small tests of change to refine innovation – Spread and maintained changes (Way different from giving a pill…) So what’s next?

Local project, part 2: find out whether change is improvement “Here’s how we learned whether our change was improvement…” – Chose outcomes: processes, patients’ clinical condition – Developed outcome measures – Created informal systems for collecting, displaying, using outcomes data (quantitative, qualitative) – Used these data locally to study the impact of changes, modify change strategy

Yes, Virginia, there is “study” in local improvement projects Informal study is an inherent part of all meaningful improvement – Used to check on impact of change (“Did it work?”) – Especially visible in “Plan-Do-Study-Act” cycles (originally called “Plan-Do-Check-Act” cycles) – Not related to whether project is meant “for publication”

Where have we gotten in our local project? In part 1: we made change happen In part 2: we produced informal outcomes data to demonstrate improvement – Good enough: allows project staff to modify, spread, maintain change BUT data quality (completeness, accuracy) is uncertain; no control for confounders, biases – “lite” study data Result: weak internal validity! – i.e., unlikely to convince skeptics elsewhere about improvement

How can we strengthen the evidence? Shift “up” to formal planning and study – Identify plausible theory of performance change – Adopt specific study design – Select/define relevant outcomes – Develop reliable data collection process, robust data quality control – Analyze results (e.g., grounded theory; statistics; time series, esp. statistical process control) Creates “research level” data Result: stronger internal validity! – i.e., more likely to convince editors, peer reviewers, rest of the world – that our change was really an improvement

Summing up Making change locally includes informal study of outcomes – Useful: makes project possible – but data somewhat “fuzzy” – Result: Internal validity is weak Formal study of change process and outcome requires “research level” methods – Scholarly: contributes to general knowledge – Not every formal study feature is required, but the more features the better – Result: internal validity is stronger

Internal validity in improvement studies: key reference Solberg L, et al. The three faces of performance measurement: improvement, accountability, and research. Joint Comm J Qual Improvement 1997;23: