The Use of Funnel Plots & Multi- Year Cumulative Data to Track Hospital Performance Herbert MA, Hamman BH, Roper KL, Ring WS, Edgerton JR, Texas Quality Initiative The American Association of Thoracic Surgeons April 26, 2015 Seattle, Washington

Nothing to Disclose

The Texas Quality Initiative 27 Hospitals in North Texas agreed to share clinical and administrative data 27 Hospitals in North Texas agreed to share clinical and administrative data All participated in the STS Database All participated in the STS Database 26,634 cardiac procedures from 1/ / ,634 cardiac procedures from 1/ / ,379 isolated CABG were analyzed for observed to expected (O/E) operative mortality 13,379 isolated CABG were analyzed for observed to expected (O/E) operative mortality There was a need to graphically represent the data There was a need to graphically represent the data – Simple – Easy to understand

The Funnel Plot

Methods A funnel plot is centered on a benchmark with 95% confidence intervals drawn on the graph. A funnel plot is centered on a benchmark with 95% confidence intervals drawn on the graph. To assess operative mortality and allow for risk correction, the observed to expected (O/E) ratio is used. To assess operative mortality and allow for risk correction, the observed to expected (O/E) ratio is used. The case volume is plotted on the horizontal axis The case volume is plotted on the horizontal axis O/E ratio on the vertical axis; O/E ratio on the vertical axis; either annual data or multi-year data can be shown. either annual data or multi-year data can be shown.

The Funnel Plot X Axis: Volume of Cases (CABG) Y Axis: O/E Ratio for isolated CABG An O/E of 1 is expected 95% Confidence intervals surround “1”

The Funnel Plot Worse than 1, but not statistically different Better than 1, but not statistically different Outlier for poor performance Outlier for good performance

The Funnel Plot At low volume it is very hard to become an outlier At low volume it is very hard to become an outlier In fact, at less than 200 cases, you cannot become an outlier for good performance In fact, at less than 200 cases, you cannot become an outlier for good performance The Problem: The Problem: Most hospitals analyze their data on an annual basis Most hospitals analyze their data on an annual basis Most Hospitals do less than 200 cases per year Most Hospitals do less than 200 cases per year They cannot reveal They cannot reveal themselves as outlier themselves as outlier due to wide confidence due to wide confidence interval at low volume interval at low volume Year after year they find Year after year they find that their results are “OK” that their results are “OK”

Five Year Data of all Hospitals allows Comparison of Results Five Year Data of all Hospitals allows Comparison of Results Five hospitals are above the upper 95% confidence interval, statistically worse than the target value of 1.0. Five hospitals are above the upper 95% confidence interval, statistically worse than the target value of 1.0. Seven Hospitals O/E is <1, but still within the confidence interval Seven Hospitals O/E is <1, but still within the confidence interval

The Funnel Plot PROBLEM No Problem

How can we account for this problem? Plotting running totals moves the result to the right, where the funnel is narrower Plotting running totals moves the result to the right, where the funnel is narrower The results can begin to show statistically significant differences from “1” The results can begin to show statistically significant differences from “1” More important….. More important….. Trends become visually apparent Trends become visually apparent Poorly performing hospitals can be identified Poorly performing hospitals can be identified – Even before the results reach statistical significance – Urgent interventions can be put in place

Plotting Running 5 Year Totals Plotting Running 5 Year Totals Year 1 Year Year 1+ 2 Year Year

Plotting Running 5 Year Totals  Year 6 2 Year 2 Year Year 2+ 3 Year Year

How Does This Help Let’s see some examples from the TQI Data Let’s see some examples from the TQI Data These are real data from real hospitals These are real data from real hospitals Some of the examples are from different time intervals, because… Some of the examples are from different time intervals, because… I Selected graphs to illustrate different scenarios I Selected graphs to illustrate different scenarios Colored dots represent annual data Colored dots represent annual data Green line represents the running 5 year total Green line represents the running 5 year total

Even with excellent outcomes, a hospital with case volumes under 200 cannot become an outlier for good performance However, Cumulative data will reveal excellence (in 1 more year)

Annual Data tightly clustered: O/E doesn’t change much With Cumulative Data … The Curve is flat, but at higher volume becomes an outlier

Annual Data is all within the funnel Hospital perceives “No Problem” Cumulative Data Unmasks Outlier for Poor Performance

Annual Data is Inconclusive: 3 out of 5 years are within the funnel Cumulative Data  Slope of the Curve is Predictive of Poor Performance At Year 3 Intervention is Needed, This is even before hospital becomes an outlier in year 5

Annual Data: 4 of 5 years O/E is > 1 Cumulative Data Shows a downward slope We have no concerns about this hospital

Conclusions The use of funnel plots allows easy comparison of individual programs The use of funnel plots allows easy comparison of individual programs Analyzing only annual data can lead to a false sense of satisfaction Analyzing only annual data can lead to a false sense of satisfaction The plotting of a five year running total will provide sufficient volume to reveal an accurate assessment The plotting of a five year running total will provide sufficient volume to reveal an accurate assessment the trend (slope) may give an indication of effectiveness of quality improvement programs in place. the trend (slope) may give an indication of effectiveness of quality improvement programs in place.

Background Behind the effort for more transparency and better outcomes is a need to measure and analyze data to present an accurate, clear picture. Since annual physician and hospital case volumes are often low, confidence intervals for many measures are wide and it is difficult to separate performance improvements from noise. The use of funnel plots with annual and especially multi-year data provide a more reliable estimate of performance compared to national benchmarks.

Results Plotting single year data indicates performance with reference to the STS benchmark (O/E=1) and shows whether the outcome is outside the 95% confidence interval. Usually volumes are small enough that confidence intervals are wide. Multi-year plots of annual results show year over year changes but suffer from similar annual volumes, which still leave the data in the wide part of the funnel. The running total adds annual volumes moving the result to a larger volume position where the funnel is narrower. At this point it is often possible to determine that the results are showing statistically significant differences from 1. Trends also become more evident.