Change Starts Here. The One about Demonstrating Change ICPC National Coordinating Center This material was prepared by CFMC (PM CO 2011), the Medicare Quality Improvement Organization for Colorado under contract with the Centers for Medicare & Medicaid Services (CMS), an agency of the U.S. Department of Health and Human Services. The contents presented do not necessarily reflect CMS policy.

Recap: measurement for IC-4 1.Time series outcomes – Effect on root cause/driver – Success of the intervention Rates; scores; rating scales Best-fit line or other signal indicating improvement What to do about outcomes not well portrayed as time-series 2.Intervention implementation – Reach/dosage of an intervention – Who was affected? Counts Rates among eligible population (offered, refused, completed)

Recap: suggested approach 1.Map out a detailed, community-level logic model of the intervention strategy. 2.Select and operationalize outcomes and processes from the logic model. 3.Develop and enforce the system for tracking implementation and outcome. 4.Effectively report time series data.

Recap: timing and duration When will improvement be detected? Considerations – How long should it take to observe an effect? – What should the effect look like? IC-4: ≥4 quarters of data within 18 months of community engagement – Ensure that the measurement period includes pre-intervention baseline data. Measure frequently – The more data points, the better. Monthly indicators lend themselves to run/control charts.

What’s this all about? Purpose – Confidently demonstrate that the interventions were effective. – Link interventions to observed changes in readmissions. – Validate or revise the logic model based on short- term outcomes.

Be thoughtful and careful Pitfalls – Waiting too long to begin the process – Checking progress infrequently – Going it alone Advice – Rapid cycle improvement Measure frequently, revise accordingly – Analytic support – NCC support

Tracking outcomes Time series – Run chart or control chart (≥12 data points) – Trend line (fewer data points) Cross-sectional; cohort – Group comparison Intervention vs. no intervention – Beneficiary-level Difference between pre- and post-intervention

Detecting improvement Does the intervention have an effect? Run chart: special cause – Co-occurrence with intervention deployment Best-fit trend line: statistical significance Between-group or pre-/post-intervention differences achieved and sustained

Time series: run charts Run chart – Data points overlaid against the median – Simple patterns suggest ‘special cause’ Runs (consecutive points above or below median) – Number of runs – Length of run Consecutive data points continually increasing/decreasing Very high or low point – Resources: Perla, Provost & Murray (2011) –

Run charts Perla, Provost & Murray (2011)

Time series: best-fit line Best-fit line among available data points – Cochran-Armitage test Statistical significance: slope of the line is different from zero – Requires analytic support – Resources: SAS documentation SAS paper: Liu (2007) –

Cross sectional and cohort data Not well portrayed as time series, per se… Group comparisons – Separate plot for each group on the same time-series graph Pre- and post-intervention change e.g., Patient Activation Measure – Consider ways to make it a group comparison Pre-intervention vs. post-intervention scores Sample of individuals who did not receive intervention Requires analytic support

What if we don’t see improvement? Refer to the logic model – External factors – Challenged assumptions What factors influenced the outcome? What adjustments could have been made? What other outcomes may have been measured?

Document what was learned Regardless of success or failure Context – What made the intervention successful or unsuccessful in your setting/community?

Additional resources Toolkit – measurement Run charts: Perla, Provost & Murray (2011) ICPCA NCC contact: Tom Ventura

Questions? The ICPC National Coordinating Center – Change Starts Here.