Change Starts Here. The One about Outcomes and Indicators ICPC National Coordinating Center This material was prepared by CFMC (PM CO 2011), the Medicare Quality Improvement Organization for Colorado under contract with the Centers for Medicare & Medicaid Services (CMS), an agency of the U.S. Department of Health and Human Services. The contents presented do not necessarily reflect CMS policy.

Recap: measurement for IC-4 1.Time series outcomes – Effect on root cause/driver – Success of the intervention Rates; scores; rating scales Best-fit line or other signal indicating improvement What to do about outcomes not well portrayed as time-series 2.Intervention implementation – Reach/dosage of an intervention – Who was affected? Counts Rates among eligible population (offered, refused, completed)

Recap: suggested approach 1.Map out a detailed, community-level logic model of the intervention strategy. 2.Select and operationalize outcomes and processes from the logic model. 3.Develop and enforce the system for tracking implementation and outcome. 4.Effectively report time series data.

Who should participate in outcomes selection and measurement? Considerations – Who will implement the interventions? – Who will be collecting data? – Community stakeholders To whom will the observed changes will be meaningful? Resources brought by the QIO – Building capacity for outcome measurement Logic models Data collection – Technical assistance; analytic support Data management and reporting Detecting improvement (tests of trend, run/control charts)

Recap: logic model outcomes Expected short-, medium-, and long-term changes and improvements Short-term – Specific improvements in the targeted driver or root cause Medium-term – Related outcomes along the causal path Long-term – Improved care transitions – Avoided readmission – Improved health care utilization Implications of potential negative changes or non-changes Short-/medium-term outcomes are measured for IC-4.

Recap: outcomes selection SMART objectives guide outcome selection. S pecific Concrete; represents what, or who, is expected to change M easureable Can be seen, heard, counted, etc. A ttainable Is likely to be achieved R esults-oriented Generates meaningful, valued results T imed Has an acceptable target date

Going further: attainability, moveabilty Short-term outcomes are more likely to show movement… …but consider downstream (medium-term) outcomes if short- term outcomes are not feasibly measured. Outputs Short-term outcomes Long-term outcomes

Indicators What does the outcome look like? How would we know it improved? Create operational definitions. – Specifics of how the outcome is measured Make it a number. Quantitative data can be plotted over time or compared across groups. – Rate, percentage Numerator and denominator – Score (continuous) e.g., 0 to 100 points – Rating scale (ordinal) e.g., [0 - ‘never’] -- [1 - ‘sometimes’] -- [2 - ‘always’] Simple counts are not very informative on their own. Find a way to use them as the numerator of a rate or percentage.

Timing and duration When will improvement be detected? Considerations – How long should it take to observe an effect? – What should the effect look like? Abrupt, sustained improvement Rises and falls, with gradual trend towards improvement IC-4: ≥4 quarters of data within 18 months of community engagement – Ensure that the measurement period includes pre-intervention baseline data. Measure frequently – The more data points, the better. Monthly indicators lend themselves to run/control charts.

Level of analysis Where will improvement be seen? Provider-/initiative-level – A change occurs across several events, people, or organizations. Process improvements (re: standardization; information transfer) – e.g., inpatient satisfaction with discharge information Utilization outcomes – e.g, 7-day readmission rates; 30-day primary care follow-up rates Patient-level – Something happens to an individual person related to his/her patient experience. Patient activation (e,g., change in PAM score) Utilization (e.g., prevented readmission) Patient-level data should be aggregated to provider-/initiative-level for reporting.

Collecting data Considerations – What method? Sampling Survey, case review, etc. – Who collects the data? Who analyzes it? – How will data quality be ensured? Make it explicit. – Standardization, data specifications – Accountability Don’t forget to track implementation (process) measures, as well. – Counts, rates among eligible population (offered, refused, completed)

More to come Detecting and reporting improvement Context and reasons for success/failure

Resources Toolkit – measurement Measuring Program Outcomes: A Practical Approach ctical_approach/program_film Excerpts: ICPCA NCC contact: Tom Ventura

Questions? The ICPC National Coordinating Center – Change Starts Here.