Measuring Improvement & Building A Measurement Plan Quality Academy Cohort 6 Residency 1 April 2013 Melanie Rathgeber, MERGE Consulting

Session Plan Purpose of QI Data –Difference between data for research, accountability, and improvement What to measure When to measure Data Collection

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident

Purpose of Data in QI Projects Need to know: - where we started (baseline) - how we change over time (e.g. each week) - when we have reached our target - Not for judgment (doesn’t go on dashboards or to external agencies) - Not for research

Data for Improvement/for Learning Not for Research Not for Accountability/Judgment

Methods for Improvement, Accountability and Research Improvement Accountability/Performance Measurement Clinical Research Purpose Improvement of care; application of evidence Comparison between organizations or to a criteria, reassurance, spur for change Generate evidence or new knowledge Data Collection Accept consistent bias in data collection Statistical adjustments to reduce bias Design of experiment to reduce bias Sample Size Small sequential samples of data 100% of available data Large samples so there is enough data for possible hypothesis Analysis Run Charts of Shewhart ChartsNAStatistics for hypothesis testing: T-tests, F-tests, Regression Analysis, etc). with p-value Adapted from: Jt Comm J Qual Improv. Adapted from: Jt Comm J Qual Improv Mar;23(3): The three faces of performance measurement: improvement, accountability, and research. Solberg LISolberg LI, Mosser G, McDonaldMosser GMcDonald

Discussion Examples in your organization, of measures/indicators for Accountability Research Improvement

Using a QI model/framework to build a measurement plan

What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement? Act Plan Study Do

1. What are we trying to accomplish? 2. How will we know that a change is an improvement? 3. What changes can we make that will result in improvement? Act Plan Study Do Measures: Key Measures Family of measures Balanced set of measures Most of the measures we use in QI help us answer Question 2.

1. What are we trying to accomplish? 2. How will we know that a change is an improvement? 3. What changes can we make that will result in improvement? Act Plan Study Do Ideas based on data - Do you have a hunch that there is variation in turnaround times? Do you think its an idea to test?

1. What are we trying to accomplish? 2. How will we know that a change is an improvement? 3. What changes can we make that will result in improvement? Act Plan Study Do Prediction and study – ‘small’ PDSA measures What were your PDSA measures during the airplane exercise? PDSA measures

How do you know what to measure?

What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement? Act Plan Study Do Start with your aim statement. You should be able to identify at least one outcome measure from there.

“Family of Measures” “Balanced Set of Measures” “Key Project Measures”

–Outcome measures Based on your Aim statement (usually) Voice of the patient/customer –Process measures come from your ideas for change (usually) Voice of the system – what is being done differently? Quicker improvement than outcomes –Balancing measures (outcomes or processes) What unintended consequences might occur? Family of Measures

Pop quiz: Percent of sepsis patients who had culture before antibiotics 28 day mortality rate post-sepsis

Pop quiz: Median wait time from appointment to procedure Percent of clinicians aware of referral criteria

Pop quiz: Percent of patients screened for depression Percent of patients with depression receiving treatment of choice

Other Examples?

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident

How do you know when to measure?

Start right now. Each day, or week, or month Not each quarter Not pre-post

Pre = 8 days wait time Post = 3 days wait time Why not pre and post? Adapted from Health Care Data Guide, p. 16 Figure 1.5 and 1.6, Provost and Murray, San Francisco: Jossey Boss

Scenario 1 when we measure the same thing over time.

Scenario 2 when we measure the same thing over time.

Scenario 3 when we measure the same thing over time.

Displaying Key Measures over Time - Run Chart -Data displayed in time order -Time is along X axis -Result along Y axis -Centre line = median -One “dot” = one sample of data -Sample size = each “dot” should have the same n

Annotated Run Charts -Have our specific changes resulted in an improvement? - Be careful about assuming direct cause and effect. May be the entire set of changes responsible for improvement.

Showing improvement: No improvement. Random fluctuation. Improvement. Trend going up.

Non-Random Signals on Run Charts A Shift: 6 or more An astronomical data point Too many or too few runs A Trend 5 or more The Data Guide, p 3-11 Evidence of a non-random signal if one or more of the circumstances depicted by these four rules are on the run chart. The first three rules are violations of random patterns and are based on a probability of less than 5% chance of occurring just by chance with no change.

-Start your run chart with one data point -General guideline: data points -Depends on duration of your data collection, e.g. you might need to group data by week, instead of month - 10 data points for baseline data How many data points?

Pop quiz: Percent of sepsis patients who had culture before antibiotics 28 day mortality rate post-sepsis Median wait time from appointment to procedure Percent of clinicians aware of referral criteria Percent of patients screened for depression Percent of patients with depression receiving treatment of choice

How do you start collecting data for your measures?

Some tips for getting started “Measurement should be used to speed things up, not to slow them down” - IHI Breakthrough Series Guide

Some tips for getting started 1. Don’t wait for the information system 2. Use sampling 3. Seek usefulness not perfection

1. Don’t wait for the information system/for decision support. “Real time data” drives improvement. This can involve new ways of doing things.

Tools -Data collection form -Chart review (with data collection form) -Observation (with data collection form) -Surveys -Electronic Data and Administrative Data

New ways of doing things: stretch yourself to……. Find ways of capturing data in real time – don’t wait for the information system. Find ways of embedding data collection into workflow

Some tips for getting started 2. Sampling – Small samples are okay. Sample size increases over time.

Small samples per day or week are okay Sample size builds over time How much data to satisfy team that it is representative? Simple strategies: - every 5 th patient - all patients on Thursday morning If you are reporting externally, or if you want to publish results of QI – may need different strategy

Some tips for getting started 3. Seek Usefulness, Not Perfection - Discuss an Operational Definition With Your Team

Key here is to understand the purpose of measures. Your data does not need to be comparable to other organizations. Some bias is okay.

Operational Definitions Deciding on an operational definition should be done with your QI team What time frame? Which patients? What criteria? What diagnosis? What constitutes “met the guideline?” What about patients that wanted something different? etcetera, etcetera, etcetera ……………………..

Operational Definition Example Basic definition: Patient satisfaction ratings from patient survey

Operational Definition Example Basic definition: Patient satisfaction ratings from patient survey Operational definition: Percent of surgical patients discharged this week that rated their experience with the discharge process as good or excellent, based on the surgical patient survey

“The Data Are Wrong”

Not a matter of right versus wrong What is your operational definition? Involve others from the start in this decision.

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident

Measurement Plan Worksheet MeasureOperational Definition Outcome, Process or Balancing Data Collection Strategy Frequency of Data Collection How will measure be displayed Baseline result Target result

Review – Key Measures –Based on aim statement and “how will we know our change is an improvement” –Provides feedback that changes are having the desired impact –Collected over the life time of your project – starting with baseline –Outcome, process and balancing –Displayed over time (e.g. Run Chart) – daily, weekly, or monthly –Guideline: between 3-5 measures in total –Small samples collected frequently –Operational definitions – feasibility over precision

1. Start with your Aim statement and then answer “how will we know a change is an improvement”. These are your key measures: At least one outcome measure At least one process measure (initial ideas) Balancing measure (if necessary) 2. Operational Definition for each – what EXACTLY are you going to measure? Will two different people measure it the same way based on your definition? Are you going to report as a percentage, a rate, a count, an average, a median? 3. When are you going to measure and what is your sample? Every day, week, or month? Are you going to sample or include all patients/charts/staff etc? 4. Other decisions about data collection – who is going to collect, who is going to analyze? Do you need a data collection form or some sort of audit tool? Do you need to develop a questionnaire? 5. Displaying and analyzing data – Are you using a run chart? (if not, why not?) 6. Where are you going to send and display updated data reports/run charts? Your Measurement Plan

On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 4 Not at all confident Extremely confident