Advanced Improvement Practitioner Programme Measurement Mike Davidge

Issues and questions 2

WHAT TYPE OF PERSON ARE YOU? 3

Are you a plan-doer ?

Are you an auditor?

6 Are you an improver?

5 Measurement Sins 7 Measuring the wrong thing Having no baseline (or having a compromised one) Only collecting data at 2 points in time Presenting results in a misleading way Inappropriate or mindless use of statistics

7 Steps to measurement 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps Collect data 8

The Measures Checklist Part One Why important? Who owns? Definitions Goals Part Two Collect Analyse Review 9

Step 1: Decide your aim 10 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps Collect data  A worthwhile topic  Outcome focused  Measurable  Specific population  Clear timelines  Succinct but clear  A worthwhile topic  Outcome focused  Measurable  Specific population  Clear timelines  Succinct but clear  Specific  Measurable  Achievable  Realistic  Time-bound  Specific  Measurable  Achievable  Realistic  Time-bound

Step 2: Choose measures 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps Collect data 11

Useful Tools when choosing measures 12 Process maps Driver diagrams

Using the process map End ? Handover ? Decision Point ? Start ?

The ABCD of managing a service Activity: what we have actually done Backlog: what we should have done but didn’t Capacity: what we could have done Demand: what we should have done Listed in order of frequency of measurement

Step 3: Define measures 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps Collect data An operational definition is a description, in quantifiable terms, of what to measure and the steps to follow to measure it consistently 15

 Take a piece of A4 paper and follow my instructions to get an identical shape to mine Definitions Exercise 16

Types of calculation Counts Ratios or rates Percentages & Proportions Time between or cases between

Types of calculation: When to use what Counts –when the target population does not change very much –Example: Number of falls on an elderly ward (always full) Percentages –when the numerator is a subset of the denominator –Example: Percentage of patients who fell Ratios or rates –Numerator and denominator are measuring different things –Example: Falls per 100 bed days Time between or cases between –When you are tracking a ‘rare’ event, say one that occurs less than once a week on average –Example: Days since a patient last fell on this ward

Step 4: Collect your data 19 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps Collect data

Baselines 20

21 Analyse and present “The type of presentation you use has a crucial effect on how you react to data” 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps Collect data 21

Delayed transfers of care What you get presented with What did you decide to do?

How we assess performance #1: 2 point comparisons Why has the number of complaints gone up? Our service is getting worse. We need to do something! What decision are you going to make?

UNDERSTANDING & DEALING WITH VARIATION IN ANALYSIS 25

“If I could reduce my message to management to just a few words, I'd say it all has to do with reducing variation”. W Edwards Deming 26

What’s a person’s normal body temperature? 27

“Data contains both signal and noise. To be able to extract information, one must separate the signal from the noise within the data.” Walter Shewhart

RUN CHARTS 29

Run charts Plot data in time order Calculate and display median as a line Analyse chart by studying how values fall around the median Run charts 30

Summarise – the median Median defined –The median is the middle value of a finite list of numbers where the numbers are ordered from lowest value to highest value. If there is an even number of observations, the median is not unique, so one takes the mean of the two middle values. Why do we use it? –Not affected by extremely large or small values –Half of values will always be below/ above the median What do we need to beware of? –It tells us little about the spread of values –Tedious to calculate by hand if number of values is large 31

Using the complaints handout, graph paper, ruler and a pencil: Draw and label the axes Plot the dots (each monthly value) Work out the median Add a title (with dates) Add a legend Creating your own run chart 32

Does yours look like this? 33

At least 6 points above centre lineAt least 6 points below centre line Time  Below centre Time  Above centre Rule #1: a shift 34

Time  Downward trend Time  Upward trend At least 5 points all in upward direction At least 5 points all in downward direction Rule #2: a trend 35

Time  Rule #3: astronomical data point 36

Too many or too few ‘runs’ Use table to determine Time  2 nd run Time  1 st run Rule #4: runs

Using the 4 rules listed below, see if you have any special causes in your data Interpreting your run chart 38 Rule #1: A shift in the process, or too many data points in a run (6 points above or below the median) Rule #2: A trend (5 all increasing or decreasing) Rule #3: An “astronomical” data point Rule #4: An unusual number of runs (use the table to determine)

Apply the rules

SPC CHARTS 40

Control charts Plot data in time order Calculate and display mean as a line Calculate and display control limits as lines Analyse chart by studying how values fall around mean and between control limits Control charts

An example SPC chart 42

Summarise – the mean Mean defined –the arithmetic mean (or simply the mean) of a list of numbers is the sum of all of the list divided by the number of items in the list. Why do we use it? –Provides a useful estimate of the typical value –Easy to understand and calculate What do we need to beware of? –A few extremely large values can inflate the mean –It tells us little about the spread of values

Use the SPC handout, ruler and a pencil. This time the basic chart is drawn for you. You will be calculating and drawing the mean and control or process limits Then interpreting your chart Creating your own SPC chart 44

1. Plot individual values

In Excel use the formula =AVERAGE(range) 2. Calculate Mean & plot it Mean = 58

3: Derive moving range These are required to calculate the control limits The first row contains the chart data Use the second row to record the difference between successive data values The difference is always recorded as a positive value X Data Moving Range 05141

4: Calculate Average Moving Range R1R1 R2R2 R3R3 R4R4 R5R5 R6R6 R7R7 R8R8 R1R1 R 23 R Divide by number of values = Average moving range = 19.3 Add up the individual moving range values = …+18 Now what does this mean?

5: Calculate the control limits First derive one measure of variation (referred to as 1 sigma) 1 sigma = Average moving range = sigma = 17.1 Calculate lower limit as Mean – 3 sigma Calculate upper limit as Mean + 3 sigma Lower limit =58 – 3 x 17.1 Upper limit = x 17.1 Lower limit = 6.7 Upper limit = 109.3

Lower limit = 7 5. Plot limits Upper limit = 109

Constructing the chart: Summary There are 5 steps to constructing your chart: 1.Plot the individual values 2.Calculate the mean and plot it 3.Derive the moving range values 4.Calculate the average moving range (MR) 5.Derive upper and lower limits from this and plot them

Control charts are like yoghurt... They come in different flavours!

The XmR chart (or I-chart) X – for individual values mR – for moving range Makes fewest assumptions about how the data values are distributed around the mean Will give you reliable results in almost all situations

Some software to help you create SPC charts Limited version is FREE, full version costs £50

INTERPRETING SPC CHARTS 55

SPC CHART RULES Source: Lloyd Nelson, Technical Aids: The Shewhart Control Chart – Tests for Special Causes 56

Upper control limit Mean Lower control limit Rule # 1: Any single point outside the control limits

Upper control limit Mean Lower control limit Rule # 2: A shift At least 7 points consecutively either above or below the centre line

Upper control limit Mean Lower control limit Rule #3: A drift At least 7 points consecutively ascending or descending Note: the points can cross the centre line

Upper control limit Mean Lower control limit Rule #4: At least 14 points alternating up and down Note: the points do not have to alternate above and below the centre line

Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Other tests require you to plot the intermediate variation values

Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Rule #5: 2 out of 3 points outside 2V on the same side of the centre line

Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Rule #6: 4 out of 5 points outside 1V on the same side of the centre line

Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Rule #7: Reduced variation 15 points within 1V either side of the centre line

Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Rule #8: 2 processes 8 points in a row on both sides of the centre line with none within 1V

Why 7 points in a row? Toss a coin: Chances of getting a ‘head’? Toss it twice: Chances of getting two ‘heads’? 66

How many options? 67 1 toss2 tosses3 tosses4 tosses 1 in 21 in 41 in 81 in 16

Chances of ‘n’ heads in a row 68

SAFETY CROSSES AND RUN CHARTS 69

Your Safety Cross 70 Record the dates 04/07/ /07/ /07/2013

Calculate the days between falls 71 Date 04/07/ /07/ /07/2013 Days between 11 – 4 = 7 25 – 11 = 14 What value do we get if we have 2 falls on the same day?

Plot the days between on a run chart 72 Plot X axis as text not dates The higher the value, the better it is Use the 3 run chart rules to interpret

RECOMMENDED CHARTS 73

The Golden Rule when presenting data One picture, one message 74

Types of chart and when to use them Run & control charts Used to display performance over time Shows whether common causes or special causes (or both) are present in our data 75

Types of chart and when to use them Pareto charts Used to display the number of times distinct things happen such as reason for cancellation To separate the ‘vital few’ from the ‘useful many’ (the 80/20 rule) 76

Types of chart and when to use them Bar charts Used to display survey results Display for each question – which questions look different? Also display optionally for each respondent – are there types of respondent? 77 Source: Call of Duty

QUALITATIVE ANALYSIS

Surveys

Issues to consider Are all patients rating the same thing? Is there a ceiling effect? Do scales work in a linear fashion? –A score of 6 is twice as good as 3 –Strongly agree is twice as good as agree

Collecting patient experience data Interviews Patient tracking Surveys

Analysing qualitative data Thematic analysis: Look for the common themes Construct a story around typical findings The power of a good quote

The power of a quote Source: Call of Duty

Step 6 – Review measures It is a waste of time collecting and analysing your data if you don't take action on the results 84

DTOCs part 2 Verdict Stable within limits ( ) Not Capable of achieving target

What decision do you make? Is your information presented in a way that allows you to confidently make one of these decisions? DecisionBecause Do nothingPerformance ok Contingency plansSpecial cause variation Process redesignCommon cause variation 86

Team Review Meetings

Step 7: Keep going 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps Collect data 7 Steps now on YouTube Or put ‘Mike Davidge measurement’ into YouTube search box 88 You may not get it right first time! You may need several attempts. Remember PDSA

References The run chart basic reference –“The run chart: a simple analytical tool for learning from variation in healthcare processes”; Perla R, Provost L, Murray S; BMJ Qual Saf 2011;20:46e51. doi: /bmjqs A great introduction to variation and SPC –Understanding variation, Don Wheeler, A couple of useful websites/blogs – – 89

Application to management Fourth Generation Management –Brian L Joiner, McGraw-Hill, 1994 The Leader’s Handbook –Peter R Scholtes, McGraw-Hill, 1998 The New Economics (2 nd edition) –W Edwards Deming, MIT Press, 1994

Issues and questions 91