1. Advanced Improvement Practitioner Programme Measurement Mike Davidge

2. Issues and questions 2

3. WHAT TYPE OF PERSON ARE YOU? 3

4. Are you a plan-doer ?

5. Are you an auditor?

6. 6 Are you an improver?

7. 5 Measurement Sins 7 Measuring the wrong thing Having no baseline (or having a compromised one) Only collecting data at 2 points in time Presenting results in a misleading way Inappropriate or mindless use of statistics

8. 7 Steps to measurement 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps 4-6 4 Collect data 8

9. The Measures Checklist Part One Why important? Who owns? Definitions Goals Part Two Collect Analyse Review 9

10. Step 1: Decide your aim 10 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps 4-6 4 Collect data  A worthwhile topic  Outcome focused  Measurable  Specific population  Clear timelines  Succinct but clear  A worthwhile topic  Outcome focused  Measurable  Specific population  Clear timelines  Succinct but clear  Specific  Measurable  Achievable  Realistic  Time-bound  Specific  Measurable  Achievable  Realistic  Time-bound

11. Step 2: Choose measures 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps 4-6 4 Collect data 11

12. Useful Tools when choosing measures 12 Process maps Driver diagrams

13. Using the process map End ? Handover ? Decision Point ? Start ?

14. The ABCD of managing a service Activity: what we have actually done Backlog: what we should have done but didn’t Capacity: what we could have done Demand: what we should have done Listed in order of frequency of measurement

15. Step 3: Define measures 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps 4-6 4 Collect data An operational definition is a description, in quantifiable terms, of what to measure and the steps to follow to measure it consistently 15

16.  Take a piece of A4 paper and follow my instructions to get an identical shape to mine Definitions Exercise 16

17. Types of calculation Counts Ratios or rates Percentages & Proportions Time between or cases between

18. Types of calculation: When to use what Counts –when the target population does not change very much –Example: Number of falls on an elderly ward (always full) Percentages –when the numerator is a subset of the denominator –Example: Percentage of patients who fell Ratios or rates –Numerator and denominator are measuring different things –Example: Falls per 100 bed days Time between or cases between –When you are tracking a ‘rare’ event, say one that occurs less than once a week on average –Example: Days since a patient last fell on this ward

19. Step 4: Collect your data 19 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps 4-6 4 Collect data

20. Baselines 20

21. 21 Analyse and present “The type of presentation you use has a crucial effect on how you react to data” 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps 4-6 4 Collect data 21

23. Delayed transfers of care What you get presented with What did you decide to do?

24. How we assess performance #1: 2 point comparisons Why has the number of complaints gone up? Our service is getting worse. We need to do something! What decision are you going to make?

25. UNDERSTANDING & DEALING WITH VARIATION IN ANALYSIS 25

26. “If I could reduce my message to management to just a few words, I'd say it all has to do with reducing variation”. W Edwards Deming 26

27. What’s a person’s normal body temperature? 27

28. “Data contains both signal and noise. To be able to extract information, one must separate the signal from the noise within the data.” Walter Shewhart

29. RUN CHARTS 29

30. Run charts Plot data in time order Calculate and display median as a line Analyse chart by studying how values fall around the median Run charts 30

31. Summarise – the median Median defined –The median is the middle value of a finite list of numbers where the numbers are ordered from lowest value to highest value. If there is an even number of observations, the median is not unique, so one takes the mean of the two middle values. Why do we use it? –Not affected by extremely large or small values –Half of values will always be below/ above the median What do we need to beware of? –It tells us little about the spread of values –Tedious to calculate by hand if number of values is large 31

32. Using the complaints handout, graph paper, ruler and a pencil: Draw and label the axes Plot the dots (each monthly value) Work out the median Add a title (with dates) Add a legend Creating your own run chart 32

33. Does yours look like this? 33

34. At least 6 points above centre lineAt least 6 points below centre line Time  Below centre Time  Above centre Rule #1: a shift 34

35. Time  Downward trend Time  Upward trend At least 5 points all in upward direction At least 5 points all in downward direction Rule #2: a trend 35

36. Time  Rule #3: astronomical data point 36

37. Too many or too few ‘runs’ Use table to determine Time  2 nd run Time  1 st run Rule #4: runs

38. Using the 4 rules listed below, see if you have any special causes in your data Interpreting your run chart 38 Rule #1: A shift in the process, or too many data points in a run (6 points above or below the median) Rule #2: A trend (5 all increasing or decreasing) Rule #3: An “astronomical” data point Rule #4: An unusual number of runs (use the table to determine)

39. Apply the rules

40. SPC CHARTS 40

41. Control charts Plot data in time order Calculate and display mean as a line Calculate and display control limits as lines Analyse chart by studying how values fall around mean and between control limits Control charts

42. An example SPC chart 42

43. Summarise – the mean Mean defined –the arithmetic mean (or simply the mean) of a list of numbers is the sum of all of the list divided by the number of items in the list. Why do we use it? –Provides a useful estimate of the typical value –Easy to understand and calculate What do we need to beware of? –A few extremely large values can inflate the mean –It tells us little about the spread of values

44. Use the SPC handout, ruler and a pencil. This time the basic chart is drawn for you. You will be calculating and drawing the mean and control or process limits Then interpreting your chart Creating your own SPC chart 44

45. 1. Plot individual values

46. In Excel use the formula =AVERAGE(range) 2. Calculate Mean & plot it Mean = 58

47. 3: Derive moving range These are required to calculate the control limits The first row contains the chart data Use the second row to record the difference between successive data values The difference is always recorded as a positive value X Data 005489 Moving Range 05141

48. 4: Calculate Average Moving Range R1R1 R2R2 R3R3 R4R4 R5R5 R6R6 R7R7 R8R8 R1R1 R 23 R 24 72325 162033351318 Divide by number of values = 463 24 Average moving range = 19.3 Add up the individual moving range values = 7+23+25+25+…+18 Now what does this mean?

49. 5: Calculate the control limits First derive one measure of variation (referred to as 1 sigma) 1 sigma = Average moving range 1.128 = 19.3 1.128 1 sigma = 17.1 Calculate lower limit as Mean – 3 sigma Calculate upper limit as Mean + 3 sigma Lower limit =58 – 3 x 17.1 Upper limit =58 + 3 x 17.1 Lower limit = 6.7 Upper limit = 109.3

50. Lower limit = 7 5. Plot limits Upper limit = 109

51. Constructing the chart: Summary There are 5 steps to constructing your chart: 1.Plot the individual values 2.Calculate the mean and plot it 3.Derive the moving range values 4.Calculate the average moving range (MR) 5.Derive upper and lower limits from this and plot them

52. Control charts are like yoghurt... They come in different flavours!

53. The XmR chart (or I-chart) X – for individual values mR – for moving range Makes fewest assumptions about how the data values are distributed around the mean Will give you reliable results in almost all situations

54. Some software to help you create SPC charts Limited version is FREE, full version costs £50

55. INTERPRETING SPC CHARTS 55

56. SPC CHART RULES Source: Lloyd Nelson, Technical Aids: The Shewhart Control Chart – Tests for Special Causes 56

57. Upper control limit Mean Lower control limit Rule # 1: Any single point outside the control limits

58. Upper control limit Mean Lower control limit Rule # 2: A shift At least 7 points consecutively either above or below the centre line

59. Upper control limit Mean Lower control limit Rule #3: A drift At least 7 points consecutively ascending or descending Note: the points can cross the centre line

60. Upper control limit Mean Lower control limit Rule #4: At least 14 points alternating up and down Note: the points do not have to alternate above and below the centre line

61. Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Other tests require you to plot the intermediate variation values

62. Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Rule #5: 2 out of 3 points outside 2V on the same side of the centre line

63. Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Rule #6: 4 out of 5 points outside 1V on the same side of the centre line

64. Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Rule #7: Reduced variation 15 points within 1V either side of the centre line

65. Mean Mean+1V Mean+2V Mean+3V Mean-1V Mean-2V Mean-3V Upper control limit Mean Lower control limit Rule #8: 2 processes 8 points in a row on both sides of the centre line with none within 1V

66. Why 7 points in a row? Toss a coin: Chances of getting a ‘head’? Toss it twice: Chances of getting two ‘heads’? 66

67. How many options? 67 1 toss2 tosses3 tosses4 tosses 1 in 21 in 41 in 81 in 16

68. Chances of ‘n’ heads in a row 68

69. SAFETY CROSSES AND RUN CHARTS 69

70. Your Safety Cross 70 Record the dates 04/07/2013 11/07/2013 25/07/2013

71. Calculate the days between falls 71 Date 04/07/2013 11/07/2013 25/07/2013 Days between 11 – 4 = 7 25 – 11 = 14 What value do we get if we have 2 falls on the same day?

72. Plot the days between on a run chart 72 Plot X axis as text not dates The higher the value, the better it is Use the 3 run chart rules to interpret

73. RECOMMENDED CHARTS 73

74. The Golden Rule when presenting data One picture, one message 74

75. Types of chart and when to use them Run & control charts Used to display performance over time Shows whether common causes or special causes (or both) are present in our data 75

76. Types of chart and when to use them Pareto charts Used to display the number of times distinct things happen such as reason for cancellation To separate the ‘vital few’ from the ‘useful many’ (the 80/20 rule) 76

77. Types of chart and when to use them Bar charts Used to display survey results Display for each question – which questions look different? Also display optionally for each respondent – are there types of respondent? 77 Source: Call of Duty

78. QUALITATIVE ANALYSIS

79. Surveys

80. Issues to consider Are all patients rating the same thing? Is there a ceiling effect? Do scales work in a linear fashion? –A score of 6 is twice as good as 3 –Strongly agree is twice as good as agree

81. Collecting patient experience data Interviews Patient tracking Surveys

82. Analysing qualitative data Thematic analysis: Look for the common themes Construct a story around typical findings The power of a good quote

83. The power of a quote Source: Call of Duty

84. Step 6 – Review measures It is a waste of time collecting and analysing your data if you don't take action on the results 84

85. DTOCs part 2 Verdict Stable within limits (10 - 72) Not Capable of achieving target

86. What decision do you make? Is your information presented in a way that allows you to confidently make one of these decisions? DecisionBecause Do nothingPerformance ok Contingency plansSpecial cause variation Process redesignCommon cause variation 86

87. Team Review Meetings

88. Step 7: Keep going 1 Decide aim 2 Choose measures 3 Define measures 6 Review measures 5 Analyse & present 7 Repeat steps 4-6 4 Collect data 7 Steps now on YouTube http://youtu.be/Za1o77jAnbw Or put ‘Mike Davidge measurement’ into YouTube search box 88 You may not get it right first time! You may need several attempts. Remember PDSA

89. References The run chart basic reference –“The run chart: a simple analytical tool for learning from variation in healthcare processes”; Perla R, Provost L, Murray S; BMJ Qual Saf 2011;20:46e51. doi:10.1136/bmjqs.2009.037895 A great introduction to variation and SPC –Understanding variation, Don Wheeler, www.spcpress.com, 1986 www.spcpress.com A couple of useful websites/blogs –www.davisdatasanity.comwww.davisdatasanity.com –www.kurtosis.co.ukwww.kurtosis.co.uk 89

90. Application to management Fourth Generation Management –Brian L Joiner, McGraw-Hill, 1994 The Leader’s Handbook –Peter R Scholtes, McGraw-Hill, 1998 The New Economics (2 nd edition) –W Edwards Deming, MIT Press, 1994

91. Issues and questions 91