1. Measurement for improvement Workshop
Whakakotahi Learning Session 1
Sue Wells
A/Prof Quality Improvement
University of Auckland

2. Clinician-led cycle of care improvement (Prof Ian Scott)

3. Clinician-led cycle of care improvement (Prof Ian Scott)
What are we trying to accomplish?
How will we know a change is an improvement?
How will we know a change is an improvement?
What changes can we make?

4. Clinician-led cycle of care improvement (Prof Ian Scott)

5. Components of a system
Structure
Process
Culture
Outcome

6. Categorising measures
Structure
Process
Outcome
Structure-what you need
premises, diagnostic equipment, staffing
Process- what you do
referral, investigations, communication, medications, discharge planning
Outcome- what you expect
mortality, disease state, symptom control,
residual disability, patient satisfaction
Balancing measure- check for unanticipated consequences, other factors influencing the system
Balancing measures- check for unanticipated consequences
Measures which allow us to make inferences about the quality of
health care
Qualitative information
Quantitative information 6

7. Why Measure? To understand current situation Move away from anecdote
To manage by fact- need to quantify and verify possible causes (need evidence)
To baseline current performance
Get everyone on the same page
To develop solutions – need data to determine most effective approach
To show impact of changes
To prompt need for further improvement
Understand what is going on
Understand interrelationships
Are we delivering what we said we would? Are we delivering what we think we are?
Time consuming (esp if manual)
No ownership
No training
Inaccurate / inconsistent
Results don’t match reality
Too many / inappropriate
Not used to take action
Threatening

8. Example Improving patient experience:
A patient requiring long term medication has complained about the difficulty about getting a prescription filled on the same day.
You ask… how easy it is for our patients to request and receive a repeat prescription?

9. Understand your current state
You need to measure
“In God we trust. All others bring data.” Deming

11. Gather and review relevant data
How many scripts/week requested?
What day do they generally come in?
Morning or afternoon?
What route do they come in?
Phone
Portal
Walk-in
Pharmacy

12. Gather feedback from patients and staff
© NHS Institute for Innovation and Improvement 2011

13. What are the major barriers and challenges to achieving same day service?

14. Map your current process for prescriptions
Simple visual for everyone – shared understanding of what process is for each team member- where the problems are and where the solutions might be
© NHS Institute for Innovation and Improvement 2011

15. Check Sheet: Delays in getting prescriptions in last week
Causes of delay in getting scripts filled
Problem
Tally
Subtotal
Doctor busy 39
Nurse busy 4
Receptionist busy 5
Patient loses script 3
Script gets lost by pharmacist
Script request gets lost in practice 13

16. Bar Chart from Check Sheet

17. Pareto Chart- prescription delays

18. Things to think about BEFORE you collect data
Your question/s- well defined
2) Stratification needed? separation of data according to strata or factors that might influence care patterns
Time of day or day of week
Season
Type of order (urgent vs routine)
Big or small practice
Type of worker
Patient factors eg; severity scores patient illness
3) Type of data
4) Sampling frame (e.g. PMS), sample size, sample strategy
For surveying what’s happening – may not be interested in outcomes – just what structures are in place or what processes are or are not occurring.
For investigating the impact of your intervention into a health service- will have outcome of interest

19. Understanding types of data
Continuous data- measured on a ‘continuous scale’
Discrete- categories
Discrete- count data
Qualitative
eg, time, volumes, temperatures, height
eg, presence/absence, ethnicity, ranking of preference scores, satisfaction scale
number of infections,
opinions, advice, experiences

20. Sampling in QI
NOT research…..QI is simply asking “what is happening here?”
no intention to generalise the results beyond the local setting
critical question is….
“does this sample represent the care/processes so that the team will accept results and act upon the findings?”

21. Minimum sample sizes (rule of thumb)
Estimating mean 5-10
Standard deviation 25
Proportion
Errors sample until 5 errors
Histogram/pareto chart 50
Scatter graph 25
Run/Control chart
Thornley Group Lean Six Sigma Training 2010

22. Simple sampling strategies
Block sampling – straight sequence in a single time frame eg consecutive patients first 2 weeks of this month
Random sampling – simple, stratified
Systematic or Purposive sampling- regular selection every 10th patient, every hour on the hour or set time of day, day of week
R Lloyd 2011

23. Variation
variation should be viewed in one of two ways special or common cause.
Common cause
Inherent part of every process
Random fluctuation
Stable, predictable or “in control”

24. Special Cause Variation
Indicates something has changed
Unstable or “out of control”
“systematic” change or shift from the usual process
Maybe a purposeful change
Or indicates something is not right

25. Run Charts & Control Charts plotting measurements of process or outcomes over time useful for understanding variation & demonstrating impact of interventions

26. Run charts Allows a team to study data over specific period of time
Performance of the process/system
Used to detect shifts, trends or cycles
Measure performance before and after an intervention
X axis always time
Y-axis- whatever process/measure
Simple rules for interpretation

27. How do I plot/interpret a Run Chart?
Median
Brassard and Ritter, 1994

28. Run chart exercise

29. Prescription process
Plot % of patients who get same day prescriptions:
Week Percent Week Percent
1 60% %
2 55% %
3 70% %
4 66% %
5 66% %
6 58% %
7 72% %
8 62% %
9 75% %

30. Fill in the template and join the dots

31. Do you get something like this?

32. Work out the median
Rewrite each data point (percentages) so that they are in rank order, going from highest to lowest (or lowest to highest)
Identify the median (middle) value (if an even number of values take the average of the 2 middle values)

33. Do you get something like this?
Percent (in rank order, descending) 75 74 72 71 70 66 64
Median 62 61 60 58 57 55

34. Draw the median line on the graph

35. Do you get something like this?

36. Before you can apply the 4 rules
Count the number of “useful observations”
Count the number of “runs”

37. Useful observations
All observations apart from any that fall on the median

38. How many useful observations?

39. How many useful observations did you get?
Two points on the line have same value as median
Useful observations = =16

40. Runs >1 data points on same side of median
Exclude data on the median

41. How many runs?

42. How many runs did you get?
Number of runs= 10

43. Four run rules
A special cause of variation may be signalled by meeting one of more of the following 4 run rules:
Shift in the process (too many data points in a run: >6 consecutive points above or below median)
Trend (>5 consecutive points all increasing or decreasing)
Too many or too few runs (use table to work out)
An “astronomical” data point (the interocular test)

44. Examples

45. Are any of the run chart rules met for our example?
Rule 1: Shift (>6 above/below median)?
Rule 2: Trend (>5 all increasing or decreasing)?
Rule 3: Too many or too few? (see table next slide – need to know number of useful observations [16] & number of runs [10])
Rule 4: Astronomical data point?

46. Rule 3: Too many or too few?
Number of useful observations
Minimum number of runs
Maximum number of runs 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 5 6 7 8 10 11 12 13 14

47. Clinician-led cycle of care improvement (Prof Ian Scott)

48. What was the impact of the changes?
Useful obs 32 weeks- 3 points on median =29 useful observations

49. What was the impact of the changes?
Useful obs 32 weeks- 3 points on median =29
= 12 runs

50. Four run rules
A special cause of variation may be signalled by meeting one of more of the following 4 run rules:
Shift in the process (too many data points in a run: >6 consecutive points above or below median)
Trend (>5 consecutive points all increasing or decreasing)
Too many or too few runs (use table to work out)
An “astronomical” data point (the interocular test)

51. Rule 3: Too many or too few?
Number of useful observations
Minimum number of runs
Maximum number of runs 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 5 6 7 8 10 11 12 13 14

52. Control charts

53. Control charts simplified
Run charts and control charts same purpose
Central line mean (run chart median)
Limits (UCL and LCL) provide additional tests to identify special cause
More statistically robust
Better able to identify special cause variation (more sensitive)
Type of control chart depends on the type of data (many types vs only one type of run chart)
Control chart rules different
Purpose to distinguish common cause from special cause variation in the data produced by a process

54. Control chart type determined by data type
“Variables” (continuous) data
Quantitative data that can be measured
Infinite number of possible values depending
on the precision of measurement
Does not have to be collected as a whole number
E.g. weight, height, BP, volume of workload
eg, time, volumes, temperatures, height
eg, presence/absence, ethnicity, ranking of preference scores, satisfaction scale
number of infections,
opinions, advice, experiences
“Attribute” (non-continuous) data
Count (“non-conformities”, “defects”)
Classification (“nonconforming units”, “defectives”)
Occurrences only (don’t count those that don’t occur)
Numerator only
E.g. falls, medication errors, CLAB infections
Can count occurrences and non-occurrences
Numerator and denominator
E.g. % mortality,
% readmitted, % c-sections

55. Which control chart should I use?
Continuous?
Yes No
1 observation per subgroup? (usually: is there 1 measurement per time period?)
Classification? No
Yes No
Do you need to convert to a rate?
Yes No
Yes I
X and S P C U

56. Key points about measurement
Purpose:  learning; x judgement
Be aware of the limitations of the measure
Balanced set: process, outcome, balance
Plot over time
Report regularly (has the process improved, stayed the same or become worse?)
Measures should be:
linked to the team’s aim
Used to guide improvement and test changes
Integrated into the team’s daily routine
Focus on the vital few (Pareto principle / 80:20 rule)