1. Scottish Improvement Skills
Analysing data:
Interpretation of run charts
SIS: Group C Measurement: Module – Interpretation of Run charts: Facilitator

2. System of Profound Knowledge
Analysing data we are mainly in Understanding Variation (also the other components in so far as we are using that understanding of variation to appreciate the system, develop theory, and communicate about data to effectively involve relevant stakeholders).
We’re going to be working with run charts.
Elicit: what is the main purpose of displaying our data in run charts?
To identify any signals of non-random variation in our system, to help us understand what factors are associated with improved processes and outcomes. They help us to find and then evaluate causes of non-random variation.
Deming 2000

3. Analysing data: interpretation of run charts
By the end of this session you will be able to:
Explain the importance of using annotation to help interpret run charts
Discuss how the type of median used affects interpretation of signals from run charts and impacts on decision making
Explain use of phasing, stratification, and what to do with extreme values on run charts.
Learning outcomes

4. Analysing data: Interpretation of run charts
Annotation
Median
Phasing
Extreme values
Stratification
DISCOVERY
Timing
2 mins for introductory slides
Lead facilitator
What we are going to look at in this session:
What a run chart can help us understand about the system that we are working in, and how we can best use a run chart to do this.
Annotation – what kind of notes we need to add to a chart
Median – we can create the median in different ways – we’ll look at how we decide which kind of median to use
Phasing – this is also about making the best use of medians
Extreme values – what action we may need to take if we have a lot of extreme values eg 0% or 100%, and why
Stratification – this is about how we break down our data – something we need to consider when developing our measures and operational definitions
Info for Facilitator: Quote from Sandy Murray (IHI):
Stratification is the separation and classification of data according to selected variables or factors.
The object is to find patterns that help in understanding the causal mechanisms at work

5. Aim: promote staff wellbeing
Aim  Driver  Driver Change ideas
Offer Bereavement support programme for managers
Promote the physical, mental and emotional wellbeing of staff. Within 12 months (1) reduce staff absences from 5.2% to 4.2% (2) reduce staff related incidents from 140 to 100 per month.
A workplace that is safe for staff
Flyer on staff health and wellbeing programmes
Staff education
Hold staff health fairs
Smoking cessation campaign
Staff engaged in health and wellbeing practices
Set up walking groups
Healthy lifestyle programmes for staff
We are going to look at some examples of each of these features of run charts.
Most of the following charts (but not all) relate to this scenario
Staff wellness driver diagram – the run chart examples display data relating to the aim ‘reduce staff absences’
Improve staff diet at work
Offer lunchtime yoga
Enable cycling to/from work

6. Annotation Hours lost through sickness absence
Median 5.2
Fit Note introduced
Aim
Participants use the annotations in a run chart to better interpret the data
Participants annotate their own run charts with key information that will support data analysis.
Participants keep their run charts uncluttered.
Key messages:
Annotation must include: median value, changes introduced as part of an improvement project, any changes in the external or internal environment that may have an influence on your data
Annotation should include: an indication of what ‘good’ means eg by signalling ‘goal’, or less specifically indicating whether up or down is good (using an arrow in that direction)
Timing
(for Annotation, Medians, and Phasing)
10 – 15 minutes (some participants find this a particularly challenging part of the programme, so allow plenty of time)
Lead Facilitator
For this ‘improving staff wellbeing’ project, is this measure (% hours lost through sickness absence) outcome, process, or balancing? (outcome)
Always annotate with any information of relevance to your improvement project.
This must include start (and end) of any changes.
And may include other features that will help you and others to interpret the chart.
Eg here, something in the external environment that may have had an impact on your data.
Some people annotate the Goal too. If you don’t indicate the goal, it is useful to indicate whether good is up or down.
Visually – good practice is to keep the data as uncluttered as possible, so put the annotation at the bottom linked to the date, rather than attached to the data point itself, unless the chart doesn’t have space near the X axis for this.
Goal

7. Median (1) Women walking 10,000 steps
Baseline median Extended median
Pedometers issued
Challenge announced
Aim
Participants can select the appropriate median for their data at different stages in their project
Participants can explain the reasons for selection of a given type of median.
Key messages
Baseline median + extended median should be the standard, with an ‘improper’ median only used when there are not enough data points for any other type (eg when gathering baseline data).
Lead facilitator
This is the type of median that we’re using most of the time.
Baseline median – refers to the section used to calculate the median.
Extended median – refers to the extension of this line – the data points around it were not used to create the median. This is indicated using a dotted line.
Here, there could be no baseline without the pedometers, so the pedometers were issued before announcing the 10,000 steps challenge. However, simply issuing the pedometers may have had an impact on the number of steps people walked during the baseline period.
Why is it important to use baseline + extended medians? Because it affects the decisions you make based on the data and application of the rules.
Compare this with the same data using an ‘improper median’ (see next slide)

8. Median (2) Women walking 10,000 steps
Signal detected here
Lead facilitator
Improper median: all data points are used to calculate the median.
Applying the four run chart rules each time we add a data point, we would detect a signal of non-random variation on the final Monday.
Elicit: which rule is this signal associated with? (shift – 6 consecutive data points all above the median)
NB the data point for the Thursday before the signal is detected is on the median, so doesn’t break the run/shift, but is not counted.
Pedometers issued
Challenge announced

9. Median (1) Women walking 10,000 steps
Baseline median Extended median
Signal detected here
Pedometers issued
Challenge announced
Lead facilitator
So, by correctly using the extended median, you detect signals of non-random variation sooner, and can act on this.
For example, here you might want to communicate the improvement (which itself may encourage the behaviour that you want), scale up your next test of change, or introduce a new change. The ‘challenge’ to each member of staff to walk 10,000 steps has had some impact; what could we do to increase the % further? (Elicit some ideas)
Depending on your project changes and measures, this may make a significant difference eg if you can scale up your tests of change sooner, then more of your service users (whether internal or external) will get the benefit sooner.

10. Creating a baseline median
Use historical data if available
Collect new baseline data before introducing a change, if this makes sense in your context
If no baseline data is available, create a baseline median using the first 10 data points.
Key messages:
Having no baseline data, or only a little, need not put off introducing changes.
How you use baseline data, how much you need etc depends on the context and the nature of the improvement goal.
Lead facilitator
If baseline data is available, it allows you to identify whether the variation in your system is all random before you introduce your first change, or whether there is some non-random variation. If you identify any signals of non-random variation, you know this is not related to your change and can investigate and take action.
Normally you need 10 or more data points to create a median.
In most contexts, it’s UNlikely that you will get a significant change in your data immediately after introducing a change into your system (particularly for an outcome measure). So, if there is an urgent need to make a change and you don’t have 10 data points of baseline data, you can use the first 2 or 3 data points to create the baseline median.

11. Phasing Women walking 10,000 steps
Pedometers issued
Challenge announced
Looking at the same measure:
Having detected a signal of change (a shift), once you are confident that it’s sustained you may decide to use that as a new ‘baseline’, from which you introduce a new change.
So, first we used the extended median to detect whether there was a signal of special cause variation.
Then, you apply the same rules in relation to the new ‘phased’ median. You need to wait until you have enough data points to do this.
So, applying the rules to the new median (here the orange median line):
Rule 1 – shift – no
Rule 2 – trend – no
Rule 3 – too few or too many runs – 11 useful data points, 9 runs – no
Rule 4 – astronomical point – no
Random variation – these fluctuations are due to chance. So, to improve further, you would need to introduce a new change.
From the point where the new change is introduced and new data is collected, create an extended median from this orange ‘baseline’, until there is a signal of non-random variation, then consider phasing the median again.
The same rules apply around each median (NB a baseline + extended median is a single median so far as the rules are concerned).
Depending how participants are getting on with this, it may be useful to add:
If you have a high degree of belief, you may decide to introduce a new change, or move to implementation, before you can apply the rules to the phased median.

12. Interpretation of run charts
Each run chart can be improved to make the data easier to interpret. Possible improvements may relate to:
Type of median line used
Measure used
For each chart, decide what improvements would be most useful, and why.
Annotation
Design
PRACTICE
Timing
10 minutes
Materials
Interpretation of run charts
Run chart checklist
Lead facilitator
Work individually first, then compare notes in pairs or small groups.
Facilitators
If there are enough facilitators, have one at each table of participants to support discussions. Otherwise, monitor around the room, respond to any participant queries, and intervene as required.
No plenary debrief here – do this after the next section on extreme values and stratification.
NB the Discovery stage is split in two because if all done at once before the Practice, it tends to be a long time for participants to focus on slides and the Lead facilitator.

13. Extreme values Frequent events Rare events
More than half the data falls on the median line
More than half the data is at ‘extreme’ values eg 0 or 100 on percentage scale
DISCOVERY
Aim
Participants can correctly interpret data in charts with frequent or rare events or when more than half the data falls on the median line.
Participants consider the option of changing the measure in order to get more useful data.
Key messages (for this series of slides)
Extreme values: the median cannot be applied because it will be the extreme value itself; so, the 4 rules cannot be used.
These may be a signal that a change has been sustained, so it may be time to consider stopping collecting data
Or, it may be necessary to change the measure used.
Data on median line – the rules cannot be applied – again, it may be necessary to change the measure used.
Timing (extreme values and stratification)
10 – 15 minutes

14. Extreme values (1) Compliance with hand hygiene
Lead facilitator
Elicit – would you create a new median, and if so, when? How would it help?
No point creating a new median, because it would be 100%
If someone queries this, demonstrate how this would work: to calculate the median using the data points in this shift (just using the first 9 data points as an example), they would be 99, 99, 99, 100, 100, 100, 100, 100, 100
- the middle number of these is 100

15. Extreme values (2a): Number of needlestick incidents per month
Lead facilitator
This slide relates to the other half of our staff wellbeing project – a workplace that is safe for staff.
See the median at the bottom (in blue – NB facilitators – the median shows up on slideshow, not on normal view)
In this case, it is a rare event, so there are a lot of zeros.
Still, even though it’s a rare event, it needs to be improved – we want to reduce the frequency even further, so we use a different measure – see next chart.
A needlestick incident is a penetrating wound from a needle or sharp object that may result in exposure to blood or other bodily fluids.

16. Extreme values (2b): Number of patients seen between needlestick incidents
Number of cases between events. Using this measure you can apply the usual rules.
Here we have a different x (horizontal) axis from usual – here instead of dates we have a series of events, in sequence. This is still a time series.
And as usual the vertical axis is the measure.
Elicit: here, do we want the data to go up or down? (up – in the previous ‘incidents per month’ chart we wanted the data to go down)
The measure here means: the number of patients seen in the clinic between incidents. Ie, from the time when one member of staff experiences a needlestick incident from the time another member of staff experiences a needlestick incident, how many patient/staff interactions are there?

17. Stratification (1): Inpatient requests for MRI
Aim
Participants look for opportunities to stratify their data while developing their measurement plan, based on their appreciation for a system and theory of knowledge.
When interpreting a run chart, participants look for patterns that may indicate a need for stratification.
Lead facilitator
Different scenario – this slide is not related to the same project.
Here there are too many runs – according to Rule 3, this is a signal of non-randomness.
But visually you can see the variation is consistent across this whole time period.
This tells us more than just whether or not there is non-random variation. It may not relate to change or improvement. It may be a feature of what data is being collected and how.
For example:
Elicit possible problem and solution in this scenario. Bring up the next slides when appropriate.

18. Stratification (1): Inpatient requests for MRI
The different colours help identify two distinct sets of data.
These should be collected and analysed separately.
This is the kind of decision that should be made when planning data collection, to avoid the problem of having data that is not useful. If you’ve done that, you won’t ever come across a chart that looks like this. But from time to time you do.
Thinking about how you will label your x axis (usually time, or eg patient sequence), should help prevent this problem occurring.

19. Stratification (1): Inpatient requests for MRI
This shows even more clearly the two distinct sets of data

20. Stratification (2) Staff walking 10,000 steps (men and women)
Median 35.25
This is another example of the need for stratification based on the 10,000 steps measure we looked at earlier.
Here the data for men and women that we looked at earlier has been combined.
What do the run chart rules tell you?
This chart signals a shift, and you would plan action on that basis.
However, if we think about how we might stratify this data, we can see that this chart is disguising more useful signals. (See next 2 slides.)
Challenge announced
Pedometers issued

21. Stratification (2) Staff walking 10,000 steps (women)
Median 29
Pedometers issued
Challenge announced
For women there’s a clear shift.

22. Stratification (2) Staff walking 10,000 steps (men)
Median 41
The extent of the change among women obscured the men’s data, which shows random variation ie no significant improvement. The median is higher, but it was higher to start with – the challenge did not change anything for men.
The evidence base clearly demonstrates an existing difference in steps per day for men and women, so they have a different starting point. From these charts we can see that to increase steps per day we may need different strategies to target men or women.
Key messages
Use the evidence base to establish measures and any stratification needed if possible in advance of collecting data.
And, look out for the possible need for stratification in data once you’ve put it in a chart.
Pedometers issued
Challenge announced

23. Interpretation of run charts
Each run chart can be improved to make the data easier to interpret. Possible improvements may relate to:
Type of median line used
Measure used
For each chart, decide what improvements would be most useful, and why.
Annotation
Design
PRACTICE
Timing
10 – 15 minutes
Materials
Interpretation of run chart (already used above)
Run chart checklist (already used above)
Facilitators
Return to the same run charts. Discuss re extreme values and stratification.
If enough facilitators, one at each table to support discussions. The Lead facilitator only a short plenary debrief using the following slides.
If the number of facilitators does not allow this, monitor around the room, respond to any participant queries, and intervene as required. Then Plenary debrief will be longer.
For each of the three charts, elicit from participants their improvements based on all the features addressed in this module.
The slides give some possible improvements; accept other responses as appropriate.

24. Interpretation of run charts (1) Complaints dealt with in 20 days
Complaints admin on holiday
Quiet workstations introduced
KEY
What is the goal? It would be helpful to indicate it to help decide what to do if/when the shift is sustained on the basis of this single change.
Depending on goal, consider changing range on Y axis.
Annotate median
Complaints ‘dealt with’ – need a clear operational definition, but no space to include here in either title or measure name.
See next slide for proposed solution.

25. Interpretation of run charts (1) Complaints dealt with in 20 days
Goal
Median 62
Quiet workstations introduced
Complaints admin on holiday
KEY
What is the goal? It would be helpful to indicate it to help decide what to do if/when the shift is sustained on the basis of this single change.
Depending on goal, consider changing range on Y axis.
Annotate median
Complaints ‘dealt with’ – need a clear operational definition, but no space to include here in either title or measure name.

26. Interpretation of run charts (2) Outpatients cost per attendance
KEY
Needs annotations – have any changes been introduced? Has anything else that may impact on this measure happened over this period?
Fuller dates needed – year as well as month
What type of median is it? Is this appropriate? To decide, we need more information, eg when changes were introduced. If this is all baseline, it’s fine to use an ‘improper’ median – this helps us to understand whether our current system is stable before we start introducing changes. But if we have made changes, a baseline and extended median will be more useful.
Continue X axis for a couple of points to demonstrate that you are planning to continue collecting data. Otherwise it looks as if data is no longer going to be collected for this measure.
Which way do we want cost to go? Indicate this either with a ‘goal’ annotation, or an arrow going up or down. Remember that up is sometimes good and sometimes bad. And you don’t always get the result you’re aiming for. Here we have a shift that seems to have been sustained over 8 months – is this good or bad?
Design – eliminate distractions eg remove colour as much as possible – a little colour can be used to highlight; remove gridlines
See next slide for proposed solution.

27. Interpretation of run charts (2) Outpatients cost per attendance
New referral process introduced
KEY
Needs annotations – have any changes been introduced? Has anything else that may impact on this measure happened over this period?
Fuller dates needed – year as well as month
What type of median is it? Is this appropriate? To decide, we need more information, eg when changes were introduced. If this is all baseline, it’s fine to use an ‘improper’ median – this helps us to understand whether our current system is stable before we start introducing changes. But if we have made changes, a baseline and extended median may be more useful.
Continue X axis for a couple of points to demonstrate that you are planning to continue collecting data. Otherwise it looks as if this is fixed.
Which way do we want cost to go? Indicate this either with a ‘goal’ annotation, or an arrow going up or down. Remember that up is sometimes good and sometimes bad. And you don’t always get the result you’re aiming for. Here we have a sustained shift – is this good or bad?
Design – eliminate distractions eg remove colour as much as possible – a little colour can be used to highlight; remove gridlines

28. Interpretation of run charts (3) Ventilator Associated Pneumonia (VAP) rate
Chlorhexidine oral gel introduced
Closure of neuro ICU
‘daily goals’ introduced
KEY (no improved version to follow – too many options and need too much information).
Needs other types of median. Depends on what counts as baseline, which could be up to Closure of ICU, or could be up to ‘daily goals’ - need local knowledge to decide. That would also affect when you would start phasing.
Phasing would help to identify whether you have a rare event, and would benefit from using a different measure. A clear signal of a rare event is the median being zero – if you don’t change how you calculate your median to include phasing, this won’t stand out as clearly.
Design – for clarity, preferably change colour of run line to black; remove decimal points from Y axis; add median annotation; possibly add more annotation of changes and other potentially relevant factors.
Needs title

29. Analysing data: Interpretation of run charts: summary
Annotation
Median
Phasing
Extreme values
Stratification
Aim
To briefly recap the session content:
To support a sense of learning and accomplishment
To aid memory of the session later
An opportunity for participants to ask any outstanding questions from any part of the session
Timing
2 – 5 minutes, depending on time available
Lead facilitator
Elicit key messages relating to each of the bullets eg
What annotations should you include in every run chart?
When would you expect to use an improper median?
How does a baseline+extended median normally help us with decision making?
When would you decide to use a phased median?
What are the two main options to consider if you data includes a lot of extreme values?
When should you start to consider stratifying your data?

30. References and further resources
Provost Lloyd P & Murray S (2011)
The Health Care Data Guide: Learning from Data for Improvement
Jossey-Bass
There are more resources (online learning modules, videos) to help participants with run charts on the Workplace Learning document, issued at the end of Workshop 1 of the full Scottish Improvement Skills programme.