1. Scottish Improvement Skills
Data analysis:
Introduction to run charts
SIS: Group C Measurement: Module – Introduction to Run charts: Facilitator

2. System of Profound Knowledge
In this module we are looking at what we do with data once we’ve collected it.
This is all about Understanding Variation
Deming 2000

3. Data analysis By the end of this session you will be able to:
Explain why it is important to measure data over time
Interpret run charts using rules to differentiate between random and non-random variation
Use run charts to explain outcomes of improvement work to others
Use and explain the importance of using a family of measures.
Learning outcomes – Facilitator read out or participants read.

4. Understanding Variation
Random variation – affects everyone and all outcomes over time
Non-random variation – does not affect everyone or not part of the system all the time; arises because of specific circumstances.
Lead facilitator
Here’s a reminder of what we mean by variation.
When we analyse our data, we want to find out if variation in the data is random or non-random.
And if non-random, is this because of a change we’ve introduced, or for another reason?

5. Analysing data: before and after
‘When you have two data points, it is very likely that one will be different from the other.’
W Edwards Deming
DISCOVERY
Aim
Participants can explain why it’s important to analyse data in time series.
Key messages
To understand what’s going on in our system, we need our data to tell a story.
Timing
5 minutes
Lead facilitator
Imagine this is Vanessa’s data, for her personal project ‘new healthier me’.
Here her data is in a ‘before and after’ chart.
Imagine she weighed herself once a week for a few weeks before making a change (either to her calories in or calories out), and averaged the data.
Then she averaged the data over a period of several weeks after making the change.
Elicit: Why is this not very helpful?
A: It doesn’t tell us anything about the process. It’s a snapshot, and doesn’t tell us the full story behind that snapshot.
Click: As Deming says … (see quote)
We get more useful information by using a time series chart. See next slides.

6. Data analysis: Introduction to run charts
Weight (lbs)
Lead facilitator
Talk through the charts, going clockwise from top right. Elicit as much of this as possible.
Chart top right
Imagine you are helping a friend or family member to lose weight – this is their data. How would you explain it to them?
Here weight is going down, but it was going down anyway – it is not because of the change we introduced into the system.
We need to find out what is causing this.
If we made a before and after bar chart using this data, would look like this one.
Chart bottom right
Here weight did seem to go down associated with our change, but then it went up again.
Again, bar chart would be the same.
Chart bottom left
What story is this chart telling us?
Here the data is showing us non-random variation: we have a New Healthier Me.
If we plotted any of these time series charts as before and after charts, they would look like the one above. So we wouldn’t know which of these stories is associated with this data – we wouldn’t know whether or not the change made had resulted in an improvement.
There are many other scenarios that would produce that same before and after chart.
So, when measuring for improvement we most often use a line chart with data in time series.
But these charts are not run charts. To really understand our data we need to add another feature to this chart [move on to next slide].

7. Data tells a story: New healthier me!
Aim
Participants can distinguish a run chart from a line chart (run chart must have a median line).
Participants can recognise the key features of a run.
Lead facilitator
A time series chart like the ones in the previous slide is helpful, but to make it even more helpful, and easier to analyse to identify whether we have any non-random variation, we apply 4 rules. These rules are based on the distribution of data points around the median.
What is a median? (elicit)
By applying 4 rules we can find out if any variation in the data is random – also sometimes called common cause variation – or whether it is associated with specific circumstances (we hope, the change that we introduced) – also sometimes called special cause variation.
[NB – try to avoid special/common cause, as these terms refer to Control charts, not run charts. But explain the terminology if a query comes up.]
What is a run chart? Features – point out on the slide
A run chart is a line graph with a measure (vertical or y-axis) plotted over time (horizontal or x-axis).
Each time data is collected, a new data point is added to the chart.
The time steps depend on how often you are collecting data. For example this may be daily or hourly if measuring wait time for a particular service, but weekly, monthly or quarterly if measuring patient satisfaction.
As well as the line of data points, a run chart also includes a median line. The median is the middle value of the data points plotted, so that half are above and half are below the median line.
At this stage it is not necessary to introduce the concept of baseline and extended medians. They are used in this module to model good practice, so that participants are exposed immediately to the median that we want them to use. The module Analysing data, interpretation of run charts introduces different types of median, and Visual display of data: using Excel to create run charts goes through how to create them.
If a participant asks about the two different blue lines, or points out that in this chart there are not ‘half above and half below’ the median line, just explain that the median here was created using the data points up to and including 25 Feb.

8. Introduction to run charts
A ‘time series’ chart tells a story.
Baseline data helps us to see whether a change is an improvement.
Any changes made are shown on the chart.
Summarise some key points from the previous example.
As you could see from these charts:
We need a time series chart – to tell a story
We need baseline data – data collected before the change.
We need to know on the chart when a change was made (and what that change involved)
We also need to indicate anything else we are aware of that happened at a particular time that may have influenced the data.
Before we look at the 4 rules, let’s see why it’s called a run chart.

9. What is a run? Vanessa’s Weight
Aim
Participants can correctly count the number of runs in a run chart
Key messages
A run is a series of data points all on the same side of the median line.
Either all above the median, or all below the median.
The minimum number of data points in a run is 1.
Data points on the median do not break a run, but are not counted as part of a run.
Timing
15 minutes for all ‘counting runs’ section
Material
How many runs?
Lead facilitator
In advance of the session decide which of the two ways of counting runs you prefer to highlight. Try to use only one method in the session. Depending how participants get on with this, it may be necessary to introduce the second approach.
Option A: This slide series is based on circling data points on the same side of the median line.
Option B: Count the number of times the run line (the black line) crosses the median, and add 1. If using this approach, it would be better to remove the red circles from the following slides, and replace them with circles around the points where the run line crosses the median line. Participants often ask ‘why add 1?’: imagine you have an unsliced loaf of bread and you cut through it once ie you have one slice + the rest of the loaf. The cut is the equivalent of ‘crossing the median’. If you cut twice, you get two slices + the rest of the loaf.
Talk through key messages.
Q: how many runs are there in this chart?
Allow a couple of minutes for participants to think through this alone and/or help each other in pairs/small groups.
Click three times to bring up circles round the three runs in this chart.
When you are analysing data in a run chart, it’s important always to be clear from the start whether ‘good’ is going up or going down.
PRACTICE
This slide is for Vanessa’s outcome measure: weight.
The worksheet has three run charts, one for each of the change ideas that she tried.
Count the runs – start doing this individually, then compare answers in pairs.
Debrief using the following slides.

10. How many runs (1) ?
Debrief: show the run chart, elicit the number of runs, then show the next slide which has the answer.

11. How many runs (1) ? Four runs.
4 data points on the median line – these are not ‘useful’ points.
If a query comes up about what ‘useful’ means, (try to avoid this until introducing the run chart rules):
There are rules that we use to help us understand whether the variation in our chart is random or non-random.
Three of these rules are based on the statistical probability of data points being distributed in particular ways around the median.
Because these data points are on the median line, they cannot be considered to be distributed on one side or the other of the median, so they do not contribute to application of these rules.

12. How many runs (2) ?

13. How many runs (2) ? Five runs.
No data points on the median these are all useful data points

14. How many runs (3) ?

15. How many runs (3) ?
Six runs.

16. Run charts: signals that identify non-random variation
Six or more data points in a run (all above or all below median)
Five or more consecutive data points all increasing or decreasing
Too many or too few runs
An ‘astronomical’ data point
A shift
A trend
See table
Consider
DISCOVERY
Aim
Participants can identify whether there is any non-random variation by applying the four run chart rules.
Key messages
We apply these rules to find out if there are any signals of non-random variation.
There is a signal of non-random change if any one of the four rules occurs. If more than one rule occurs, this strengthens the signal.
All four rules should be applied to a run chart.
The signal provides evidence of improvement if the change is in the desired direction. When a signal is identified, the improvement team should investigate to understand what caused the signal.
Timing
25 minutes
Materials
Run chart rules
Run charts: apply the rules
Lead facilitator
Don’t talk through the rules at this point – this slide provides a summary for reference later.
The first three rules are based on the statistical probability of data points being distributed in particular ways.   

17. Run charts: Rule 1 – a shift
6 or more consecutive data points either all above or all below the median line.
Data points that fall on the median do not break a shift.
A shift is always a run, but a run is not necessarily a shift.
A shift is a run with at least 6 data points. A run is not a shift if it has 5 data points or fewer.

18. Run charts: Rule 2 – a trend
5 or more consecutive data points all going up or all going down.
If two or more consecutive data points have the same value, only count one of them.
A trend can cross the median line.
This chart includes two sections that our everyday use of the word might be considered as ‘trends’, but they are not:
Feb – July 2012 – in this series of data points, some of them are lower than the one before
Oct 2012 – Mar 2013 – this includes two ‘pairs’ of data points, where 2 data points have the same value, so in this series of data points we only count 4, not six, from top to bottom.

19. Run charts: Rule 3 (a) Too few or too many runs
Rule 3: too many or too few runs
For this rule you need to use a data table.
Calculate the number of useful data points.
Find this number in the first column of the table.
Count the number of runs.
If the number of runs is below the lower limit or above the upper limit in relation to the number of useful data points, the chart is signalling non-random change.
This chart has 16 useful data points, and 4 runs
See next slide.

20. Run charts: Rule 3 (b) Total useful data points Total data points
For the chart below:
Useful observations
Lower number of runs
Upper number of runs 15 5 12 16 13 17 18 6 14 19 20
Calculate the number of ‘useful’ observations (subtract the number of data points on the median from the total).
Find this number in the first column.
Count the number of runs.
If this number is below the lower limit or above the upper limit, the chart is demonstrating special cause. 21 7 22 23 24 8 25 26 9 27 10 28 29 30 11
Total useful data points
Total data points
For 16 useful data points, the lower limit for random variation is 5 runs, and the upper limit for random variation is 13 runs.
The chart has 4 runs, which is below the lower limit. This signals non-random variation.
Because this rule is based on runs, and a shift is a type of run, Rule 3 and Rule 1 (shift) often occur together. When they do occur together, this provides a stronger signal of non-random variation.

21. Run charts: Rule 4 Rule 4: an astronomical data point
Unlike the other rules, this one is subjective, not based on statistical probability.
This is a data point that is an obviously different value from the ones around it. Anyone analysing the chart would agree that it is very unusual.
To identify an astronomical data point, the chart must include data points both before and after the data point in question.
Astronomical data points are often an indicator of a person-dependent process. For example if a staff member is absent, or a new member of staff has not been adequately briefed on the process.

22. Applying the rules Change PRACTICE Material
Run charts: apply the rules
Lead facilitator
Now you are going to apply the four rules to find out if there are any signals of non-random variation.
The charts we are looking at are from the Porter Productivity case study.
Example in plenary to prepare for the following task.
This chart is for the outcome measure: Porter Productivity.
Elicit whether this meets each of the 4 rules in turn, then – is there evidence of non-random variation?
Rules:
1 – yes
2 – no
3 – no (18 useful data points, 7 runs – this is between the upper and lower limits)
4 – no
Click to show circle around the shift.
Now work on the three charts – process measures.
Participants work individually, then compare notes in pairs or small groups
Facilitators
It is likely that participants will have questions, so after they’ve had a few minutes to get started, be available to support individuals or groups.
Debrief using the following slides.

23. Applying the rules (1) Change
Debrief: elicit whether there is a signal of non random variation for each rule.
1 – no
2 – no
3 – no (8 runs, 15 useful data points – assuming that 3 data points are on the median line)
4 - no

24. Applying the rules (2) Change 1 – no 2 – no 3 – no 4 – no
Some people might suggest that is an astronomical point. It isn’t really, given that the data points above the median go nearly as far up as this goes down.
However, it still may be worth investigating. In this case, do you want the data to go up or down? Presumably down, so it may be helpful to know why the number of jobs cancelled was lower on that date.
Some people may think there’s a trend – there are several ‘almost’ trends here, but either they don’t have enough data points (only 4), or the consecutive data points going in one direction are interrupted (on ), by one data point going in the other direction.

25. Applying the rules (3) Change 1 – yes (click to bring up red circle)
2 – no
3 – no (7 runs, 18 data points – this is between the upper and lower limits)
4 – no

26. Applying the rules (4)
Looking at all these together, we can see the ‘basket of measures’ or ‘family of measures’, with 3 process measures and 1 outcome measure – the outcome measure is titled in blue.
Given what we learned by applying the rules to the charts, which is the process that is having an impact on the outcome?
(requests providing all required information) – next click highlights this with a circle.
There is one signal of non-random variation, so we would want to find out more about this. How were we able to get a higher percentage of requests providing all the required information? What might we need to do next to ensure this process is followed more systematically, that it becomes standard practice?
The run chart rules give us information about our variation. That is not where things end. It’s the beginning of understanding our system better, and will help us to embed practices that give us the outcomes we are aiming for.

27. Baseline data How urgent is a change?
Is it necessary to identify whether the system has any non-random variation before introducing a change?
What is the source of historical data?
If there is existing data, make use of it.
If there is no existing data, decide whether to start collecting data before introducing the change.
DISCOVERY
Aim
Participants can name criteria for deciding whether or not baseline data is required before introducing a change into the system.
Key messages
Make use of existing or easily collected baseline data
Importance of having baseline data to make sense of data after introducing a change.
Timing
10 minutes
Lead facilitator
Why do we need baseline data?
- To see if a change is an improvement – for comparison
- To find out whether there is any existing non-random variation in the system, and to help us decide if we need to do anything about that
What obstacles might there be to getting it?
A gatekeeper – finding out the source of the data and getting access to it
The urgency of a change – we may need to make a change urgently, and can’t wait until we’ve collected baseline data
Can’t make sense of new data after a change without having some data before the change.
But in most contexts any change is unlikely to have an immediate impact, so you can often use the first few data points as your baseline. If available, do use baseline data.

28. Project work: baseline data
Does baseline data exist somewhere? If so, how can you access it?
If you are going to collect it, how long will you collect baseline data for before introducing a change? Why?
PROJECT
Participants spend a few minutes thinking about baseline data in relation to their own project.

29. Data analysis: Introduction to run charts: summary
Data tells a story
Look for signals of non-random variation
Rules:
Shift
Trend
Too few or too many runs
Astronomical point
Baseline data
Aim
To briefly recap the session content:
To support a sense of learning and accomplishment
To aid memory of the session later
An opportunity for participants to ask any outstanding questions from any part of the session
Timing
2 – 5 minutes, depending on time available
Lead facilitator
Elicit key messages relating to each of the bullets eg
Why doesn’t ‘before and after’ data help us to understand our system?
What is the minimum number of data points in a shift? Can they cross the median line? (6, no)
What is the minimum number of data points in a trend? Can they cross the median line? (5, yes)
Too few or too many runs for what? (to be a system with only random variation)
What’s the difference between Rule 4 and the other rules? (subjective)
Why is it important to have baseline data?
If you don’t have any baseline data, do you have to wait until you have collected some before you can introduce a change? (no – depends on context and the urgency of improvement)

30. References and further resources
Provost Lloyd P & Murray S (2011)
The Health Care Data Guide: Learning from Data for Improvement
Jossey-Bass
Point out that this book includes the table for Rule 3 – a much bigger one with fewer and more data points.
There are more resources (online learning modules, videos) to help participants with run charts on the Workplace Learning document, issued at the end of Workshop 1 of the full Scottish Improvement Skills programme.