1. Scottish Improvement Skills
How will we know that a change is an improvement?
Introduction to Measurement
SIS: Group C Measurement: Module – Introduction to measurement: Facilitator
Aim
Participants to be able to draft three types of measure for their own project (Outcome, Process, Balancing), and explain why these are all needed
Key messages
Measurement for improvement is different from measurement for research or accountability/audit
Measurement allows us to know whether a change we introduce is an improvement
We need multiple measures: Outcome, Process, Balancing
Timing
60 mins for the whole module

2. System of Profound Knowledge
Understanding variation
Elicit/recap what this is about: identifying random vs non-random variation, and understanding the cause of any non-random variation.
Deming 2000

3. Introduction to measurement
By the end of this session you will be able to:
describe why data is needed to support improvement work
describe 3 types of measure and explain why it is important to use all of them.
Learning outcomes: Read out, or ask participants to read.
This is a brief introductory overview, with much more on measurement in future modules.

4. Three questions What are we trying to accomplish?
How will we know that a change is an improvement?
What change can we make that will result in improvement?
In the Model for Improvement, we are now looking at this question.

5. Measurement: why? Safe Effective Person- centred
To make improvement visible
To plan
To monitor progress
To tell an improvement story
To use a shared language.
Safe
Effective
Person-
centred
DISCOVERY
Aim
Participants to be able to describe why we need data to support improvement work.
Key messages
We measure so that we can know whether a change we have introduced into our system is an improvement.
Measurement is not an end in itself – we should spend as little time as possible dealing with measurement, but enough time to get the information
Timing
10 mins
Lead facilitator
On heading only:
Lead facilitator briefly outline their personal improvement project, focussing on the variation features. If this is the same Lead Facilitator who introduced Variation in the module Improvement Principles, it will just need a minute, if a different facilitator and example, then it may need more than this. This is just as an example before participants look at their own:
Get out your Profound Knowledge notes. In pairs, find out what your partner measures out of work, and why. Eg do they run, are they learning a language or some other skill, or supporting a child to get better grades? Does anything strike you as particularly useful, not useful, or do you have any suggestions about other measures they may like to use?
2-3 mins. Longer will be required if participants did not attend Improvement Principles, or if it was a long time earlier.
In plenary invite participants to share some of their examples, and discuss why we measure.
Click (brings up bullets): now thinking about healthcare:
Compare with and build on what participants said.
What does ‘better’ look like? We need to know at all stages. So we can plan for it, and so we will recognise it when it happens.
“Measurement as a language helps translate complex and often nebulous concepts into a more precise form” (Kaplan & Norton The Balanced Scorecard)
Click (brings up box) Above all we must not forget that our aim is improvement, not measurement. We are measuring in order to make healthcare more safe, effective and person centred. Measurement is not an aim in itself, so we should always look for ways to use measurement that will minimise the time we take away from other elements of our day to day work, including other aspects of improvement, and delivering patient care.

6. What is measurement for improvement?
Accountability
Research
Purpose
New knowledge
Understanding of process
Evaluation of change
Comparison
Reassurance
Hypothesis
Flexible – hypothesis changes as learning takes place
No hypothesis
Fixed
Testing
Small, sequential tests
Observable tests
No tests
One large test
Tests blinded/controlled
Data
Gather just enough data to learn in each cycle
Large amounts of data
Gather as much data as possible ‘just in case’
Timescale
Short and current
Long and past
Confiden-tiality
Data used only by those involved with improvement
Data available for public consumption and review
Research subjects’ identities protected
Timing
5 mins
Lead Facilitator
Talk through the slide – focus mainly on the Improvement column, using the others for brief comparison and to help respond to any queries.
Accountability refers to what many people typically refer to as ‘audit’ (but the literature often uses the term ‘accountability’, which is why we use it here)
Key messages (in bold)
The aim of measurement for improvement is to improve healthcare systems, processes and outcomes, in the local healthcare context
Collect data to test if a change is an improvement
Also: to help get useful ideas for improvement
And: to find out if an improvement has been maintained
In improvement projects we do not work with a fixed hypothesis; data helps us to adjust a hypothesis as we work towards improvement
We try to develop measures from data that is already available, or easy to obtain. That way data collection can easily be integrated into the daily work routine, and if we find that the change is an improvement, it is easier to embed in the culture. Usually people collect data for improvement projects within the healthcare department or organisation where they work.
You need to see variation in data over time to know if a change is an improvement. Link the improvement to ‘the way we do things around here’, so it becomes part of the culture.
Make it EASIER TO DO THE RIGHT THING
Things we do NOT do:
Data for measurement is NOT used to evaluate performance
If bias exists in the process that we are measuring, we assume it is there all the time. We aim to design data collection so that any biases are stable.
You can observe the impact of a change while testing. When we measure for improvement, we assume that any biases are stable.
The small scale of each test of change reduces risk as each test is based on learning from the previous one.
Many people are persuaded by data – each small test of change generates data that can be used to persuade more people to get involved.
Key is not just what you measure but HOW you use that data.... Measurement for improvement requires a fundamental cultural shift from ‘sort it’ to ‘how can we help’.

7. Access to primary care
We’re going to look at some different measures relating to this scenario.

8. Three types of measure
Outcome
Tells a team whether the changes it is making are helping to achieve the stated aim
The aim in structuring and sequencing the slides this way is to introduce each of the three types of measure separately – not to introduce them all at once. So, at this stage don’t refer to Process or Balancing measures. The next slide is designed to raise the need for the other measures before introducing them.
Timing
10 mins max for this series of 4 slides
Lead facilitator
Outcome measure
This is the voice of the patient or other service user.
This measure is essential to justify the resources being put into a project.
Almost all improvement projects should include one or more outcome measures.

9. Three types of measure Lead facilitator
Aim
At primary care practice X increase by 20% the % of patients offered an appointment within 24 hours, by end December 2014
Lead facilitator
Imagine that for the last 3 months you’ve been doing work to achieve this aim.
Elicit: What measure do you think you might use? [eg % of patients offered an appointment within 24 hours]
You’ve been collecting data for this measure, and it’s showing no improvements at all.
Why do you think this might be? Groups discuss for 2 mins.
Plenary debrief should include:
There hasn’t been enough time yet to see an impact on outcomes
Our theory is wrong – we haven’t made the right change that will enable us to accomplish this
The changes that we have made might be the right changes, but we haven’t applied them in a reliable way.

10. Three types of measure
Outcome
Tells a team whether the changes it is making are helping to achieve the stated aim
Process
Tells a team whether a specific process change is having the intended effect
What we need is process measures, as well as outcome measures.
Process
This is the voice of the system. It addresses how key parts of the system are performing.
It should be logically connected to achieving the outcome(s), and directly related to changes you are making in your system.
Process measures generally show improvement before the outcome measure does. They provide an earlier indication of whether a change is an improvement.
Process measures are the only way to tell if failure to achieve improvements in our outcome is due to the wrong theory, (or not reliably applying the change.)
Often compliance with a process is key to achieving the desired outcome, so process measures help us to identify whether changes in a process help to make it easier for people to do the right thing.
Process measures should be simple to collect.
Elicit a couple of examples of process measures for the Access to Primary Care scenario.

11. Three types of measure Lead facilitator
Outcome
Tells a team whether the changes it is making are helping to achieve the stated aim
Process
Tells a team whether a specific process change is having the intended effect
Balancing
Makes sure that changes to improve one part of the system are not causing problems in other parts of the system
Lead facilitator
All improvement is change, but not all change is improvement.
Ie we need to know whether our change, which may seem to be an improvement, has resulted in any negative effects anywhere else in the system. NB the interdependence of elements of a system.
Balancing
A change is not an improvement if it has a negative effect elsewhere in the system.
Balancing measures help us to detect unintended consequences. They look at the system from a different perspective.
Sometimes they help to identify other factors that may explain an improvement.
When planning a test of change, it is helpful to try to anticipate one or two balancing measures; others may emerge once the test of change is under way.
An unintended consequence may be negative or positive.
Involving people from all parts of your system will help you identify potential negative balancing measures before doing, and actual ones once project under way.
Elicit a couple of examples for the Access to primary care scenario.
eg if systems become more automated, might this favour particular tech-savvy patients?

12. A New Healthier Me Aim A new healthier me!
Time taken to prepare meals
Cost of Mon-Thurs meals
Aim
A new healthier me!
Lose 7 pounds by end July 2015
Calories in
Calories out
Leisure activity
Calories in meals
Calories in drinks
Calories in alcohol
Work activity
Calories in snacks
Primary Drivers
Secondary Drivers
No alcohol Monday to Thursday
Max 1 x juice or soft drink per day
Reduce portion size
Cook evening meals from scratch
Replace biscuits/cakes with fruit
Keep to shopping list
Get up from desk to talk, instead of phone or
Use stairs not lift
Walk to a daily step target
Swim at least twice a week
Change ideas
Family perception of new lifestyle
Number of items bought not on shopping list
Daily Calorie intake
Fruit replaces cake/week
Days between cakes
Key messages:
You have to work out what is the most suitable measure for your project – usually there is more than one possibility. We’ll look at that in more detail in future modules.
You will normally have one (maybe 2) outcome measures, several process measures, and one or two balancing measures.
Don’t become a data factory – only use the measures and collect the data that you really need.
The measures you choose will also depend on whether you have the tools/resources needed for that particular measure.
(If operational definitions crop up, cover briefly – planning to do this in detail in another module
Timing
A New Healthier Me – 5 mins
ICU measures – 10 mins
Lead facilitator
Examples of different measures:
One category at a time, elicit some possible measures, then show the examples and discuss.
Click 1 – outcome measures
Click 2 – process measures
Click 3 – balancing measures (top left hand of slide)
If participants hadn’t had many ideas before you showed them the examples at each click, elicit more examples before moving on.
Participants may query the different outcome measures – eg why have measures that are different from the aim? Point out that while at the ‘thinking’ stage (the three questions in Model for Improvement), we might adjust our aim based on our thinking about measurement. The Three Questions are iterative, so this may lead to other changes in the driver diagram too. Eg our scales may be unreliable or broken, so we decide to use a different measure.
PRACTICE
Material
ICU types of measure
ICU types of measure KEY
Categorise each measure as O, B, or P
Do the first one as an example in plenary. Elicit answers and discuss if more than one answer given.
Participants do the rest individually, then compare answers in pairs.
Facilitators
Light touch monitoring – intervene if people are struggling.
Plenary debrief. The measures are numbered, so it isn’t necessary to read out the whole measure.
Weight
BMI
Body Fat
Waist Size
Jeans fit
Flights of stairs climbed/day
Calories
Burned/day

13. Create measures: Staff Wellbeing
1 outcome measure
3 process measures
1 balancing measure
Aim
To start thinking about how to define a measure.
Timing
10 – 15 mins
Material
Measures Staff Wellbeing
Lead facilitator
Do the outcome measure in plenary.
Process measures: If five groups/tables of participants, assign one Secondary Driver (and its associated change ideas) to each table, or similar.
Work together to create 3 process measures (these could include more than one measure for the same change idea), and one balancing measure.
Facilitators
Monitor and support as required.
Optional
Materials
Wall cards – Staff Wellbeing measures
If time, or for early finishing groups – an initial ‘debrief’ to guide their learning, Facilitators distribute cards for walls, matched to the tables working on those processes. They compare with their own, then put the cards on wall next to the relevant process.
Balancing measures to side or below.
Debrief (this could be plenary or if enough facilitators, each work with one table to debrief) – based on issues arising. Eg reinforce that it’s ok if their measures were different from the ones on cards, so long as they can justify (more detail on this in later module(s).

14. Project work: create measures
For your aim and highest priority change idea, create measures
1 outcome measure
2 process measures (for the same change idea)
1 balancing measure
PROJECT
Aim
Participants to begin drafting their own measures so that when they work on measurement planning in more detail, including operational definitions, they aren’t starting from a blank sheet of paper.
Timing
5 – 10 mins
Lead facilitator
These are just to be rough outlines – you will work on these again in a future module.
Facilitators monitor and support as required.
If time very short, participants could do this after the session and before the Planning Measurement module.

15. Introduction to measurement: Summary
Aim
To briefly recap the session content:
- to support a sense of learning and accomplishment
- to aid memory of the session later
An opportunity for participants to ask any outstanding questions from any part of the session.
Timing
1 – 5 minutes, depending on time available
Lead Facilitator
Elicit what content was covered eg:
Why is measurement in an improvement project? [to know whether a change is an improvement]
What’s the measure called that relates to our aim? [outcome]
What’s the measure called that relates to our change ideas? [process]
Why do we need balancing measures? [to check whether our change is having an impact on other parts of our system]