1. Practice Improvement –
Making Quality Improvement Work for You
A series of Small Group Learning Sessions:
Session 2: How Will We Know That a Change is an Improvement?
Session Date
Presenters’ Name(s)

2. Faculty/Presenter Disclosure
Speaker’s Name: Speaker’s Name
Relationships with commercial interests:
Grants/Research Support: PharmaCorp ABC
Speakers Bureau/Honoraria: XYZ Biopharmaceuticals Ltd
Consulting Fees: MedX Group Inc.
Other: Employee of XYZ Hospital Group
Please fill out all applicable areas (highlighted in red). One slide per speaker.

3. Disclosure of Commercial Support
This program has received financial support from [organization name] in the form
of [describe support here – e.g. educational grant].
This program has received in-kind support from [organization name] in the form
of [describe the support here – e.g. logistical support].
Potential for conflict(s) of interest:
[Speaker/Faculty name] has received [payment/funding, etc.] from
[organization supporting this program AND/OR organization whose product(s) are
being discussed in this program].
[Supporting organization name] [developed/licenses/distributes/benefits from
the sale of, etc.] a product that will be discussed in this program: [enter generic
and brand name here].
Please fill out all applicable areas (highlighted in red).

4. Mitigating Potential Bias
[Explain how potential sources of bias identified in slides 1 and 2 have been
mitigated].
Refer to the College of Family Physicians of Canada’s “Quick Tips” document.
Please fill out all applicable areas (highlighted in red).
Please visit the following link for the CFPC’s “Quick Tips” document:

5. Agenda Small Group Learning process Action Plan Report Out
Why measure? Core concepts
Break (10 min)
Types of measures
Partner activity
Table activity
Action Planning
Here is our plan for the day. Over half of the time in this session today is interactive – the orange text delineates these sections.
We’ll start with a quick run through the concept and process of SGLS, some general housekeeping, and talk about learning objectives and why we are here.
Then we will get into quality improvement, you’ll get to hear from another doctor about using QI for practice improvement in health care, and then we’ll do a couple of partner activities which will take up the bulk of the first half of the session.
We’ll take a 10 minute break, and then talk about the guts of quality improvement. We’ll have an ongoing discussion around these ideas, look at some of the tools you can take back to practice with you, and then do another partner activity where you can apply these tools to your own experience as a physician.

6. Introductions: Peer Mentor and RST Coordinator
Please introduce yourselves and emphasize the working together/helping each other atmosphere, not just between Peer Mentor and RST but among each other (eg. Peer Mentor and audience)
Peer Mentor to provide example of case highlighting benefits to patients, practice and the physician experience. (ie your story about QI experience)

7. What is a SGLS?
Focus on application of knowledge and practical problem solving
Group of physicians and practice team members
Short &
Interactive
Session
Ongoing support
Action Plan
In-Practice visit with RST
Integrated Learning Package
Facilitators: Peer Mentors & Regional Support
Team Coordinators/Coaches
Small Group Learning Sessions (SGSL) are offered by the Practice Support Program (PSP) as a result of demand for short, accessible training sessions on priority content.
Small groups of physicians (and in some cases their teams) get together for short, interactive education sessions that are certified by the College of Family Physicians of Canada for group learning credits.
Peer Mentors guide participants through the content, encouraging questions and discussion throughout.
There is a focus on applying knowledge and lots of support for integrating session learnings and applying them in the practice.
Lots of options for ongoing support and learning
Sessions provide opportunity for both social and educational learning

8. Housekeeping www.gpscbc.ca/psp-learning/
Please put your cells on vibrate or turn them off. We appreciate that you may need to step out to take urgent calls. Please feel free to do so, but - as a courtesy to others - please keep your phones on vibrate.
Washrooms are here and there. We know emergencies sometimes come up so please feel free to leave the room if you need to take a call
We’ll be taking a 15 min coffee break at _____ and then there will be ______ when we finish at ______
Timing – We have two and half hours together today, and we’re going to work really hard to stick to this. We will be providing timelines for the activities and we ask for your help in keeping us all on track! We’ve tried to account for quite a lot of discussion and interaction time, so our hope is that we can achieve our timelines and get your questions answered. That being said, if we move on from a topic that you’d like to discuss further or get more information on, please make a note and let the RST know after the session; they can help make sure that you get what you need in an in practice visit.
We have a post session evaluation and ask that you fill it out carefully so that we can share it with the provincial office and try to integrate improvements to these sessions– in the true spirit of quality improvement!

9. Learning Objectives The learner will be able to:
Develop outcome measures
Develop process measures
Develop balancing measures
At the close of this session, we hope you will feel very comfortable in doing these -
These are the learning objectives that we’ve committed to. You will leave the session ready to take these on, and know where and how to access support to do so.

10. Action Period Report-Out
Tests of change
Measurement
Successes
Challenges
Each QI session will include time to review your Action Period work.
As you progress further in the series, you will be able to report back on an increasing number of elements of the QI process.
Suggest starting with the Peer Mentor sharing their AP work, to model and inspire others. …/

11. Steps for QI project Decide what needs to improve? Share learnings.
Build a team who knows about the process.
Clarify what the current process is.
Brainstorm ideas and test changes.
Implement sustainable changes.
Share learnings.
These tenants are reflected through the steps for a QI project, which also hopefully looks familiar from our last session, where we talked about understanding what needs to improve and started to work on clarifying the current process. You have started to understand the team you might need in your office to realize improvement via your action plan work.
Today we are going to start on one of the bigger steps, referred to here as: brainstorm ideas and test changes. There’s actually a lot contained in this one little bubble – so we are going to further unpack it in the next slide.

12. Model of Improvement & PDSA
Aim Statements
Measures
Change ideas
Each PDSA cycle repeats the questions for a very specific, narrow question.
Hopefully this looks familiar. We spoke in brief about it at the last session and the Action Plan is based on it. This is the model of improvement, and it represents one approach to quality improvement that is quite practical to use in your office.
We started working on the steps in the first session, and through your action plans. Hopefully by now we’ve all identified an opportunity for improvement in our practice, unpacked it a bit, and thought about who should be involved in addressing the problem. Now that we each understand our problems a bit better, we’ll look at how to move forward into addressing problems through the model of improvement, and then by testing changes through a PDSA cycle.
Remember, its important to start with just one specific problem.

13. change is an improvement?
Model for Improvement
What are we trying to
accomplish?
_________________________________ 
How will we know that a
change is an improvement?
______________________________________
What changes can we make that
will result in improvement?
We’re going to continue working through the model of improvement. We’re now onto asking “How will we know a change is an improvement?”

14. Model for Improvement How will we know a change is an improvement?
After two weeks, the team decides that this has been a success because Liz has received about 50% less complaints. Liz was very diligent about keeping a tally of how many complaints about the wait time she received, and estimates there’s been a 50% reduction compared to before the team implemented the change.
Dr. Smith is trying to decrease the amount of time a patient sits in his waiting room before actually getting to see him; his MOA, Liz, gets a lot of comments about this and they both feel like it’s impacting care because patients are often tired and annoyed by the time they see him (he has a lot of older patients) and he suspects this contributes to poor communication in the visit. So, he implements a change idea to see if it can improve the situation.
How will we know a change is an improvement?
This slide is animated; click your mouse anywhere on the screen when indicated below to cycle through the content.
Measurement seeks to provide answers to the question: How do you know if the changes you are making are leading to improvement?
[Click]
Successful measurement is a cornerstone of successful improvement.
Measurement in quality improvement allows a team to demonstrate current performance (or baseline), set goals for future performance, and monitor the effects of changes as they are made.
Determine what to measure.
Determine how to collect this data.
I’m going to give you an example of the importance of good measurement in being sure that a change is an improvement.
Dr. Smith is trying to decrease the amount of time a patient sits in his waiting room before actually getting to see him; his MOA, Liz, gets a lot of comments about this and they both feel like it’s impacting care because patients are often tired and annoyed by the time they see him (he has a lot of older patients) and he suspects this contributes to poor communication in the visit. So, he implements a change idea to see if it can improve the situation.
After two weeks, the team decides that this has been a success because Liz has received about 50% less complaints. Liz was very diligent about keeping a tally of how many complaints about the wait time she received, and estimates there’s been a 50% reduction compared to before the team implemented the change.
How do you know whether the change is an improvement?
Prompt: You don’t! There’s two big problems here.
One - Dr. Smith and Liz aren’t measuring the right thing. They didn’t measure the amount of time the patient waited in their waiting room before seeing Dr. Smith, they only thought about the patients that complained, which indeed may be associated with the amount of time a patient waits, and which is likely also desirable to decrease, but this is not the primary goal.
Two – Liz did some measurement by keeping that tally of complaints (even though it was the wrong thing to track) but she had to estimate how much of a change there was – because she had no initial data to compare her tally to. She had no baseline data, so this information is not very useful.
As you can see, poor measurement does little to help you know whether the change you’ve introduced is really effective. Therefore, good measurement is essential to ensuring that you and your team are using your time effectively when making changes in order to ensure they are resulting in the improvement you are seeking.

15. What do you measure in your practice already?
Why Measure?
Why are we measuring?
Why do we need data?
What do you measure in your practice already?
I’d like to pause for a minute and go back to the key question that underpins quality improvement - and our work as clinicians - asking WHY?
Why are we measuring? Why do we need data? I want to highlight that there is a big difference between the data you might gather for research purposes and the kind of data we want to have in order to do QI. Thankfully! Otherwise this would be a gargantuan task and it really doesn’t have to be.
First I will put this question to you:
Why do you think we need data for QI?
Why are we bothering to measure?
Do you have examples of measuring in your practice?
Prompt: Your measures monitor whether the changes you make are actually improving things or not and will help you gauge how close you are to achieving what you set out to accomplish.

16. The principle: Just enough data to get going!
The Purpose of Data in QI Projects
The principle: Just enough data to get going!
Need to know:
Where we started (baseline).
How we change over time (e.g. each week).
How we are doing (vs. baseline).
Data in QI projects revolves around these 3 “must knows”
Where we started (baseline).
How we change over time (e.g. each week) . Here is important to ask the question “How often should you measure?”
How we are doing - in comparison to the baseline, so we know whether to adjust what we are trying.
Measurement does not have to be difficult or time-consuming. The key is to pick the right measures so that the quality improvement team can see results quickly and are able to adapt their interventions accordingly, putting less strain on resources and more focus on outcomes.
Not all changes are going to lead to improvement. You need data to inform the team whether the changes are working.
This is where you specify what you will measure, and what data you need to measure it, in order to know whether or not you have actually made an improvement.

17. Measurement in QI
Measurement for Research
Measurement for Improvement
Purpose
To discover new knowledge
To bring new knowledge into daily practice
Tests
One large “blind” test
Many sequential, observable tests
Biases
Control for as many biases as possible
Stabilize the biases from test to test
Data
Gather as much data as possible, “just in case”
Gather “just enough” data to learn and complete another cycle
Duration
Can take long periods of time to obtain results
“Small tests of significant changes” approach accelerates the rate of improvement
For those of you still not convinced – here is a breakdown of the difference between measurement for research vs QI. It’s very different. The orange half is what we are aiming for.
The goal is NOT the maximum amount of data; its just enough data.
The goal is to bring new knowledge into daily practice, and use small tests based on just enough data about one very specific thing.
We will cover more about how to measure after we’ve talked about the rest of the Model of Improvement.
Measurement is vital to ensuring that the changes you are making lead to improvement.
*Measurement for Quality Improvement, Health Quality Ontario

18. Measurement: Core Concepts
Start early.
Measure often and over time.
Have multiple measures.
KIS (Keep it Simple!)
Okay, but where to start?
Start with your aim statement.
These are the core principles to keep in mind when thinking about measurement, and seeking to answer the question - how will we know when a change is an improvement?
Start early – you need to have something to compare to. Measures may change over the course of your project.
Measure often and over time – gives more accurate information than data measured only before and after a change is implemented. Keep in mind that you are seeking a balance between getting enough information and getting more information than is strictly necessary – this can yield more work and no added benefit.
Have multiple measures - You will need a few different measures to tell the whole story.
4. Having said all this - the most important thing about measuring is its actually doable. This is why it’s important to balance the above goals with KIS – Keeping it simple! Some measurement of your progress is better than none, and as you build your technique and learn what measurement works for your practice and what is actually useful, you can get even closer to what ideal measurement will look like.
When thinking about what to measure, it’s useful to go back to your aim statement to think about what you are trying to accomplish.
These are the things you are trying to measure. You will now really appreciate a clearly defined and measurable aim statement!
The next slides show why data over time is crucial in measurement.

19. Why Not Before and After?
Because QI is about making changes, one of the first instincts can be to measure before a change and then after a change.
This example shows data around a change meant to decrease the wait time.
The team measured wait time before the change as 8 days, and then measured again after the change and found the wait time was 3 days.
This measurement appears to show a change that was successful.
However – there is a problem with this approach.
(Adapted from Provost and Murray, 2011)

20. Scenario 1 What does this slide tell us??
Prompt: There is no obvious improvement after the change was made, there is just lots of variation from week to week.
If we just measured on week 4 and week 1 like we did with the pre and post, our bar chart would lead us to think the wait time had improved.
(Adapted from Provost and Murray, 2011))

21. Scenario 2
What about this slide - Does it look like the change has been successful?
It does, but only temporarily. Again, this does not show up on the bar chart.
When you collect more data over time and review it weekly, the data clearly tells a different story.
(Adapted from Provost and Murray, 2011)

22. Scenario 3
From this example, we can see the importance of collecting data over time.
We need to see data over time to analyze the fluctuation or variation. There will always be some up and down, and we need to determine what that variation means:
Is it random fluctuation and therefore not really showing anything?
Is it actually evidence of improvement? Is the data going up (or down) in a way that is not random?
If it is improved, does it stay at that level?
Data over time helps us identify signs of improvement.
(Adapted from Provost and Murray, 2011)

23. Break 10 minutes
We’ll take a 10 minute break and then move onto the HOW of the model of improvement.

24. Partner Activity
Work with a partner to develop a outcome measure for your aim statement.
5 minutes each; I will let you know when to switch.
Ask for support/feedback at any time!
Outcome Measures
Show if changes are leading to improvement and achieving the overall aim of the project.
Okay, time to move! Switch tables, partner with someone you don’t know, and work to develop measures which reflect your aim statement.
I will leave this screen up - it has reminders of what the different measures you are looking to develop are.
If you’d like to talk these out with myself or your RST coordinator, please don’t hesitate to flag us down. You may also approach another pair for feedback and discussion. This is yet another piece of QI that benefits from a team approach and multiple heads!
If there is time or interest, you may want to ask some teams to briefly share their measures any learnings from the process.

25. Partner Activity
Work with a partner to develop a process measure for your aim statement.
5 minutes each; I will let you know when to switch.
Ask for support/feedback at any time!
Process Measures
Show whether a specific change is having its intended effect.
Okay, time to move! Switch tables, partner with someone you don’t know, and work to develop measures which reflect your aim statement.
I will leave this screen up - it has reminders of what the different measures you are looking to develop are.
If you’d like to talk these out with myself or your RST coordinator, please don’t hesitate to flag us down. You may also approach another pair for feedback and discussion. This is yet another piece of QI that benefits from a team approach and multiple heads!
If there is time or interest, you may want to ask some teams to briefly share their measures any learnings from the process.

26. Partner Activity
Work with a partner to develop a balancing measure for your aim statement.
10 minutes each; I will let you know when to switch.
Ask for support/feedback at any time!
Balancing Measures
Help ensure that changes to improve one part of the system are not causing new problems in other areas.
Okay, time to move! Switch tables, partner with someone you don’t know, and work to develop measures which reflect your aim statement.
I will leave this screen up - it has reminders of what the different measures you are looking to develop are.
If you’d like to talk these out with myself or your RST coordinator, please don’t hesitate to flag us down. You may also approach another pair for feedback and discussion. This is yet another piece of QI that benefits from a team approach and multiple heads!
If there is time or interest, you may want to ask some teams to briefly share their measures any learnings from the process.

27. Putting it all together…
Switch tables
A fresh worksheet will be distributed
Everyone at each table will independently develop measures for the same aim statement (5 minutes)
Discuss with your table (10 minutes)
What measures developed at your table make sense to you?
Are there types of measures that are still confusing?
How can you use this in practice?
Ask folks to redistribute themselves amongst the tables.
Station a facilitator at each table. If there are more than two tables, reconsider how the activity is structured.
Distribute a Measures worksheet to each participant.
Take 5 minutes and have each person independently develop all three measures for the same aim statement.
Then, take 10 minutes to discuss with your table - share your measures, challenges, and questions.
If there is time or interest, you may want to ask some teams to briefly share their measures any learnings from the process.

28. Action Plan Intended to provoke thought and discussion with your team.
We’re going to talk briefly about the ongoing learning and supports offered.
We’ve handed out paper copies of the Action Plan. We hope you will be completing these with your office team, and we can compensate all of you for your time in working on this together (up to two hours). The Action Plan is a standard set of five questions that are core to the quality improvement process – you may recognize some of them from the model of improvement. You will learn more about this model as you progress through the QI SGLS sessions, and you may find that how you interpret the questions changes, or that you have more specific ways for the tools to help you answer the questions. This reflects your learning process and allows you to complete the Action Plan based on your current understanding and your comfort level with the concepts addressed.
You will start the AP today, work on it with your team back at the office, and then in a few weeks you will be ready to answer the final question, submit it and your sessional form.
Filling out the Action Plan is intended to provoke thought and discussion among your team about what you have learned, and your goals for practice improvement. There is no right or wrong here. The Plan may provoke questions and even be used to structure an in-practice visit with an RST or peer mentor. They will support you in working with your team to get everyone on the same page and get started.
Intended to provoke thought and discussion with your team.
5 questions core to the QI process.
The same for each SGLS.

29. Integrated Learning Package
Option for more supported learning time with team:
Brief review of key session concepts
Team discussion
Worksheets
Problem Statements
Building a QI Team
If you want to share more of the learnings back with your team, you can also complete the optional Integrated Learning Package. This reflects the learnings from today and integrates them into practice. If you complete this package, you can receive up to 4 CFPC certified credits in the self-learning category, as well as an additional two hours of sessional payment for team members involved in completing it.
So that you are familiar with this package -
The first three pages outline team activities centered around the learnings, and provide room to document this discussion. This is what you will submit to your RST.
The remaining pages are two worksheets which review core concepts from the session and support you in completing the package.
The information in your package is enough to sit down and do this with your team if you choose as it provides step by step directions. An RST can also support the completion of the Integrated Learning Package in an in-practice visit, working with your team to get everyone on the same page and get started.

30. Ongoing Learning and Support
Who to lean on and work with going forward:
Your colleagues
Your office team
Peer Mentor(s)
Regional Support Team Coordinators/Coaches
We are nearing the close of the session. I hope you have found this time useful and that you have some learnings to take away and begin to apply to your daily operations in your practice. I want to highlight that you have lots of resources to draw on as you begin – the group here today is a valuable resource for hashing out challenges and learning from each other’s successes, however big or small. If there is interest, we can facilitate sharing of contact information today so you can keep in touch and further support each other.
Your office team will be a key support in planning for change, and the Action Plan includes work with your team in order to start this conversation.
There are also a number of PSP resources available to you – the peer mentor team, including myself, and the Regional Support Coordinators, all of whom are available to work with you one-on-one in your practice to help integrate ideas that you learned today, complete your Action Plan, and work towards realizing improvements in your practice.

31. Next Steps
Action Plan – start today, due in 8 weeks (trigger for sessional)
Post Activity Participant Questionnaire (PARQ) – in 8 weeks (trigger for Mainpro+ credits)
Integrated Learning Package (2 hours) Optional
In-Practice RST or Peer Mentor visit Optional
External Resources:
UBC/BCCFP - “Shine a Light on Your Learning”: A Self-directed Assessment Tool (Independent, online, up to 24 Mainpro+ credits).
Our next steps are:
Completing the action plan (we will take a few minutes shortly to start this). Please submit your completed AP to your RST within 8 weeks in order to receive up to two hours sessional payment for your time in doing this work.
After these 8 weeks, you will be asked to complete the Post Activity Participant Questionnaire (PARQ), which hopes to identify the impact of participating in this session. Submitting the PARQ triggers the receipt of your Mainpro+ certificate.
You also have the opportunity to have an in-practice visit with an RST, who can help you and your team complete the action plan and/or the integrated learning package if you choose, as well as help address any challenges you are experiencing. Please connect directly with your RST to get schedule an in practice visit. One of the things the RSTs can do with you is the PSP Practice Assessment Tool, which may help you decide where you’d like to focus your attention for the Action Plan.
Finally, we’ve included a link to a tool developed by UBC CPD and the BC College of Family Physicians that is complimentary to the learnings today, and will allow you to get credits in a different category. This is a self-assessment tool which focuses on making improvements in your practice and can walk you through the process in more detail, providing ideas and support. This tool is something you can work through step by step and get up to 24 Mainpro+ assessment credits. If you have questions about this tool please contact….

32. Get Started on Action Plan
10 minutes to start – Go!
We’ve handed out paper copies of the Action Plans so you can take a few minutes to reflect while the ideas are still fresh, and discuss with your table today.
You now have 10 minutes to start work on your action plan. We are here to support you - please flag us down if you’d like to talk.
As AP is being completed, hand out the evaluation form. Leave this slide up until the 10 minutes is complete.

33. Evaluation Please complete the Post Session Evaluation form
Your feedback informs improvements to this session and future SGLSs.
Thank you!
Key Messages from today:
This is a culture change – self-reflection is what is going to be expected for purposes of both accountability and credibility.
How to get this going? – remember that curiosity is sign of wellness, just as cynicism and stagnation is a sign of burnout!
Measurement has to be done, in the right way, to make clinical quality improvement practical and meaningful
That’s it for learning today! We ask that everyone complete the post-session evaluation so that we can improve this session and make it more useful for your peers.
Thank you for your time, your attention, and your ideas. We hope the session has been of use, and look forward to working with you on planning and implementing improvements in your practice.