1. Measuring the Effectiveness of Change Rosa West PhD, LMHC, LMFT
How’re We Doing?
…..measuring the effectiveness of change
Rosa West PhD, LMHC, LMFT
This product was supported by the Florida Department of Children and Families Substance Abuse and Mental Health Program Office funding.
Rosa West PhD, LMHC, LMFT

2. Leading Successful Change
Requires:
Leadership
Staff Buy-In
Resources
Commitment
R. West PhD

3. Change is Hard
Information or Data is invaluable to leading successful change
R. West PhD

4. R. West PhD

5. Good Data Provides… Enhanced services
Greater likelihood of finding a solution
Stronger Support & Clarity
Starting point!
R. West PhD

6. So Why Measure?
We can use data to create change by measuring what matters!
R. West PhD

7. Types of Measures Outcome Measures Process Measures Balancing Measures
Tells whether changes made lead to improvement
Outcome Measures
Tells whether a specific process change has been accomplished and if it is having intended effect
Process Measures
Used to determine if changes to improve one part of the system is or isn’t causing problems in another part
Balancing Measures
R. West PhD

8. Outcome Measures Reducing Appointment times
Average length of time in days between the day a patient requests an appointment and the next available appointment
14 day avg reduced to 3 day avg
R. West PhD

9. Process Measures Patient Continuity
Percentage of visits the patient makes to their own clinician vs. another provider
R. West PhD

10. Continuity of Care Formula
Formula: Number of Patients of Provider X that were seen by Provider X, divided by the total number of Patients for Provider X that have been seen.
94 of Dr. Smith’s patients were seen in clinic this month.
78 patients were seen by Dr. Smith.
16 of Dr. Smith’s patients were seen by another provider.
78 / 94 = .829
.829 X 100 =
82.9% Continuity
R. West PhD

11. Balancing Measures
How would you rate the length of time waiting for today’s visit?
R. West PhD

12. When Implementing Change…
It’s important to assess readiness in instituting change
R. West PhD

13. Ready…
Set…
Go!!!
R. West PhD

14. Change Readiness Diagnostic: Risk Assessment
Rate each from 1 to 5 with
1 = very low, 2 = low, 3 = medium, 4 = high, 5 = very high.
Totals points Watch Out!
points Keep a Close Eye on Things.
points You/ Your Organization are Very Likely to Successfully Implement the Change Program.
Risk Factor 1: Adequacy of Risk Management Process
Risk Factor 2: Adequacy of Change Program Definition
Risk Factor 3: Effectiveness of Change Management Process
Risk Factor 4: Adequacy of Sponsorship and Resources
Risk Factor 5: Adequacy of Communication and Involvement
Risk Factor 6: Range of Linked / Consequential Actions Identified
Risk Factor 7: Coherence in the Sequencing of Linked Actions
Change Readiness Diagnostic: Risk Assessment

15. Change Readiness Diagnostic:
Risk Assessment
Rate of Return Factor 1: Extent and Timing of Benefits
Rate of Return Factor 2: Change Program Budgets Economy
Rate of Return Factor 3: Extent to which Project Time, Specifications and Costs are Managed
Rate of Return Factor 4: Degree of Focus on Business Results
Latent Opportunity Factor 1: Program Scope
Latent Opportunity Factor 2: Linking Change Drivers Through Actions to Performance
Latent Opportunity Factor 3: Appropriateness of Benchmark Targets
Latent Opportunity Factor 4: Quality of the Benchmarking Process

16. Mobilizing Commitment Stakeholder Analysis/Commitment Chart
Grumpy
Grumpy
Snooty
Snooty
Grouchy
Grouchy
Slouchy
Slouchy
Steps:
1. Plot where individuals currently are with regard to desired change (O = current).
2. Plot where individuals need to be (X = desired) in order to successfully accomplish desired change identify gaps between current and desired.
3. Indicate how individuals are linked to each other, draw lines to indicate an influence link using an arrow () to indicate who influences whom.
4. Plan action steps for closing gaps. Can use Resistance Analysis tool.

17. Include as member of Change Team
Mobilizing Commitment Resistance Analysis
Include as member of Change Team
Job Security
Grumpy
Ambiguous Roles
High ?
Med ?
Snooty
Structure
Grouchy
Slouchy

18. Collecting Baseline Data
Measuring the Effectiveness of Change
Collecting Baseline Data
Before you begin any process improvement project you need to collect baseline data. Keep this in mind “How will you know how far you gone if you don’t know where you were?” (click to the next slide)
Rosa West Ph.D.
Rosa West PhD, LMHC, LMFT

19. Measuring the Effectiveness of Change
Aims & Goal Setting
Reduce Waiting Times
Reduce No Shows
Increase Admissions
Increase Continuation
After you conduct your initial walk through want to make sure you identify your aim. The Aim that you choose to focus on should be selected based on the insights gained from your walkthrough or aligned with the priorities of your CEO and/or Executive sponsor. A quick review of the NIATx aims (click to the next slide)
Rosa West Ph.D.
Rosa West PhD, LMHC, LMFT

20. An Example: Reducing Waiting Times
Measuring the Effectiveness of Change
An Example: Reducing Waiting Times
Let’s consider an example. Although we may not actually see this standing in front of our desk, this is the vision; particularly this gentlemen staring at watch wondering when it’s his turn. I think this is a concern for many of us as we want to make sure we provide timely service (click to next slide)
Rosa West Ph.D.
Rosa West PhD, LMHC, LMFT

21. Measurement
R. West PhD

22. Measuring the Effectiveness of Change
Key Principles…
What gets measured gets done!
Change is virtually impossible when we do not measure our performance.
R. West PhD
Rosa West PhD, LMHC, LMFT

23. Measuring the Effectiveness of Change
Key Principles…
It’s important to choose the most appropriate means of measuring the change
Change is virtually impossible when we do not measure our performance.
R. West PhD
Rosa West PhD, LMHC, LMFT

24. When Choosing Measures:
Relevance: does it have a significant, demonstrable relation to the goals/objectives?
Reliable: is the measure consistent? Will we get the same results after repeating the measure?
Availability: are the data necessary for measuring the change available at a reasonable cost?
R. West PhD

25. If You have the Wrong Problem, You will get the Wrong Solution.
Problem exploration is the first stage in selecting the change and establishing measures
However……
Exploration may be limited because you think:
The assumptions are known (although assumptions can be so ingrained you fail to notice them)
Complexity means this will be hard to solve
You know what you want
The problem is obvious (although perception is flawed)
R. West Ph.D.

26. Considerations in Selecting Measures
Measuring the Effectiveness of Change
Considerations in Selecting Measures
Re-evaluating existing measures (minimize influence of):
Fear
Ownership
What-Ifs
Estimates
Staff/Managers may be fearful of measuring performance. What will the data show? How will the findings effect us?
R. West PhD
Rosa West PhD, LMHC, LMFT

27. Considerations in Selecting Measures
Measuring the Effectiveness of Change
Considerations in Selecting Measures
It’s important to measure processes and results
Measuring processes can provide information about the effectiveness of the process and measure the ultimate results of these process.
R. West PhD
Rosa West PhD, LMHC, LMFT

28. Considerations in Selecting Measures
Measuring the Effectiveness of Change
Considerations in Selecting Measures
Measures Should Foster Goal-Driven Teamwork
R. West PhD
Rosa West PhD, LMHC, LMFT

29. ESTABLISH BASELINE RESULTS
Plan: Change is driven by the action steps established and measurements set in place to assess performance.
Hold the
Gain
Study
Act
(implement) Do
Plan
Standardize
ESTABLISH BASELINE RESULTS
R. West PhD

30. Measuring Effectiveness of Change
Measuring the Effectiveness of Change
Measuring Effectiveness of Change
What change was most important?
What changes worked?
Before
What changes resulted in an improvement?
During
What didn’t work?
We need to make sure we measure the effectiveness of our change by asking ourselves multiple questions…
By collecting data before, during, and after your organization’s change cycle, you can measure the progress you have made towards achieving your goals and the impact of your change cycle. (click to next slide)
After
R. West PhD
Rosa West PhD, LMHC, LMFT

31. 6-Step Process of Measuring Change
Define Measures
Establish Aims
Collect Baseline Data
Ask Questions
Consistently Collect Data
Chart Progress
R. West PhD

32. Measuring the Effectiveness of Change
Define Measures
Define your measures before you start the change process
Definition MUST be clear to measure change successfully
Defining measures does not have to be overly complicated…keep it simple with tally sheet, this something quick an easy anyone can use that yields the same results as any other measure. Or..and excel spreadsheet.
Click to next slide
R. West PhD
Rosa West PhD, LMHC, LMFT

33. Measuring the Effectiveness of Change
Define Measures
Defining measures does not have to be overly complicated…keep it simple with tally sheet, this something quick an easy anyone can use that yields the same results as any other measure. Or..and excel spreadsheet.
Click to next slide
Tally Sheet
R. West PhD
Rosa West PhD, LMHC, LMFT

34. Excel Spreadsheet
R. West PhD

35. Collecting Data Be specific (clarity)
What is “one day”?
Do PDSA on data collection method
Check data periodically during collection process to ensure procedures are consistent
R. West PhD

36. Collecting Data Be mindful of potential variations
i.e. sampling today may not yield same sample than during holidays
R. West PhD

37. Measuring the Effectiveness of Change
Baseline Data
BEFORE
Before you change anything you must collect baseline data. Baseline data shows you where you are today! It allows you to measure the impact of any changes that are made. It also provides a before and after picture of a situation. Now let’s go back to our original example…(Click to next slide)
AFTER
R. West PhD
Rosa West PhD, LMHC, LMFT

38. An Example: Reducing Waiting Times
Measuring the Effectiveness of Change
An Example: Reducing Waiting Times
To collect baseline data, each day have the office assistant document the number of days until the next available appointment. (Data can be collected for 1 week, 2 week, etc) Once we have our data we can establish our aim…click to next slide
Record the number of days to the next available appointment
R. West PhD
Rosa West PhD, LMHC, LMFT

39. Measuring the Effectiveness of Change
Establish Aims
Establish a clear aim.
Reduce wait time from 21 days to 15 days
Be flexible when establishing your goals. And keep in mind, that goals can be too ambitious or not ambitious enough. Set incremental goals to allow you to celebrate your successes along the way. Set a realistic goal that still challenges your organization to improve. Once you achieve your goal, set a new one! Click to next slide.
R. West PhD
Rosa West PhD, LMHC, LMFT

40. Consistently Collect Data
Measuring the Effectiveness of Change
Consistently Collect Data
Continue to collect data on the number of days to the next available appointment as you implement changes.
This is a critical part of the change process. You may already have measure in place to track & collect data. However, you may need to establish measures to collect data relevant to the changes you intend to make. Click to next slide
R. West PhD
Rosa West PhD, LMHC, LMFT

41. Measuring the Effectiveness of Change
Chart Your Progress
WAIT TIMES
Pre-Change
Baseline
=21 days 20
One graph One Message 15
Excel Chart
Number of days 10 5
Overtime you will collect both pre-change & post change data. Make sure you share this data with others in your organization. An effective means of sharing this information would be through an excel chart. Any visual aids you use should be simple and to the point.
R. West PhD
Rosa West PhD, LMHC, LMFT

42. Measuring the Effectiveness of Change
Chart Your Progress
Pre-Change
Baseline
=21 days
WAIT TIMES 20
17 days 15 10
Number of days 5
Once the change was implemented there was almost an immediate reduction in wait time, with a 4 day drop in in wait time within one week. You can also track a steady decline in wait time over the course of the change project.
R. West PhD
Rosa West PhD, LMHC, LMFT

43. Measuring the Effectiveness of Change
Ask Questions
What does the information tell me about changes in my organization?
If change is successful, the information should tell you which information was useful in helping you meet your goals. Remember to ask questions throughout the change process. What is the information telling me about change in my organization?
R. West PhD
Rosa West PhD, LMHC, LMFT

44. Celebrate your successes!!!
R. West PhD

45. Warning!!! Do not collect too much data
Do not focus on too many measures
Avoid analysis paralysis!
R. West PhD

46. Tips for Success Identify the question you want answered.
Build measurement into daily workflow.
Post data where everyone can see it.
Create team ownership of the data.
Use data to drive decision-making.
R. West PhD

47. Final Comments/Questions?
R. West PhD