Reliability Theory Concepts: (How to be a Coach) Delivering Uniformly Excellent and Highly Reliable Interventions Across a Healthcare System Roger Resar MD Senior Fellow Institute for Healthcare Improvement Sept 2012

© 2000 Institute for Healthcare Improvement Session Objective Learn how to evaluate improvement work for reliable design concepts

Highlights of Reliable Process Design ( See Word Document Handout for Details )

Key Questions To Analyze Testing and Implementation Key QuestionYour Evaluation Is the connection between goals and process clear? Is the design strategy primarily vigilance and hard work? Has some degree of segmentation been used to test the design? Is standard work with testing been part of the design? Is a design methodology being used? Are small tests of change being used in a rapid cycle? Is data collection rapid enough?

© 2000 Institute for Healthcare Improvement Testing : Connection Between Goals and Process Key Question: Is the connection between goals and process clear?

© 2000 Institute for Healthcare Improvement Prime Importance Designing and testing a process or a sequenced set of processes which fails to connect to the outcome goal makes no sense

© 2000 Institute for Healthcare Improvement Outcome GoalsKey DriversProcesses Decrease FallsReliable risk assessment of patients for falls Red socks for at risk patients

© 2000 Institute for Healthcare Improvement Outcome Goals Key DriversProcesses Decrease Falls Toileting in at risk patients Every 2 hour toileting rounds on at risk patients

Key Questions To Analyze Testing and Implementation Key QuestionYour Evaluation Is the connection between goals and process clear? Is the design strategy primarily vigilance and hard work? Has some degree of segmentation been used to test the design? Is standard work with testing been part of the design? Is a design methodology being used? Are small tests of change being used in a rapid cycle? Is data collection rapid enough?

© 2000 Institute for Healthcare Improvement Implementation and Spread Vigilance and Hard Work as a Design Strategy Key Question: Am I guilty of hard work and vigilance as my only design strategy?

Improvement Concepts Associated with less than 95% Performance (Primarily can be described as intent, vigilance, and hard work)  Common equipment, standard order sheets, multiple choice protocols, and written policies/procedures  Personal check lists  Feedback of information on compliance  Suggestions of working harder next time  Awareness and training

Improvement Concepts Associated with 95% Performance ( Uses human factors and reliability science to design sophisticated failure prevention, failure identification, and mitigation)  Decision aids and reminders built into the system  Desired action the default (based on scientific evidence)  Redundant processes utilized  Scheduling used in design development  Habits and patterns know and taken advantage of in the design  Standardization of process based on clear specification and articulation is the norm

© 2000 Institute for Healthcare Improvement Hard Work and Vigilance  Is a commendable trait, but not a good design strategy  Is necessary but not sufficient to attain and sustain 95% performance  Feels comfortable because we have been trained to expect hard work and vigilance in our colleagues

I would be So Bold as to Say: If human factor concepts do not make up at least 25% of the improvement effort on a given project, chances are the project is stalled or will never achieve sustained 10-2 performance

How to measure the human factor component Determine the number of tests of change involving human factors testing Study the finished design Look at the defects generated and the approach to solving them

Key Questions To Analyze Testing and Implementation Key QuestionYour Evaluation Is the connection between goals and process clear? Is the design strategy primarily vigilance and hard work? Has some degree of segmentation been used to test the design? Is standard work with testing been part of the design? Is a design methodology being used? Are small tests of change being used in a rapid cycle? Is data collection rapid enough?

© 2000 Institute for Healthcare Improvement Testing and Spread Segmentation and testing designs Key Question: Have we taken advantage of segmentation?

© 2000 Institute for Healthcare Improvement Why Segmentation is Helpful  Allows for the control of some variables  Defines the boundaries around which sequential expectations for success can be found  More likely to test the validity of the design rather than deal with barriers  Fosters a deeper understanding of the design complexity required for the project  Forces understanding of the differences between segments as design strategies  Allows the formation of more predictable timelines

Finding your first segment The segment must represent a reasonable volume The segment should have clear cut defined boundaries The segment should have willing participants so the barrier of agreeing is not a problem The segment should allow for key articulated variables or barriers to be neutralized The first segment should establish a design theme

20 Identification of Other Segments The total number of segments for a topic should not exceed 4-5 Segments should follow some theme in design (route of admission, type of physician, etc) Segments should differ by a distinct design feature The initial division of segments can be adjusted as the design is developed The segments should cover the population involved in the topic

Team Results IHI Reliability Collaborative

Key Questions To Analyze Testing and Implementation Key QuestionYour Evaluation Is the connection between goals and process clear? Is the design strategy primarily vigilance and hard work? Has some degree of segmentation been used to test the design? Is standard work with testing been part of the design? Is a design methodology being used? Are small tests of change being used in a rapid cycle? Is data collection rapid enough?

Testing and Implementation Standardization versus Standard Work Key Question: Is the team committed to really setting up standard work ?

Standard Work Standard work allows for the training of new employees and the testing of current employees If 5 front line process users can describe the process you have a good chance to achieve 95% performance and to sustain the performance over time

The Pneumovax Example Commonly “Give Pneumovax if indicated” Poorly defines a process Default is too commonly not to give the Pneumovax No testing of competency or training of new employees can occur

New Standardization Concepts Standardize to provide the appropriate infrastructure (the how, what, where, who and when) The “what” we are standardizing is based on medical evidence The “how” does not need medical evidence but rather on systems knowledge Initial standardized protocols with small time investment by experts tested at a very small scale Changes to the protocol in the initial stages should be required and encouraged Defects are studied and used to redesign the process

Key Learning Point A single standardized process within the acceptable science is superior to allowing multiple processes while we decide which is the best because it allows testing for competency and training new employees

Key Questions To Analyze Testing and Implementation Key QuestionYour Evaluation Is the connection between goals and process clear? Is the design strategy primarily vigilance and hard work? Has some degree of segmentation been used to test the design? Is standard work with testing been part of the design? Is a design methodology being used? Are small tests of change being used in a rapid cycle? Is data collection rapid enough?

The Three Step Design for Reliability Design TechniquesSteps 1-Identify the process to standardize 2-Segment the population to test the design for anomalies Prevent initial failure by standardizing the process to achieve 80% (step 1) 1-Utilize a robust HF concept to make visible failures from step 1 after step 1 has achieved 80% reliability 2-Once the failure is identified, apply an action to mitigate the failure Identify failures in step 1 and apply an action to achieve 80% for these failures (step 2) 1-Identify common failures 2-Develop a method to measure and study failures 3-Utilize knowledge of common failures to redesign either step 1 or step 2 In either step 1 and/or step 2 detect the failures, and use the knowledge from analysis of the failures to redesign (step 3)

Methodology High Points Perfection is the enemy of reliable design (step one 80%; step two 80%) People who use the design do the designing Constant testing based on what defects were observed Segmentation tests the design

Key Questions To Analyze Testing and Implementation Key QuestionYour Evaluation Is the connection between goals and process clear? Is the design strategy primarily vigilance and hard work? Has some degree of segmentation been used to test the design? Is standard work with testing been part of the design? Is a design methodology being used? Are small tests of change being used in a rapid cycle? Is data collection rapid enough?

© 2000 Institute for Healthcare Improvement Testing Rapid Cycle Change Methodology Key Question: How frequent are the cycles of testing?

© 2000 Institute for Healthcare Improvement How to Evaluate  Are testing cycles being used on an acceptable basis?  Are huddles occurring after the tests for quick redesign?  Are records being kept of the tests?  Do all team members have test responsibilities?

Critical Failure Mode Essentials A measurement of critical failure modes needs to be part of the initial design strategy Assesses the defects that occur from the current design Should be prioritized in terms of overall affect on the reliability of the process change Should be used to redesign the process

Key Questions To Analyze Testing and Implementation Key QuestionYour Evaluation Is the connection between goals and process clear? Is the design strategy primarily vigilance and hard work? Has some degree of segmentation been used to test the design? Is standard work with testing been part of the design? Is a design methodology being used? Are small tests of change being used in a rapid cycle? Is data collection rapid enough?

Testing and Implementation Data Collection and Measurement Key Question: Are small samples being collected by the team as the project is ramped up and placed on a run chart?

Measurement Small samples over time should be use to determine if the process is improving Data should be collected by the team with strict attention to the agreed upon tempo Data can initially be collected for segments Process measurements should be the primary team measures Outcome measures are needed but do not need to be collected by the team Outcome aims can be set at 0 or 100%, but your process aims should be 95% (10-2)

5 charts/day run chart

© 2000 Institute for Healthcare Improvement Testing Implementation and Spread Hitting the Wall Key Question: What should we do when the team just seems not to be making headway?

© 2000 Institute for Healthcare Improvement 1-Use the 7 Question Analysis for testing and implementation 2-Check the rules of engagement 3-Use the spread analysis

What Teams Should Expect From Leadership Clearly describe the organizational outcome goals (VAP, CLI, Mortality etc) Understand the relationship between the processes the teams are working on and the outcome goals of the organization Set process expectations for the teams all elements of the ventilator bundle will be done 95% of the time on eligible patients) Demand data to show how reliable the process has become Setting reasonable timelines If outcomes have not improved and process reliability is high provide resources to determine the “correctness of performance” of the processes

What Leaders Should Expect of Teams to Reliably Achieve the Organizations Safety Goals Expect the Initial focus of work should be on “getting the process right” with a known connection to an outcome Expect the team to take a set of processes to an agreed upon level of reliability within a specified timeline Expect the teams to use good design principles in improvement work not just hard work and vigilance Expect teams to develop good designs by using rapid cycle small tests of change Expect adequate process structure to sustain the work

Key Questions To Analyze Spread Key QuestionYour Evaluation Have you repeated the small test cycles as you spread from the initial site Is the process of spread dependent on one person Has some degree of segmentation been used to spread Has customization been allowed or encouraged Is the same team who developed the pilot now responsible for spread Have you shifted your focus from process reliability to outcomes too early