T EAM STEPPS 05.2 Mod Page 1 TeamSTEPPS Measurement and Evaluation of TeamSTEPPS

TeamSTEPPS National Conference June 22, 2012

T EAM STEPPS 05.2 Mod Page 3 TeamSTEPPS Session Moderator Andrea Amodeo, MS, Research Associate, IMPAQ International

T EAM STEPPS 05.2 Mod Page 4 TeamSTEPPS Purpose Provide Information on How to Measure and Evaluate TeamSTEPPS’ Success Provide Best Practices Describe an Example

T EAM STEPPS 05.2 Mod Page 5 TeamSTEPPS Agenda Introduction Why Measure and What to Measure: Perspectives from the C-Suite Ten Considerations for Measurement Developing a Measure of Trauma Team Performance: An Example Panel Q&A

T EAM STEPPS 05.2 Mod Page 6 TeamSTEPPS Panel Introductions Anthony Slonim, MD, DrPH, Barnabas Health Eduardo Salas, PhD, University of Central Florida David P. Baker, PhD, IMPAQ International

T EAM STEPPS 05.2 Mod Page 7 TeamSTEPPS Why Measure and What to Measure: Perspectives from the C-Suite Anthony Slonim, MD, DrPH Executive Vice President and Chief Medical Officer, Barnabas Health Professor, Medicine, Pediatrics, Preventive Medicine and Community Health University of Medicine and Dentistry of New Jersey New Jersey Medical School

T EAM STEPPS 05.2 Mod Page 8 TeamSTEPPS Acknowledgments No Conflicts of Interest AHRQ Funding and Contracts: Contract # HHSA i Task Order #12 Subcontract: SAMIKE, LLC: “Proactive Risk Assessment in Ambulatory Surgery Centers” Anthony D. Slonim, MD, DrPH Principal Investigator.” Objectives Participants will understand the importance of measuring teamwork success Participants will understand what measures are important for leadership

T EAM STEPPS 05.2 Mod Page 9 TeamSTEPPS Why Measure? To know if your TeamSTEPPS implementation worked Step 5 of Implementation Planning Assist in modifying implementation plan Planning future implementations Information for leadership Producing a scholarly product

T EAM STEPPS 05.2 Mod Page 10 TeamSTEPPS TeamSTEPPS Phases

T EAM STEPPS 05.2 Mod Page 11 TeamSTEPPS TeamSTEPPS Phases

T EAM STEPPS 05.2 Mod Page 12 TeamSTEPPS TeamSTEPPS Phases Outcomes

T EAM STEPPS 05.2 Mod Page 13 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

T EAM STEPPS 05.2 Mod Page 14 TeamSTEPPS Teamwork Climate Provider to Patient Patient Provider To Provider Outcomes Macrosystem Microsystem

T EAM STEPPS 05.2 Mod Page 15 TeamSTEPPS Macrosystem Safe Equitable Patient Centered Effective Timely Efficient Safe Equitable Patient Centered Microsystem Safe, Equitable, Pt Centered Effective, Timely, Efficient

T EAM STEPPS 05.2 Mod Page 16 TeamSTEPPS How do Hospitals Work? H CEO/Management Team Medical Staff Board of Directors

T EAM STEPPS 05.2 Mod Page 17 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

T EAM STEPPS 05.2 Mod Page 18 TeamSTEPPS The Microsystem Team Patient Family Providers Teamwork

T EAM STEPPS 05.2 Mod Page 19 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

T EAM STEPPS 05.2 Mod Page 20 TeamSTEPPS Structure People Outcomes Process Interactions between: Buildings Organization Technology Providers and Patients Providers and Providers Providers and Technology Vitality Economic Quality Access Trained Providers Primary Care Services EMS Emergency Departments ICUs Specialized Equipment Regionalized Care Financing Evidence Based Practices ‘Bundles’ of Care Specialist Availability Family Centered Care Trainees Teamwork Multidiscplinary Care Mortality Cost Length of Stay Infections Complication Rates Quality of Life Slonim AD, Marcin JP, and Pollack MM. Outcomes in Pediatric Critical Care Medicine: Implications for Health Services Research and Patient Care. In: Fuhrman B and Zimmerman J. Pediatric Critical Care 4th Edition Mosby Patient

T EAM STEPPS 05.2 Mod Page 21 TeamSTEPPS Clinical Processes Provider to Provider: Policies and Procedures Core Measures Bundle Adherence Family Centered Teamwork Availability Consistency Knowledgeable Provider to Patient: Safe Effective Outcome Based Clinical Experiential

T EAM STEPPS 05.2 Mod Page 22 TeamSTEPPS Bedside Care Outreach Welcome, Register, Risk Assess, Triage Sales, Marketing Relationships and Transport Needed Services Care Team Introductions And Follow-up Assess, Plan And Initiate Care Direct Care Delivery Technology Care Delivery Treatment Plan And Care Delivery Deliver Care Clinical Support Services Provide Feedback and Monitor Service Evaluate Care, Outcomes and Business Slonim AD, Marcin JP, and Pollack MM. Outcomes in Pediatric Critical Care Medicine: Implications for Health Services Research and Patient Care. In: Fuhrman B and Zimmerman J. Pediatric Critical Care 4th Edition Mosby 2011.

T EAM STEPPS 05.2 Mod Page 23 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

T EAM STEPPS 05.2 Mod Page 24 TeamSTEPPS Culture vs. Climate Climate: Organizational Structure Connectiveness Historical/Environmental Forces Vision/Strategy Standards Accountability Behavior Communication Rewards Trust Culture: Values Beliefs Myths Norms Traditions Kennedy Group Executive Strategies: Consulting at the Kennedy Group.com

T EAM STEPPS 05.2 Mod Page 25 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

T EAM STEPPS 05.2 Mod Page 26 TeamSTEPPS What Leadership Cares About: An Integrated Quality Program: Content Clinical Quality: Continual Survey Readiness Core Measures + Pain Nursing and Doctor Quality Readmissions Standardized Ratios LOS Mortality PSIs/PQIs/PPCs Patient Experience: Patient Centered: HCAHPS Survey Scores Likelihood to Recommend Wait times and Pain/empathy Employee Centered: Engaged in Mission Accountable Delivering on Excellence

T EAM STEPPS 05.2 Mod Page 27 TeamSTEPPS Practice Evaluations and Learning Activities Performance achievement High-recognize and promote Moderate-target improvement opportunities Low Graduated Counseling Education Supervision Assurance of competency Behavior modification Other areas for improvement

T EAM STEPPS 05.2 Mod Page 28 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

T EAM STEPPS 05.2 Mod Page 29 TeamSTEPPS The Patient’s Responsibility Open, Honest Follows Through Therapy Recommendations Rules of engagement Asks Questions, Demands Answers

T EAM STEPPS 05.2 Mod Page 30 TeamSTEPPS

T EAM STEPPS 05.2 Mod Page 31 TeamSTEPPS Designing a Team Performance Measurement System: Ten Considerations Eduardo Salas, PhD Department of Psychology & Institute of Simulation & Training University of Central Florida

T EAM STEPPS 05.2 Mod Page 32 TeamSTEPPS To Begin… Measurement is not “Sexy, Flashy”…Yet, of the utmost importance! No “Silver Bullet”… No Perfect Tools! All need to be adapted, refined, expanded for different purposes!

T EAM STEPPS 05.2 Mod Page 33 TeamSTEPPS 1. Consider Level of Analysis Individual MD Nurse Technician Team Emergency Department Team Radiology Team Multi-team – Team Structure Core Care Team Contingency Team (e.g., RRT) Administrative Team

T EAM STEPPS 05.2 Mod Page 34 TeamSTEPPS 2. Clarify the Purpose Diagnose root causes of performance deficiencies Identify specific weaknesses Provide feedback Relay information regarding strengths and weaknesses as a remediation plan Assessment Evaluate the level of proficiency or readiness

T EAM STEPPS 05.2 Mod Page 35 TeamSTEPPS 3. Decide What to Measure: Outcomes vs. Processes

T EAM STEPPS 05.2 Mod Page 36 TeamSTEPPS 4. Select a Measure: Checklist Items/actions use dichotomous responses YES/NO RIGHT/WRONG PERFORMED/NOT PERFORMED

T EAM STEPPS 05.2 Mod Page 37 TeamSTEPPS 4. Select a Measure: Frequency Count # of times a behavior, action, or error occurs Better for measuring acts of commission vs. acts of omission Useful when purpose = know how often a specific action is taken or task is performed Can be recorded during a critical event in an exercise or across the entire scenario

T EAM STEPPS 05.2 Mod Page 38 TeamSTEPPS 4. Select a Measure: Graphic Rating Scale Numeric or descriptive judgment of how well a task was performed 1.The Team Leader assigned roles to the Trauma Team 2. The PGY2 used check back to confirm orders Ineffective Very Effective Ineffective Very Effective

T EAM STEPPS 05.2 Mod Page 39 TeamSTEPPS 4. Select a Measure: Anchored Rating Scale

T EAM STEPPS 05.2 Mod Page 40 TeamSTEPPS 5. Decide the Timing; When?

T EAM STEPPS 05.2 Mod Page 41 TeamSTEPPS 6. Consider Fidelity of Setting; Where?

T EAM STEPPS 05.2 Mod Page 42 TeamSTEPPS 7. Train Observers; Who? Choosing observers Clinical + Teamwork competence Training and supporting raters Develop and maintain high inter-rater reliability Rater training Scoring guides Developing coaches and facilitators How is data going to be used to improve performance? Facilitation skills Debrief assessment

T EAM STEPPS 05.2 Mod Page 43 TeamSTEPPS 8. Calibrate the Measurement System: Test It!

T EAM STEPPS 05.2 Mod Page 44 TeamSTEPPS 9. Dilemma – Generic vs. Specific Tools

T EAM STEPPS 05.2 Mod Page 45 TeamSTEPPS 10. It is Paramount for Debriefing! To maximize learning from experience Practice alone isn’t good enough Structure practice with diagnostic feedback is required To ensure that the ‘right’ lessons are learned Everyone walks away with the same lessons learned The team’s interpretation of what happened and why it happened is cross-checked with standards To promote self-reflection and continuous learning Develop the team’s self-correction skills ‘Calibrate’ team members to rating their own performance

T EAM STEPPS 05.2 Mod Page 46 TeamSTEPPS Remember the Issues to Consider… Why…Always keep purpose in mind! “In the end, are the questions this measurement tool could answer what I really want to know?” What…What content needs to be captured by the measurement system? Where…Where will team be assessed? Training room vs. in-situ When….when is it best to measure each competency in the scenario? Who…Who will be using this measure? What training will they have? How… What scale is best?

T EAM STEPPS 05.2 Mod Page 47 TeamSTEPPS Developing a Measure of Trauma Team Performance: An Example David P. Baker, PhD IMPAQ International

T EAM STEPPS 05.2 Mod Page 48 TeamSTEPPS Acknowledgements No conflicts of interest Part of Dr. Jeannette Capella’s Surgical Education and Research Fellowship Research Team Jeannette Capella, MD (PI) Andi Wright, RN Ellen Harvey, RN Sonya Ranson, PhD

T EAM STEPPS 05.2 Mod Page 49 TeamSTEPPS Purpose To develop and test a new tool for observing and measuring team performance during trauma resuscitation Trauma Team Performance Observation Tool (TPOT) To conduct an investigation of the impact of TeamSTEPPS on: Trauma team performance Patient outcomes

T EAM STEPPS 05.2 Mod Page 50 TeamSTEPPS Background Within the OR and trauma room, there has been a growing body of evidence validating the importance of teamwork Christian et al. (2006) A prospective study of ten general surgery cases Problems in communication, managing workload, and prioritizing competing tasks within the surgical team were found in all ten cases Few team performance evaluation tools, particularly in the areas of surgery and trauma

T EAM STEPPS 05.2 Mod Page 51 TeamSTEPPS Methods Conducted interviews 31 trauma team members (e.g., physicians, nurses, residents) Two different organizations Goals of the interviews To identify the steps in trauma resuscitation To identify technical and team requirements that comprise each step Similar to a mini FMEA

T EAM STEPPS 05.2 Mod Page 52 TeamSTEPPS Results PhaseTechnicalTeamwork Transport 1.EMS/rescue team or trauma team brings patient to resuscitation/trauma area 2.EMS/rescue team and/or trauma team continues ABCs: assesses patient airway, breathing, circulation, disability, and exposure/environment 3.EMS/rescue team conducts verbal handoff of patient information to trauma team (Communication) 1.Team members are quiet while EMS/rescue team gives report Primary Survey 1.Perform ABCs—this should be completed in seconds: A: Airway (secure airway; identify problems; initiate timely intervention) B: Breathing (assess lung sounds; identify problems; initiate timely intervention) [….and much more….] (Leadership) 1.Team leader continually communicates and advocates the plan of care (Situation Monitoring) 1.Team members know their role and responsibilities 2.Team member(s) prepare(s) patient and/or equipment

T EAM STEPPS 05.2 Mod Page 53 TeamSTEPPS Methods An initial pool of items was developed for the TPOT through an extensive item writing process Items were linked to the following four team constructs: Leadership Situation monitoring Mutual support Communication

T EAM STEPPS 05.2 Mod Page 54 TeamSTEPPS Results PhaseTechnicalTeamwork Transport 1.EMS/rescue team or trauma team brings patient to resuscitation/trauma area 2.EMS/rescue team and/or trauma team continues ABCs: assesses patient airway, breathing, circulation, disability, and exposure/environment 3.EMS/rescue team conducts verbal handoff of patient information to trauma team (Communication) 1.Team members are quiet while EMS/rescue team gives report Primary Survey 1.Perform ABCs—this should be completed in seconds: A: Airway (secure airway; identify problems; initiate timely intervention) B: Breathing (assess lung sounds; identify problems; initiate timely intervention) [….and much more….] (Leadership) 1.Team leader continually communicates and advocates the plan of care (Situation Monitoring) 1.Team members know their role and responsibilities 2.Team member(s) prepare(s) patient and/or equipment

T EAM STEPPS 05.2 Mod Page 55 TeamSTEPPS Leadership – The team leader ….. 1. Conducts a brief prior to patient arrival (e.g., identifies self, assigns members roles and responsibilities, discusses initial plan based on current information, anticipates interventions [e.g., chest tube, OR]) 2. Continually renders plan of care to team 3. Feedback provided to team members is constructive 4. Ensures task prioritization (e.g., important tasks performed first, ABC’s and survey sequence are being completed) 5. Asks non-response team members to leave when they are distracting

T EAM STEPPS 05.2 Mod Page 56 TeamSTEPPS Situation Monitoring – Team members …. 1. Prepare equipment before patient arrival (e.g., set up IV, ultrasound machine, suction) 2. Work quickly and efficiently 3. Conduct tasks in right order 4. Are not distracted by major injuries 5. Ensure that NEW team members perform expected role and responsibilities 6. Adapt quickly and efficiently to deterioration of patient’s condition (e.g., decreased O2 sats, decreased blood pressure, decreased mental status)

T EAM STEPPS 05.2 Mod Page 57 TeamSTEPPS Mutual Support – Team members …. 1. Feedback provided to other team members is constructive 2. Assist when moving patient to next unit (e.g., CT scanner, OR, ICU) 3. Provide assistance when needed/Complete other team members’ tasks 4. Identify/Call out when patient safety issue is suspected

T EAM STEPPS 05.2 Mod Page 58 TeamSTEPPS Communication – Team members …. 1. Remain quiet while team gives report 2. Request additional information from EMS (e.g., medications given, vital signs, mechanism of injury) 3. Use call-outs to share important patient information (i.e., Team leader “Airway status?” Airway doc responds “Airway clear!”) 4. Use check-backs to verify important information is exchanged (i.e., Doctor “Give 25 mg Benadryl IV.” Nurse “25 mg Benadryl IV” to confirm. Doctor “That’s correct”) 5. Use clear and concise language 6. Request information from others when it’s not readily shared

T EAM STEPPS 05.2 Mod Page 59 TeamSTEPPS Scoring Rating Scale 12345NA Very poor PoorAverageGoodExcellentNot applicable Very poorly done Poorly done Acceptable performance Good performance Perfect Performance Did not need to be done Should have been done but was not Should have been done more often Could have been done more often/ consistently but is acceptable as is Done most of the time Done at all times appropriately Was not done and did not need to be done

T EAM STEPPS 05.2 Mod Page 60 TeamSTEPPS Observer Training Five staff (two trauma nurses and three trauma registrars) were trained to use the TPOT Training involved four steps 1. Reviewing and discussing the TPOT 2. Practice and feedback rating the videotapes 3. Revising the TPOT items for clarity, as needed 4. Independent evaluation of the videotapes

T EAM STEPPS 05.2 Mod Page 61 TeamSTEPPS Rater Agreement ScenarioTeamMeanICC Inter-Rater Agreement % % % %

T EAM STEPPS 05.2 Mod Page 62 TeamSTEPPS Implementation Part of a larger study to assess the impact of TeamSTEPPS Trained raters Observed and rated 33 live trauma resuscitations in the trauma bay over a 3-month period (Pre- Training) Observed and rated 40 live trauma resuscitations in the trauma bay over a 3-month period (Post- Training)

T EAM STEPPS 05.2 Mod Page 63 TeamSTEPPS Alphas and Inter-correlations Team SkillsLDRSMMSCOMMeanStdev Leadership Situation Monitoring Mutual Support Communication Alpha

T EAM STEPPS 05.2 Mod Page 64 TeamSTEPPS Impact Study Design Quasi-experiment Pre-test/Post-test, no control group design Interventions Training – Didactic and Simulation 2-hour TeamSTEPPS Essentials; 2 hours simulation (skills practice and feedback) Nurses and Doctors separately trained Trauma Room Roles and Responsibilities Policy Briefing, STEP, CUS, Call-Outs and Check-Backs One-day Nursing Crash Course

T EAM STEPPS 05.2 Mod Page 65 TeamSTEPPS Measures (Kirkpatrick) Level I Reactions End of Training Level II Learning – Pre/Post Test Knowledge, Attitudes, Skills Level III Transfer – Observed Teamwork (TPOT) 3 months pre and post-training Level IV Outcomes Clinical Outcomes (ICU LOS, Hospital LOS, Complication Rate, Mortality) Clinical Process (Time to CT SCAN, Surgery, Intubation)

T EAM STEPPS 05.2 Mod Page 66 TeamSTEPPS Expected Findings Reactions - Positive Learning Knowledge No change – staff know what to do Attitudes Positive – Believe in teamwork or social desirability Skills Significant improvement in simulator Transfer Not sure about trauma bay – many environmental factors Outcomes Unlikely due to base rate issues

T EAM STEPPS 05.2 Mod Page 67 TeamSTEPPS Descriptive Data Pre-TrainingPost-Training Observed3340 Trauma ISS M=13.97, SD=11.85 M=11.63; SD=11.04

T EAM STEPPS 05.2 Mod Page 68 TeamSTEPPS Level III Transfer Pre-training N=33 Post-training N=40 p value Leadership Situation monitoring Mutual support Communication Total <0.001 Note: Pre-training and post-training reflect observations in trauma bay by 4 trained raters.

T EAM STEPPS 05.2 Mod Page 69 TeamSTEPPS

T EAM STEPPS 05.2 Mod Page 70 TeamSTEPPS Challenges Measurement How best to train observers? Moderate reliability Capella & Baker, ACOS Grant to study different training strategies Available tools Had to develop tools, few available, few validated

T EAM STEPPS 05.2 Mod Page 71 TeamSTEPPS Thank You! For more information, please contact our team at: