1. T EAM STEPPS 05.2 Mod 1 05.2 Page 1 TeamSTEPPS Measurement and Evaluation of TeamSTEPPS

2. TeamSTEPPS National Conference June 22, 2012

3. T EAM STEPPS 05.2 Mod 1 05.2 Page 3 TeamSTEPPS Session Moderator Andrea Amodeo, MS, Research Associate, IMPAQ International

4. T EAM STEPPS 05.2 Mod 1 05.2 Page 4 TeamSTEPPS Purpose Provide Information on How to Measure and Evaluate TeamSTEPPS’ Success Provide Best Practices Describe an Example

5. T EAM STEPPS 05.2 Mod 1 05.2 Page 5 TeamSTEPPS Agenda Introduction Why Measure and What to Measure: Perspectives from the C-Suite Ten Considerations for Measurement Developing a Measure of Trauma Team Performance: An Example Panel Q&A

6. T EAM STEPPS 05.2 Mod 1 05.2 Page 6 TeamSTEPPS Panel Introductions Anthony Slonim, MD, DrPH, Barnabas Health Eduardo Salas, PhD, University of Central Florida David P. Baker, PhD, IMPAQ International

7. T EAM STEPPS 05.2 Mod 1 05.2 Page 7 TeamSTEPPS Why Measure and What to Measure: Perspectives from the C-Suite Anthony Slonim, MD, DrPH Executive Vice President and Chief Medical Officer, Barnabas Health Professor, Medicine, Pediatrics, Preventive Medicine and Community Health University of Medicine and Dentistry of New Jersey New Jersey Medical School

8. T EAM STEPPS 05.2 Mod 1 05.2 Page 8 TeamSTEPPS Acknowledgments No Conflicts of Interest AHRQ Funding and Contracts: Contract # HHSA290200600019i Task Order #12 Subcontract: SAMIKE, LLC: “Proactive Risk Assessment in Ambulatory Surgery Centers” Anthony D. Slonim, MD, DrPH Principal Investigator.” Objectives Participants will understand the importance of measuring teamwork success Participants will understand what measures are important for leadership

9. T EAM STEPPS 05.2 Mod 1 05.2 Page 9 TeamSTEPPS Why Measure? To know if your TeamSTEPPS implementation worked Step 5 of Implementation Planning Assist in modifying implementation plan Planning future implementations Information for leadership Producing a scholarly product

10. T EAM STEPPS 05.2 Mod 1 05.2 Page 10 TeamSTEPPS TeamSTEPPS Phases

11. T EAM STEPPS 05.2 Mod 1 05.2 Page 11 TeamSTEPPS TeamSTEPPS Phases

12. T EAM STEPPS 05.2 Mod 1 05.2 Page 12 TeamSTEPPS TeamSTEPPS Phases Outcomes

13. T EAM STEPPS 05.2 Mod 1 05.2 Page 13 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

14. T EAM STEPPS 05.2 Mod 1 05.2 Page 14 TeamSTEPPS Teamwork Climate Provider to Patient Patient Provider To Provider Outcomes Macrosystem Microsystem

15. T EAM STEPPS 05.2 Mod 1 05.2 Page 15 TeamSTEPPS Macrosystem Safe Equitable Patient Centered Effective Timely Efficient Safe Equitable Patient Centered Microsystem Safe, Equitable, Pt Centered Effective, Timely, Efficient

16. T EAM STEPPS 05.2 Mod 1 05.2 Page 16 TeamSTEPPS How do Hospitals Work? H CEO/Management Team Medical Staff Board of Directors

17. T EAM STEPPS 05.2 Mod 1 05.2 Page 17 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

18. T EAM STEPPS 05.2 Mod 1 05.2 Page 18 TeamSTEPPS The Microsystem Team Patient Family Providers Teamwork

19. T EAM STEPPS 05.2 Mod 1 05.2 Page 19 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

20. T EAM STEPPS 05.2 Mod 1 05.2 Page 20 TeamSTEPPS Structure People Outcomes Process Interactions between: Buildings Organization Technology Providers and Patients Providers and Providers Providers and Technology Vitality Economic Quality Access Trained Providers Primary Care Services EMS Emergency Departments ICUs Specialized Equipment Regionalized Care Financing Evidence Based Practices ‘Bundles’ of Care Specialist Availability Family Centered Care Trainees Teamwork Multidiscplinary Care Mortality Cost Length of Stay Infections Complication Rates Quality of Life Slonim AD, Marcin JP, and Pollack MM. Outcomes in Pediatric Critical Care Medicine: Implications for Health Services Research and Patient Care. In: Fuhrman B and Zimmerman J. Pediatric Critical Care 4th Edition Mosby 2011. Patient

21. T EAM STEPPS 05.2 Mod 1 05.2 Page 21 TeamSTEPPS Clinical Processes Provider to Provider: Policies and Procedures Core Measures Bundle Adherence Family Centered Teamwork Availability Consistency Knowledgeable Provider to Patient: Safe Effective Outcome Based Clinical Experiential

22. T EAM STEPPS 05.2 Mod 1 05.2 Page 22 TeamSTEPPS Bedside Care Outreach Welcome, Register, Risk Assess, Triage Sales, Marketing Relationships and Transport Needed Services Care Team Introductions And Follow-up Assess, Plan And Initiate Care Direct Care Delivery Technology Care Delivery Treatment Plan And Care Delivery Deliver Care Clinical Support Services Provide Feedback and Monitor Service Evaluate Care, Outcomes and Business Slonim AD, Marcin JP, and Pollack MM. Outcomes in Pediatric Critical Care Medicine: Implications for Health Services Research and Patient Care. In: Fuhrman B and Zimmerman J. Pediatric Critical Care 4th Edition Mosby 2011.

23. T EAM STEPPS 05.2 Mod 1 05.2 Page 23 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

24. T EAM STEPPS 05.2 Mod 1 05.2 Page 24 TeamSTEPPS Culture vs. Climate Climate: Organizational Structure Connectiveness Historical/Environmental Forces Vision/Strategy Standards Accountability Behavior Communication Rewards Trust Culture: Values Beliefs Myths Norms Traditions Kennedy Group Executive Strategies: Consulting at the Kennedy Group.com

25. T EAM STEPPS 05.2 Mod 1 05.2 Page 25 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

26. T EAM STEPPS 05.2 Mod 1 05.2 Page 26 TeamSTEPPS What Leadership Cares About: An Integrated Quality Program: Content Clinical Quality: Continual Survey Readiness Core Measures + Pain Nursing and Doctor Quality Readmissions Standardized Ratios LOS Mortality PSIs/PQIs/PPCs Patient Experience: Patient Centered: HCAHPS Survey Scores Likelihood to Recommend Wait times and Pain/empathy Employee Centered: Engaged in Mission Accountable Delivering on Excellence

27. T EAM STEPPS 05.2 Mod 1 05.2 Page 27 TeamSTEPPS Practice Evaluations and Learning Activities Performance achievement High-recognize and promote Moderate-target improvement opportunities Low Graduated Counseling Education Supervision Assurance of competency Behavior modification Other areas for improvement

28. T EAM STEPPS 05.2 Mod 1 05.2 Page 28 TeamSTEPPS Teamwork Provider to Patient Climate Provider To Provider Outcomes Macrosystem Patient Microsystem

29. T EAM STEPPS 05.2 Mod 1 05.2 Page 29 TeamSTEPPS The Patient’s Responsibility Open, Honest Follows Through Therapy Recommendations Rules of engagement Asks Questions, Demands Answers

30. T EAM STEPPS 05.2 Mod 1 05.2 Page 30 TeamSTEPPS

31. T EAM STEPPS 05.2 Mod 1 05.2 Page 31 TeamSTEPPS Designing a Team Performance Measurement System: Ten Considerations Eduardo Salas, PhD Department of Psychology & Institute of Simulation & Training University of Central Florida esalas@ist.ucf.edu

32. T EAM STEPPS 05.2 Mod 1 05.2 Page 32 TeamSTEPPS To Begin… Measurement is not “Sexy, Flashy”…Yet, of the utmost importance! No “Silver Bullet”… No Perfect Tools! All need to be adapted, refined, expanded for different purposes!

33. T EAM STEPPS 05.2 Mod 1 05.2 Page 33 TeamSTEPPS 1. Consider Level of Analysis Individual MD Nurse Technician Team Emergency Department Team Radiology Team Multi-team – Team Structure Core Care Team Contingency Team (e.g., RRT) Administrative Team

34. T EAM STEPPS 05.2 Mod 1 05.2 Page 34 TeamSTEPPS 2. Clarify the Purpose Diagnose root causes of performance deficiencies Identify specific weaknesses Provide feedback Relay information regarding strengths and weaknesses as a remediation plan Assessment Evaluate the level of proficiency or readiness

35. T EAM STEPPS 05.2 Mod 1 05.2 Page 35 TeamSTEPPS 3. Decide What to Measure: Outcomes vs. Processes

36. T EAM STEPPS 05.2 Mod 1 05.2 Page 36 TeamSTEPPS 4. Select a Measure: Checklist Items/actions use dichotomous responses YES/NO RIGHT/WRONG PERFORMED/NOT PERFORMED

37. T EAM STEPPS 05.2 Mod 1 05.2 Page 37 TeamSTEPPS 4. Select a Measure: Frequency Count # of times a behavior, action, or error occurs Better for measuring acts of commission vs. acts of omission Useful when purpose = know how often a specific action is taken or task is performed Can be recorded during a critical event in an exercise or across the entire scenario

38. T EAM STEPPS 05.2 Mod 1 05.2 Page 38 TeamSTEPPS 4. Select a Measure: Graphic Rating Scale Numeric or descriptive judgment of how well a task was performed 1.The Team Leader assigned roles to the Trauma Team 2. The PGY2 used check back to confirm orders. 1 2 3 4 5 6 Ineffective Very Effective Ineffective Very Effective 1 2 3 4 5 6

39. T EAM STEPPS 05.2 Mod 1 05.2 Page 39 TeamSTEPPS 4. Select a Measure: Anchored Rating Scale

40. T EAM STEPPS 05.2 Mod 1 05.2 Page 40 TeamSTEPPS 5. Decide the Timing; When?

41. T EAM STEPPS 05.2 Mod 1 05.2 Page 41 TeamSTEPPS 6. Consider Fidelity of Setting; Where?

42. T EAM STEPPS 05.2 Mod 1 05.2 Page 42 TeamSTEPPS 7. Train Observers; Who? Choosing observers Clinical + Teamwork competence Training and supporting raters Develop and maintain high inter-rater reliability Rater training Scoring guides Developing coaches and facilitators How is data going to be used to improve performance? Facilitation skills Debrief assessment

43. T EAM STEPPS 05.2 Mod 1 05.2 Page 43 TeamSTEPPS 8. Calibrate the Measurement System: Test It!

44. T EAM STEPPS 05.2 Mod 1 05.2 Page 44 TeamSTEPPS 9. Dilemma – Generic vs. Specific Tools

45. T EAM STEPPS 05.2 Mod 1 05.2 Page 45 TeamSTEPPS 10. It is Paramount for Debriefing! To maximize learning from experience Practice alone isn’t good enough Structure practice with diagnostic feedback is required To ensure that the ‘right’ lessons are learned Everyone walks away with the same lessons learned The team’s interpretation of what happened and why it happened is cross-checked with standards To promote self-reflection and continuous learning Develop the team’s self-correction skills ‘Calibrate’ team members to rating their own performance

46. T EAM STEPPS 05.2 Mod 1 05.2 Page 46 TeamSTEPPS Remember the Issues to Consider… Why…Always keep purpose in mind! “In the end, are the questions this measurement tool could answer what I really want to know?” What…What content needs to be captured by the measurement system? Where…Where will team be assessed? Training room vs. in-situ When….when is it best to measure each competency in the scenario? Who…Who will be using this measure? What training will they have? How… What scale is best?

47. T EAM STEPPS 05.2 Mod 1 05.2 Page 47 TeamSTEPPS Developing a Measure of Trauma Team Performance: An Example David P. Baker, PhD IMPAQ International

48. T EAM STEPPS 05.2 Mod 1 05.2 Page 48 TeamSTEPPS Acknowledgements No conflicts of interest Part of Dr. Jeannette Capella’s Surgical Education and Research Fellowship Research Team Jeannette Capella, MD (PI) Andi Wright, RN Ellen Harvey, RN Sonya Ranson, PhD

49. T EAM STEPPS 05.2 Mod 1 05.2 Page 49 TeamSTEPPS Purpose To develop and test a new tool for observing and measuring team performance during trauma resuscitation Trauma Team Performance Observation Tool (TPOT) To conduct an investigation of the impact of TeamSTEPPS on: Trauma team performance Patient outcomes

50. T EAM STEPPS 05.2 Mod 1 05.2 Page 50 TeamSTEPPS Background Within the OR and trauma room, there has been a growing body of evidence validating the importance of teamwork Christian et al. (2006) A prospective study of ten general surgery cases Problems in communication, managing workload, and prioritizing competing tasks within the surgical team were found in all ten cases Few team performance evaluation tools, particularly in the areas of surgery and trauma

51. T EAM STEPPS 05.2 Mod 1 05.2 Page 51 TeamSTEPPS Methods Conducted interviews 31 trauma team members (e.g., physicians, nurses, residents) Two different organizations Goals of the interviews To identify the steps in trauma resuscitation To identify technical and team requirements that comprise each step Similar to a mini FMEA

52. T EAM STEPPS 05.2 Mod 1 05.2 Page 52 TeamSTEPPS Results PhaseTechnicalTeamwork Transport 1.EMS/rescue team or trauma team brings patient to resuscitation/trauma area 2.EMS/rescue team and/or trauma team continues ABCs: assesses patient airway, breathing, circulation, disability, and exposure/environment 3.EMS/rescue team conducts verbal handoff of patient information to trauma team (Communication) 1.Team members are quiet while EMS/rescue team gives report Primary Survey 1.Perform ABCs—this should be completed in 60-90 seconds: A: Airway (secure airway; identify problems; initiate timely intervention) B: Breathing (assess lung sounds; identify problems; initiate timely intervention) [….and much more….] (Leadership) 1.Team leader continually communicates and advocates the plan of care (Situation Monitoring) 1.Team members know their role and responsibilities 2.Team member(s) prepare(s) patient and/or equipment

53. T EAM STEPPS 05.2 Mod 1 05.2 Page 53 TeamSTEPPS Methods An initial pool of items was developed for the TPOT through an extensive item writing process Items were linked to the following four team constructs: Leadership Situation monitoring Mutual support Communication

54. T EAM STEPPS 05.2 Mod 1 05.2 Page 54 TeamSTEPPS Results PhaseTechnicalTeamwork Transport 1.EMS/rescue team or trauma team brings patient to resuscitation/trauma area 2.EMS/rescue team and/or trauma team continues ABCs: assesses patient airway, breathing, circulation, disability, and exposure/environment 3.EMS/rescue team conducts verbal handoff of patient information to trauma team (Communication) 1.Team members are quiet while EMS/rescue team gives report Primary Survey 1.Perform ABCs—this should be completed in 60-90 seconds: A: Airway (secure airway; identify problems; initiate timely intervention) B: Breathing (assess lung sounds; identify problems; initiate timely intervention) [….and much more….] (Leadership) 1.Team leader continually communicates and advocates the plan of care (Situation Monitoring) 1.Team members know their role and responsibilities 2.Team member(s) prepare(s) patient and/or equipment

55. T EAM STEPPS 05.2 Mod 1 05.2 Page 55 TeamSTEPPS Leadership – The team leader ….. 1. Conducts a brief prior to patient arrival (e.g., identifies self, assigns members roles and responsibilities, discusses initial plan based on current information, anticipates interventions [e.g., chest tube, OR]) 2. Continually renders plan of care to team 3. Feedback provided to team members is constructive 4. Ensures task prioritization (e.g., important tasks performed first, ABC’s and survey sequence are being completed) 5. Asks non-response team members to leave when they are distracting

56. T EAM STEPPS 05.2 Mod 1 05.2 Page 56 TeamSTEPPS Situation Monitoring – Team members …. 1. Prepare equipment before patient arrival (e.g., set up IV, ultrasound machine, suction) 2. Work quickly and efficiently 3. Conduct tasks in right order 4. Are not distracted by major injuries 5. Ensure that NEW team members perform expected role and responsibilities 6. Adapt quickly and efficiently to deterioration of patient’s condition (e.g., decreased O2 sats, decreased blood pressure, decreased mental status)

57. T EAM STEPPS 05.2 Mod 1 05.2 Page 57 TeamSTEPPS Mutual Support – Team members …. 1. Feedback provided to other team members is constructive 2. Assist when moving patient to next unit (e.g., CT scanner, OR, ICU) 3. Provide assistance when needed/Complete other team members’ tasks 4. Identify/Call out when patient safety issue is suspected

58. T EAM STEPPS 05.2 Mod 1 05.2 Page 58 TeamSTEPPS Communication – Team members …. 1. Remain quiet while team gives report 2. Request additional information from EMS (e.g., medications given, vital signs, mechanism of injury) 3. Use call-outs to share important patient information (i.e., Team leader “Airway status?” Airway doc responds “Airway clear!”) 4. Use check-backs to verify important information is exchanged (i.e., Doctor “Give 25 mg Benadryl IV.” Nurse “25 mg Benadryl IV” to confirm. Doctor “That’s correct”) 5. Use clear and concise language 6. Request information from others when it’s not readily shared

59. T EAM STEPPS 05.2 Mod 1 05.2 Page 59 TeamSTEPPS Scoring Rating Scale 12345NA Very poor PoorAverageGoodExcellentNot applicable Very poorly done Poorly done Acceptable performance Good performance Perfect Performance Did not need to be done Should have been done but was not Should have been done more often Could have been done more often/ consistently but is acceptable as is Done most of the time Done at all times appropriately Was not done and did not need to be done

60. T EAM STEPPS 05.2 Mod 1 05.2 Page 60 TeamSTEPPS Observer Training Five staff (two trauma nurses and three trauma registrars) were trained to use the TPOT Training involved four steps 1. Reviewing and discussing the TPOT 2. Practice and feedback rating the videotapes 3. Revising the TPOT items for clarity, as needed 4. Independent evaluation of the videotapes

61. T EAM STEPPS 05.2 Mod 1 05.2 Page 61 TeamSTEPPS Rater Agreement ScenarioTeamMeanICC Inter-Rater Agreement 112.67.4465% 211.22.6482% 122.98.4769% 221.21.5886%

62. T EAM STEPPS 05.2 Mod 1 05.2 Page 62 TeamSTEPPS Implementation Part of a larger study to assess the impact of TeamSTEPPS Trained raters Observed and rated 33 live trauma resuscitations in the trauma bay over a 3-month period (Pre- Training) Observed and rated 40 live trauma resuscitations in the trauma bay over a 3-month period (Post- Training)

63. T EAM STEPPS 05.2 Mod 1 05.2 Page 63 TeamSTEPPS Alphas and Inter-correlations Team SkillsLDRSMMSCOMMeanStdev Leadership2.90.68 Situation Monitoring.923.29.62 Mutual Support.75 3.88.47 Communication.85.82.702.92.58 Alpha.53.57.64.63

64. T EAM STEPPS 05.2 Mod 1 05.2 Page 64 TeamSTEPPS Impact Study Design Quasi-experiment Pre-test/Post-test, no control group design Interventions Training – Didactic and Simulation 2-hour TeamSTEPPS Essentials; 2 hours simulation (skills practice and feedback) Nurses and Doctors separately trained Trauma Room Roles and Responsibilities Policy Briefing, STEP, CUS, Call-Outs and Check-Backs One-day Nursing Crash Course

65. T EAM STEPPS 05.2 Mod 1 05.2 Page 65 TeamSTEPPS Measures (Kirkpatrick) Level I Reactions End of Training Level II Learning – Pre/Post Test Knowledge, Attitudes, Skills Level III Transfer – Observed Teamwork (TPOT) 3 months pre and post-training Level IV Outcomes Clinical Outcomes (ICU LOS, Hospital LOS, Complication Rate, Mortality) Clinical Process (Time to CT SCAN, Surgery, Intubation)

66. T EAM STEPPS 05.2 Mod 1 05.2 Page 66 TeamSTEPPS Expected Findings Reactions - Positive Learning Knowledge No change – staff know what to do Attitudes Positive – Believe in teamwork or social desirability Skills Significant improvement in simulator Transfer Not sure about trauma bay – many environmental factors Outcomes Unlikely due to base rate issues

67. T EAM STEPPS 05.2 Mod 1 05.2 Page 67 TeamSTEPPS Descriptive Data Pre-TrainingPost-Training Observed3340 Trauma176263 ISS M=13.97, SD=11.85 M=11.63; SD=11.04

68. T EAM STEPPS 05.2 Mod 1 05.2 Page 68 TeamSTEPPS Level III Transfer Pre-training N=33 Post-training N=40 p value Leadership2.873.460.003 Situation monitoring3.303.910.009 Mutual support3.403.960.004 Communication2.903.460.001 Total3.123.70<0.001 Note: Pre-training and post-training reflect observations in trauma bay by 4 trained raters.

69. T EAM STEPPS 05.2 Mod 1 05.2 Page 69 TeamSTEPPS

70. T EAM STEPPS 05.2 Mod 1 05.2 Page 70 TeamSTEPPS Challenges Measurement How best to train observers? Moderate reliability Capella & Baker, ACOS Grant to study different training strategies Available tools Had to develop tools, few available, few validated

71. T EAM STEPPS 05.2 Mod 1 05.2 Page 71 TeamSTEPPS Thank You! For more information, please contact our team at: dbaker@impaqint.com