1. Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual Meeting, San Antonio, Texas

2. Disclosures No disclosures

3. Evaluation of Surgical Skill  Skills that are uniquely surgical Evidenced in OR Surgical patient management  Observed during training events Clinical (e.g. OR) Laboratory  Occurs throughout residency Progression of skills

4. Improved Task Performance

5. Proficiency-Based Lab Training

6. p > 0.006 * Skills Transfer: “VR to OR” Study 0 1 2 3 4 5 6 7 8 9 10 Mean number of errors VR Trained Standard Training *

7. Assessment of Performance in OR  Procedure specific instruments  GOALS  OSATS  Analysis of video recorded tasks  Differences between investigative instruments and assessment in frequent, routine use. Limitations on obtrusive methods Resource intensive

8. Design of New Assessment Tool User-friendly in routine use, valid, reliable. Opportunity for delivery of feedback Leverage technology to manage large volumes of data generated. Generate useful reports on resident performance.

9. Resident OR Performance Assessment at Baystate Medical Center  Objective: Develop OR performance assessment tool for routine use  OpRate: Dept. of Surgery and IS project: Multi-item global assessment of skill Attending surgeon rater Ease of use = compliance with use. Interfaced to other databases

10. Evaluated Items  3 preop (preparedness) questions Patient, technical and disease-specific knowledge  6 open technical skills questions Tissue, instrument handling, knot tying, efficient planning and ability to function independently  6 laparoscopic technical skills questions  4 point frequency-based responses for specific technical performance behaviors

13.  799 OpRate evaluations logged  579 cases reviewed  29 residents  33 attending surgeon evaluators ◦ 6 contributed 75% of assessments

14. Preop Preparedness  Cronbach’s α (3 test items) = 0.80

15. Laparoscopic Appendectomy  Cronbach’s α (4 test items) = 0.80

16. Laparoscopic Cholecystectomy  Cronbach’s α (4 test items) = 0.80

17. Colon Resection  Laparoscopic (n = 37): Cronbach’s α (4 test items) = 0.82  Open (n = 36): Cronbach’s α (6 test items) = 0.84

18. VR laparoscopic gall bladder excision (time) vs laparoscopic OR performance p = 0.0066, r2 = 0.4450 (Spearman correlation ) J Surg Res 2009;154:177-178

19. Conclusions  Experience with OpRate confirms predictions of experience-dependent resident performance increases, but does not validate attending rating behaviors.  Still Required: Systematic rater training Inter-observer agreement Concurrent validity studies (lab and OR) Mandatory compliance

20. OpRate Build 2  CQL Server database  Edit redundant items  Add important items Bimanual skills Ability to assist Depth perception in laparoscopic cases

23. Status of Use  Remediation decisions  2 SD below mean for peer group High likelihood of performance problems  Compliance to be augmented Variable compensation linked to use Compliance monitored and posted monthly Expectation that residents will advocate for use

24. Patient Management and Teamwork Skills

25.  Electrical Burns  VTE  Septic Shock  Hemorrhagic Shock  Spinal shock  Post-Op MI  Anaphylaxis FPS Curriculum  Thoracic trauma  Hyperkalemia  Intraabdominal catastrophe  Pediatric trauma  Pheochromocytoma

26.  Recognition of importance of teamwork orientation to effective clinical care Strong foundation principles Alignment with growing body of work on “non- technical skills”  Effort to take advantage of simulated patient care, simulated hospital environments (e.g. OR) Limited precedent in surgery ACS-APDS Phase III Curriculum

27.  Unique communications rating instrument  Checklisted trauma assessment  SPIKES protocol (delivering bad news)  NOTECHS  Ottawa Global Scale  Need to strengthen and consolidate assessment Assessment

28. Conclusions  Assessment of surgical skills and OR performance: Opportunities  Expanded uses: Identification of performance outliers. Definition of areas where focused training is required for specific residents. Measurement of effects of educational interventions.  Measurement of curriculum effectiveness