1. Evaluating Surgical Skills And Operating Room Performance: Education/Remediation? Certification/Credentialling? John J. Ferrara MD Kanav Kahol PhD Phoenix Integrated Surgical Residency

2. Evaluating Surgical Skills Challenges  How to maintain cardinal surgical “art and science” traditions when the sands that support educational paradigms are shifting?  “Publish or perish” to “Produce (RVU’s) or perish”  “Duty” hours  “Public” opinion  Generational chasm  “Linear” educational construct

3. Generation X: The Bridge Boomers (46-64 years)  Defined by work ethic  Independent  Religious  Financial success  Career-driven  Wanna be lead dogs  Kumbaya  Consumer-driven/TV Millennials (18-29 years)  Defined by technology  Social agenda  Secular  Parenthood (non-traditional)  Time-driven  Lead only if asked  Blunt  Under-consumers/TV? The good news: they respect (boomerang back to) their elders

4. Technical Skills Evaluation Linear Construct Technical Skills Evaluation Simulation Environment Instrument Parameters Evaluation Hand Movement Evaluation Objective Evaluations: FLS/Endoscopy Skills “Real” EnvironmentOSATSSubjective Evaluation

5. Technology Simulation Environment “Real” Environment Technical Skills Evaluation Parallel Construct Leveraged Scalable Adaptable Integrative

6. Goals  Measure Technical Skills in a Simulated Environment  Create a system to measure skill set and to provide immediate feedback to the user  “Battleship down”  Measure Technical Skills in the Operating Room  Develop and validate a system to analyze videos of operations submitted to a panel for assessment

8. Objective Proficiency Measures  Employ neurological and kinesiological features to analyze task (surgical) proficiency  Construct task decomposition based feedback system  Breaks complex motion into simpler units that are:  Easy to analyze  Easy to comprehend  Easy to modify by the user Expert Intermediate Novice Instrument movements Rosen 2002

9. Hand Motion

10. Motorical Chunking Measure of Expertise ExpertNovice

11. Dynamic Virtual Reality Systems for Cognitive Training  Train residents for attention, working memory, intermodal transfer  Modify technique simulators to include a cognitive layer  Treat surgery as a combination of psychomotor and cognitive skill Original Task (Laparoscopic Training) Modified to target working memory

12. Marble Mania High (0.92) correlation with basic surgical gestures Fine motor skills based game Hand motions similar to laparoscopy

13. Marble Mania

14. CyberGlove Analysis Non-Dominant HandDominant HandMarble Mania

15. Ambidexterity

16. Technical Proficiency on ProMIS 5.9 3.9 6.5 2.5 6.3 1.0 2.9

17. Novices Intermediates Experts Masters Skills Evaluation

18. Measuring Skills in the “Real” Environment Proposed Solution  Computer vision instrument automatically analyzes videos  Develop means/ranges/standard deviations  Set “minimal” performance grade  Benchmarking?  Picks up events the naked eye misses  Detailed movement analysis  Cheap, “portable”, time-efficient  Web based access to rate videos for experts  Web based training tools to train experts to rate videos

19. Video Capture Laparoscopy Basic apparatus Video capture system for laparoscopic system and hand movements Hand movements captured by external camera Sites: ceiling/lighting system/tripod De-identified videos Our system “syncs” these two streams for presentation and analysis

20. Dual Capture System

21. Skills Evaluation ExpertIntermediate Novice Tremor Instrument Path Inefficiency Between Groups p<0.05 Between Groups P<0.05 Expert v Novice P<0.05

22. Benchmarking?

23. Web-Based Training  Upload/automatically analyze videos on www.ratethesurgeons.com www.ratethesurgeons.com  Experts view videos off-site  Can provide input/feedback  Novice raters View expert ratings Receive instruction to become proficient raters Reward system: pair teaching

24. Correlation of Subjective Measures with Various Objective Measures 1.0 0.4

25. Validation R=0.93 p<0.05 Experts Intermediates Novices

26. Where We are Now  Validation of the technical analysis tool  Evaluation on simulators also being done with videos

27. Future Work  Enhance Database  Develop Benchmarks  Expand Skill Set Instrument Family  Patient Care Applications

28. New Simulation Tasks

29. Motion History Images

30. Virtual World “Acute Care Surgery” Training

31. Challenges “The Uncanny Valley” Masahiro Mori (1970) Avatar

32. Challenges The Simulation Perfect Storm  Conventional computing is dead, and with it, the first generation (six figure) simulators  Computing life measured in months Core processors Naturalistic computing Gaming consoles  How to maintain a database when evaluation instruments are constantly changing?

33. Conclusions  We (all) need help  We have no magic bullet  We need genomic variation “The Two Word Definition of Dogma is Brain Dead” Zollinger (sometime during my residency)

35. Challenges Engineers

36. Clinicians

37. Video Capture Basic apparatus Video capture system for laparoscopic system and hand movements Hand movements captured by external camera Sites: ceiling/lighting system/tripod De-identified videos Our system “syncs” these two streams for presentation and analysis Mobile simulator unit

38. We are becoming increasingly challenged with teaching new dogs old tricks AND We are not very good at teaching old dogs new tricks Evaluation Poses a More Daunting Challenge

39. Analysis Basic movement tracking algorithms from computer vision, an established field with myriad algorithms to track movements and predict efficacy Proprietary state of the art tools analyze movements