Evaluating Surgical Skills And Operating Room Performance: Education/Remediation? Certification/Credentialling? John J. Ferrara MD Kanav Kahol PhD Phoenix Integrated Surgical Residency

Evaluating Surgical Skills Challenges  How to maintain cardinal surgical “art and science” traditions when the sands that support educational paradigms are shifting?  “Publish or perish” to “Produce (RVU’s) or perish”  “Duty” hours  “Public” opinion  Generational chasm  “Linear” educational construct

Generation X: The Bridge Boomers (46-64 years)  Defined by work ethic  Independent  Religious  Financial success  Career-driven  Wanna be lead dogs  Kumbaya  Consumer-driven/TV Millennials (18-29 years)  Defined by technology  Social agenda  Secular  Parenthood (non-traditional)  Time-driven  Lead only if asked  Blunt  Under-consumers/TV? The good news: they respect (boomerang back to) their elders

Technical Skills Evaluation Linear Construct Technical Skills Evaluation Simulation Environment Instrument Parameters Evaluation Hand Movement Evaluation Objective Evaluations: FLS/Endoscopy Skills “Real” EnvironmentOSATSSubjective Evaluation

Technology Simulation Environment “Real” Environment Technical Skills Evaluation Parallel Construct Leveraged Scalable Adaptable Integrative

Goals  Measure Technical Skills in a Simulated Environment  Create a system to measure skill set and to provide immediate feedback to the user  “Battleship down”  Measure Technical Skills in the Operating Room  Develop and validate a system to analyze videos of operations submitted to a panel for assessment

Objective Proficiency Measures  Employ neurological and kinesiological features to analyze task (surgical) proficiency  Construct task decomposition based feedback system  Breaks complex motion into simpler units that are:  Easy to analyze  Easy to comprehend  Easy to modify by the user Expert Intermediate Novice Instrument movements Rosen 2002

Hand Motion

Motorical Chunking Measure of Expertise ExpertNovice

Dynamic Virtual Reality Systems for Cognitive Training  Train residents for attention, working memory, intermodal transfer  Modify technique simulators to include a cognitive layer  Treat surgery as a combination of psychomotor and cognitive skill Original Task (Laparoscopic Training) Modified to target working memory

Marble Mania High (0.92) correlation with basic surgical gestures Fine motor skills based game Hand motions similar to laparoscopy

Marble Mania

CyberGlove Analysis Non-Dominant HandDominant HandMarble Mania

Ambidexterity

Technical Proficiency on ProMIS

Novices Intermediates Experts Masters Skills Evaluation

Measuring Skills in the “Real” Environment Proposed Solution  Computer vision instrument automatically analyzes videos  Develop means/ranges/standard deviations  Set “minimal” performance grade  Benchmarking?  Picks up events the naked eye misses  Detailed movement analysis  Cheap, “portable”, time-efficient  Web based access to rate videos for experts  Web based training tools to train experts to rate videos

Video Capture Laparoscopy Basic apparatus Video capture system for laparoscopic system and hand movements Hand movements captured by external camera Sites: ceiling/lighting system/tripod De-identified videos Our system “syncs” these two streams for presentation and analysis

Dual Capture System

Skills Evaluation ExpertIntermediate Novice Tremor Instrument Path Inefficiency Between Groups p<0.05 Between Groups P<0.05 Expert v Novice P<0.05

Benchmarking?

Web-Based Training  Upload/automatically analyze videos on  Experts view videos off-site  Can provide input/feedback  Novice raters View expert ratings Receive instruction to become proficient raters Reward system: pair teaching

Correlation of Subjective Measures with Various Objective Measures

Validation R=0.93 p<0.05 Experts Intermediates Novices

Where We are Now  Validation of the technical analysis tool  Evaluation on simulators also being done with videos

Future Work  Enhance Database  Develop Benchmarks  Expand Skill Set Instrument Family  Patient Care Applications

New Simulation Tasks

Motion History Images

Virtual World “Acute Care Surgery” Training

Challenges “The Uncanny Valley” Masahiro Mori (1970) Avatar

Challenges The Simulation Perfect Storm  Conventional computing is dead, and with it, the first generation (six figure) simulators  Computing life measured in months Core processors Naturalistic computing Gaming consoles  How to maintain a database when evaluation instruments are constantly changing?

Conclusions  We (all) need help  We have no magic bullet  We need genomic variation “The Two Word Definition of Dogma is Brain Dead” Zollinger (sometime during my residency)

Challenges Engineers

Clinicians

Video Capture Basic apparatus Video capture system for laparoscopic system and hand movements Hand movements captured by external camera Sites: ceiling/lighting system/tripod De-identified videos Our system “syncs” these two streams for presentation and analysis Mobile simulator unit

We are becoming increasingly challenged with teaching new dogs old tricks AND We are not very good at teaching old dogs new tricks Evaluation Poses a More Daunting Challenge

Analysis Basic movement tracking algorithms from computer vision, an established field with myriad algorithms to track movements and predict efficacy Proprietary state of the art tools analyze movements