Redefining Simulator Proficiency Using Automaticity Theory Dimitris Stefanidis, MD, PhD, Mark W Scerbo, PhD James R Korndorffer Jr, MD, Daniel J Scott, MD Carolinas Medical Center, Charlotte, NC Old Dominion University, Norfolk, VA Tulane University, New Orleans, LA UT Southwestern Medical Center, Dallas, TX

Background Training to proficiency on simulators improves operative performance 1,2 Training to proficiency on simulators improves operative performance 1,2 Proficiency levels are derived from expert performance Proficiency levels are derived from expert performance Most common performance metrics used are time and errors Most common performance metrics used are time and errors Expertise is, however, not well defined in surgery 3 and other metrics may more accurately assess performance Expertise is, however, not well defined in surgery 3 and other metrics may more accurately assess performance 1 Korndorffer Jr JR, JACS Seymour NE, Ann Surg Ericsson KA, Acad Med 2004

Background Automaticity refers to the habitual performance of a task without significant demands on attention 1 Automaticity refers to the habitual performance of a task without significant demands on attention 1 Expert performance is characterized by automaticity; it confers them the ability to multi-task Expert performance is characterized by automaticity; it confers them the ability to multi-task A secondary task performed simultaneously with the primary task can be used as an index of expertise by measuring multi-tasking ability A secondary task performed simultaneously with the primary task can be used as an index of expertise by measuring multi-tasking ability 1 Schiffrin & Schneider, Psychol Rev 1977

Background Secondary tasks must be sensitive, selective and unobtrusive 1 Secondary tasks must be sensitive, selective and unobtrusive 1 They must compete for resources that are common to the primary task They must compete for resources that are common to the primary task Laparoscopy imposes heavy visual- spatial demands on the surgeon Laparoscopy imposes heavy visual- spatial demands on the surgeon A visual-spatial secondary task that draws from the same attentional resources is ideal A visual-spatial secondary task that draws from the same attentional resources is ideal 1 O’Donnell & Eggemeier, 1986

Study Objective To assess whether a visual-spatial task that measures attentional spare capacity can distinguish among subjects of variable expertise in laparoscopic suturing To assess whether a visual-spatial task that measures attentional spare capacity can distinguish among subjects of variable expertise in laparoscopic suturing

Methods IRB- approved protocol IRB- approved protocol 12 Participants: 4 novices, 3 surgery residents, 3 laparoscopy experts, and 2 novices trained to proficiency in laparoscopic suturing 12 Participants: 4 novices, 3 surgery residents, 3 laparoscopy experts, and 2 novices trained to proficiency in laparoscopic suturing Participants had to perform under dual- task conditions for 10 minutes Participants had to perform under dual- task conditions for 10 minutes Laparoscopic suturing and knot tying on a videotrainer simulator using the FLS modelLaparoscopic suturing and knot tying on a videotrainer simulator using the FLS model Visual-spatial secondary task performanceVisual-spatial secondary task performance

Video - Primary task ExpertNovice

Video - Secondary task

Video - Dual Tasking

Metrics Laparoscopic Suturing Laparoscopic Suturing Objective score (=300-[time+10*errors])Objective score (=300-[time+10*errors]) Time (max 5 min per repetition) Time (max 5 min per repetition) Errors Errors AccuracyAccuracy Knot SecurityKnot Security Secondary task Secondary task Percent correct detectionsPercent correct detections When performed alone When performed alone When performed simultaneously with suturing When performed simultaneously with suturing

Statistical Analysis Kruskal-Wallis One Way Analysis of Variance on Ranks for the four groups with pairwise comparisons (Dunn’s method) Kruskal-Wallis One Way Analysis of Variance on Ranks for the four groups with pairwise comparisons (Dunn’s method)

Results Experts:>200 basic and >100 advanced laparoscopic cases, extensive experience with simulator Experts:>200 basic and >100 advanced laparoscopic cases, extensive experience with simulator Novices: no prior laparoscopy or simulator experience Novices: no prior laparoscopy or simulator experience Trained: achieved proficiency on simulator, no operative experience Trained: achieved proficiency on simulator, no operative experience Residents: <100 basic, <30 advanced laparoscopic cases, familiar with simulator Residents: <100 basic, <30 advanced laparoscopic cases, familiar with simulator

Results All achieved a perfect score on secondary task when performed alone (100% correct detections) All achieved a perfect score on secondary task when performed alone (100% correct detections) Secondary task performance deteriorated in all groups when performed simultaneously with primary task (p<0.001) Secondary task performance deteriorated in all groups when performed simultaneously with primary task (p<0.001) Experts and trained outperformed residents and novices on suturing task Experts and trained outperformed residents and novices on suturing task While experts did not differ from trained based on time and errors, they achieved better secondary task scores (p=n.s.) While experts did not differ from trained based on time and errors, they achieved better secondary task scores (p=n.s.)

Conclusions A visual-spatial secondary task that assesses attentional spare capacity may distinguish among levels of laparoscopic expertise (construct validity) when standard performance measures fail to do so A visual-spatial secondary task that assesses attentional spare capacity may distinguish among levels of laparoscopic expertise (construct validity) when standard performance measures fail to do so Such secondary task metrics may more accurately define expert performance for use as training endpoints during simulator curricula and possibly for assessment purposes Such secondary task metrics may more accurately define expert performance for use as training endpoints during simulator curricula and possibly for assessment purposes Further validation of secondary task metrics and automaticity is warranted and currently underway Further validation of secondary task metrics and automaticity is warranted and currently underway

?