1. Self-Assessment of Surgical Skill Acquisition with Computer-Based Video Training and the Impact on Self-Directed Training Nathan Jowett Vicki LeBlanc George Xeroulis Helen MacRae Adam Dubrowski Surgical Skills Centre The Wilson Centre University of Toronto, Toronto, Ontario, Canada

5. Rogers et al (1998): CBVT vs. didactic lecture Equal cognitive performance but lower technical performance for CBVT group Summers et al (1999): CBVT with expert vs. didactic instruction CBVT superior on performance retention testing Rogers et al (2000): CBVT vs. Concurrent feedback with CBVT Improved performance by external feedback group Xeroulis et al (2005): CBVT vs. expert feedback No difference in performance between groups Is CBVT as effective as expert feedback in basic surgical skill acquisition?

7. Research Objective Are trainees able to adequately acquire basic technical skills on their own practice schedules using CBVT? Research Question: Is technical skill acquisition by means of CBVT compromised when trainees practice in unstructured as opposed to structured learning settings? Hypothesis: Yes

8. CBVT Module

9. Practice block (3 minutes) Structured Practice (SP) Unstructured Practice (UP) Video Captured Trials with Hand Motion Analysis Self-assessment GRS + Proficiency, Expert GRS Self-assessment GRS + Proficiency Expert GRS Open practice period (variable) Additional practice period (fixed) (1 week delay) Pre-Test Post-Test 1 Self-assessed proficiency Post-Test 2Retention Test Experimental Paradigm N = 30 Mean open practice time = 22.7 minutes

10. Unstructured Retention Structured Total time (s) Normalized time to self- assessed proficiency (%) 50 100 150 200 250 300 350 0 0 25 50 75 100 Trials after proficiency 1234 Learning Curve: Time

11. Practice Group Structured Unstructured Time (s) Time Pre-testPost-test 1Retention (Self-assessed proficiency)

12. Practice Group Structured Unstructured Number of Hand Movements Hand Movements Pre-testPost-test 1Retention (Self-assessed proficiency)

13. Practice Group Structured Unstructured Path Length (decimetres) Path Length Pre-testPost-test 1Retention (Self-assessed proficiency)

14. Practice Group Structured Unstructured Expert Global Rating Scale Pre-testPost-test 1Retention (Self-assessed proficiency)

15. Did significant performance improvement occur between pre and retention tests for both groups? Yes(p < 0.05) Did SP and UP groups differ on retention test performance? No(p > 0.05) Did improvement occur between first and second post- tests among SP participants? No(p > 0.05) Summary

16. Trainees learning basic knot tying skills using CBVT in an unstructured setting performed equally as well as those in a structured setting This suggests that trainees were able to accurately self-assess the progression of their technical performance and thus determine when further practice in a single session would no longer be beneficial Conclusions

17. Our results support the initiation of self-directed online or CD-ROM CBVT for basic technical skills as a plausible solution to the economic and staffing constraints associated with teaching technical skills outside of the O.R. Implications

18. Supported by: Dean’s Excellence Fund Faculty of Medicine, University of Toronto Medical Student Summer Scholarship Faculty of Medicine, University of Toronto

19. End

20. Global Rating Scale and Proficiency Based on your current knowledge and/or performance of the double one-hand square knot, how do you expect to perform on this task a week from now with no additional practice? Time and Motion 1 Many Unnecessary moves 23 Efficient time/motion but some unnecessary moves 45 Clear economy of movement and maximum efficiency Instrument Handling 1 Repeatedly makes tentative or awkward moves with hands/fingers through inappropriate use 23 Competent use of hands/fingers but occasionally appeared stiff or awkward 45 Fluid movements with hands/fingers and no stiffness or awkwardness Flow of Operation 1 Frequently stopped operating and seemed unsure of next move 23 Demonstrated some forward planning with reasonable progression of procedure 45 Obviously planned course of operation with effortless flow from one move to the next Overall Performance 1 Very poor 23 Competent 45 Clearly superior Do you believe you have attained sufficient proficiency at this task and would therefore like to quit practice at this point? YesNo

21. Pre-intervention variable Structured Unstructured P Value* Years of medical school training, mean 1.40 (0.70) 1.55 (0.51) 0.51 Sex, No (%) Female 6 (60) 7 (35) Male 4 (40) 13 (65) 0.26 Experience: two-handed square knot, No (%) Yes 1 (10) 1 (5) No 9 (90) 19 (95) 1.00 Experience: instrument tie, No (%) Yes 6 (60) 10 (50) No 4 (40) 10 (50) 0.71 General self efficacy score†, mean (SD) 4.18 (0.46) 3.97 (0.39) 0.21 Practice time to self (min), (SD) -assessed proficiency 18.30 (6.40) 24.90 (13.67) 0.16 *Independentt test for continuous data (2-tailed), Pearsonχ 2 test for nominal data (2-tailed,exact) †Sherer general self-efficacy scale Randomization Efficacy

22. Table 6. Paired-samples t test for performance variables between first and second post tests (structured practice group). Performance VariabletP Value* Mean expert global rating scale assessment -1.83 0.10 (NS) Hand-motion analysis Time 1.75 0.12 (NS) Number of hand movements Right hand 1.66 0.13 (NS) Left hand 1.61 0.14 (NS) *paired-samples t-test, 2-tailed NS = non-significant Post tests 1 and 2 comparisons (SP group)

23. Performance variable Pre-test Post-test Retention-test Mean global rating scale score (/20) 4.5 11.6* 10.4* 4.5 11.0* 8.7* 4.5 11.2* 9.2* Hand-motion analysis Time (seconds) 158.5 36.7* 37.5* 183.0 41.9* 62.2* 174.6 40.2* 54.0* Number of hand movements 96.6 35.3* 37.3* 135.8 38.5* 49.3* 122.3 37.4* 45.4* Structured Unstructured Average Structured Unstructured Average Structured Unstructured Average P>.05 Mean Performance: Pre, Post-1, Retention