Developing an Effective Simulation Lab – “How to Evaluate Residents’ Skills with Simulation” Ann Van Heest MD Director of Education Professor, University.

Slides:



Advertisements
Similar presentations
Establishing the reliability and validity of a Virtual Reality Upper Gastrointestinal simulator using a novel video-endoscopic assessment technique. Moorthy.
Advertisements

Arthroscopic Intervention Staying Active
ARTHROSCOPIC ROTATOR CUFF REPAIR
Module #13 Brian C Toolan, MD Shepard Hurwitz, MD Basic Techniques in External Fixation Developed by the Surgical Skills Task Force of the American Board.
Management of the Upper Limb in Children with Cerebral Palsy Prof P McArthur FRCS(Plast) PhD Consultant in Congenital Hand and Upper Limb Surgery Department.
A Novel Approach to Teaching Visual-spatial Skills in Wire Navigated Procedures Jenniefer Y. Kho, M.D. J. L. Marsh, MD, Geb Thomas, PhD, Brian Johns, MS,
 Terminology  Code Organization  Special Codes  Modifiers  Case Examples  Tips for Op Report Dictation.
Our 2012 experience A Basic Surgical Skills Curriculum for PGY1 Training M. Daniel Wongworawat, MD Professor and Program Director Loma Linda University.
The principles of intra- articular fracture care Joseph Schatzker M.D., B.Sc.,(med.), F.R.C.S.(C )
Infection After ACL Reconstruction H. Makhmalbaf MD Consultant Orth.& Knee Surgeon Mashhad University of Medical Sciences.
Orthopaedic Competence Assessment Project
Module #1 James E. Carpenter, MD Peter M. Murray, MD Sterile Technique & Ortho. Surgery Room Setup Developed by the Surgical Skills Task Force of the American.
NYU School of Medicine CAN PROFESSIONALISM BE TAUGHT? FINALLY THERE IS EVIDENCE Department of Surgery New York University School of Medicine April 29,
Pediatric Forearm Fractures OTA RCFC Pediatric Considerations Periosteum Greenstick / Incomplete fractures Remodeling Cast technique.
The Objective Structured Assessment of Technical Skills (OSATS) Helen M. MacRae, M.D., FRCSC, D.H. Gales Director Technical Skills Centre, University of.
Minimally Invasive Surgery Task Decomposition - Etymology of Endoscopic Suturing Jacob Rosen* Ph.D., Lily Chang** MD, Jeff Brown ***, Andy Isch** MD Blake.
Virtual Reality Surgical Training at the University of Washington Presented By: Timothy Kowalewski Robert Sweet MD, Urology Suzanne Weghorst, Human Interface.
Comparison: Traditional vs. Outcome Project Evaluative Processes Craig McClure, MD Educational Outcomes Service Group University of Arizona December 2004.
Value of Upgrading to Release 12.1 of Oracle Manufacturing Abhay Satpute, Director Product Development Stephen Slade, Sr. Director Industrial Mrktg Joan.
Validation of the UW Virtual Reality TURP Simulator Version 1.0 Presented By: Timothy Kowalewski Robert Sweet, MD Peter Oppenheimer Suzanne Weghorst Jeff.
Physical Therapy Treatment Plans also called
©2014 Delmar, Cengage Learning. All Rights Reserved. May not be scanned, copied, duplicated, or posted to a publicly accessible website, in whole or in.
A Surgical Skills Training Curriculum for PGY-1 Residents J. Lawrence Marsh, MD*; Matthew Karam, MD; Tameem Yehyawi, MD; Brian Westerlind, BA, Jenniefer.
Integrating High Fidelity Simulation into Clinical Orientations INTEGRIS Baptist Medical Center Nursing Education & Research.
Orthopaedic Electives Clerkship Coordinator: Amanda Schwanz
J. L. Marsh, MD Chair - Orthopaedic RRC Director of the ABOS Carroll B. Larson Chair Residency Program Director University of Iowa Hospitals and Clinics.
Fund of Knowledge: Basic research methodology Pre-test mean: 56% ± 8% Post-test mean: 65% ± 6% N=11, p value
Rater Training for Clinical Skills Verification Module 5 Performance Standards Michael Jibson, M.D., Ph.D. Jeffrey Hunt, M.D. David Kaye, M.D. Richard.
End of Shift Milestone Evaluation Forms
What is Biomechanics Biomechanics is a branch of science which employs mechanical and engineering principles to study biological systems The objectives.
Division of Emergency Medicine Cincinnati Children’s Hospital
Copyright © 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins Timby/Smith: Introductory Medical-Surgical Nursing, 10/e Chapter 62: Caring for.
Matthew Roberts, Vanessa Bacal, Mohammed Mahdi, Ethan D. Grober Mount Sinai & Women’s College Hospital, Division of Urology, Department of Surgery, University.
The Royal College of Surgeons of England Regional Representatives Meeting Simulator Training – in practice Implementation of a Surgical Skills Strategy.
Does success on the American Board of Surgery Qualifying Examination guarantee Certifying Examination success? Thomas W. Biester MS, Jonathan D. Rubright.
Can a Brief On-line Education Tool Improve Surgical Resident Operative Dictations? A Prospective Evaluation Alicia Kieninger, MD, Yi Wei Zhang, MD, Anna.
Rural Surgery Innovation in training KM Shyamprasad, Vice-President National Board of Examinations (Postgraduate Medical Education) Ministry of Health,
Introduction to the Orthopaedic 452 course Dr.Abdulaziz Alomar, MBBS, MSc, FRCSC Assisstant professor of Orthopaedic surgery Sport Medicine & Arthroscopic.
Assessment Tools.
Simulation: Why, when, what? Terminology, definitions, frameworks and applications Sandrijn van Schaik, MD PhD 2015 UCSF Center for Faculty Educators &
Continuous Loop Double Endobutton Reconstruction for AC Joint Dislocation Steven Struhl, MD 1, Theodore Wolfson, MD 1 1 Department of Orthopaedic Surgery,
Randomized double-blinded trial investigating the impact of a curriculum focused on error recognition on laparoscopic suturing training J Bingener, T Boyd,
BOOT CAMP: Educational Outcomes After Four Successive Years of Preparatory Simulation-based Training at Onset of Internship Gladys L Fernandez MD, Richard.
Surgical Coaching from Head-Mounted Video in the Training of Fluoroscopically Guided Articular Fracture Surgery by Matthew D. Karam, Geb W. Thomas, Daniel.
An Evaluation of an On-line Anatomy Course by Lab Instructors: Building on Instructional Design Guo, X., Katz, L., & Maitland, M. The University of Calgary,
Biomechanical comparative study of the JuggerKnot™ soft anchor technique with other common mallet finger fracture fixation techniques Jason Pui Yin Cheung,
Matthew Schill, BS, Debbie Tiemann, RN, Mary Klingensmith, MD, L. Michael Brunt, MD Department of Surgery and Institute for Minimally Invasive Surgery.
Proficiency-Based Training for Phacoemulsification Princeton Lee Research Fellow RCSI & ICO Alcon Education Night 27/02/2009.
Measuring Non-technical Aspects of Surgical Clinician Development in an Otolaryngology Residency Training Program Jennifer J. Shin, M.D. S.M. Michael J.
Great Debate: Simulation A Resident’s Perspective Ernest (Ted) Gomez, MD, MTR PGY-3 Resident Department of Otorhinolaryngology – Head and Neck Surgery.
Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual.
Comparing Performance on the Virtual Laparoscopic and Robotic Simulators Among Medical Students Pursuing Surgical versus Non-surgical Residencies Amanda.
Applying Advances Rosalyn P. Scott, MD, MSHA Professor of Surgery Wright State University Boonshoft School of Medicine.
Department of Surgery Development of PGY-1 Surgery Preparatory Course Curricula: Identification of Key Curricular Components Mara B. Antonoff MD Jonathan.
Practice Based Learning and Improvement Stephen J. Kimatian MD Assistant Professor of Anesthesiology and Pediatrics The Penn State, Milton S. Hershey Medical.
Insert Program or Hospital Logo Introduction ► Due to changes in delivery room practices, improvement in clinical care, and limitations on the time spent.
APPROVED CLINICAL INSTRUCTOR WORKSHOP Evaluation.
Clinical Simulation in Family Medicine to address the ACGME Core Competencies Beth Anne Fox, MD, MPH Glenda F. Stockwell, PhD Martin Eason, MD, JD.
Primary Care Management The Hip Andrew Pearse Consultant Trauma and Orthopaedics Worcestershire Acute Hospitals NHS Trust.
1 Utilization of Operating Room Simulation and Debriefing to Enhance Surgical Resident Participation in the Surgical Timeout Checklist Edward Dominguez.
Design and Evaluation of a Virtual Reality Simulation Module for Training Advanced Temporal Bone Surgery Sudanthi Wijewickrema Department of Surgery (Otolaryngology)
Operating Room Team Training With Simulation Program
Teaching the Hands-On Skills for the Team Physician
Clinical Assessment Dr. H
in first year residents of different specialties.
Surgical Interventions and Postoperative Management
North Central Region Emergency Care Council
Mohammed Alshouli. MD Orthopedic Consultant and Foot and Ankle Surgery
Disorders of the Musculoskeletal System
Clinical Practice Guidelines: What, Why, Who?
Presentation transcript:

Developing an Effective Simulation Lab – “How to Evaluate Residents’ Skills with Simulation” Ann Van Heest MD Director of Education Professor, University of MN Dept of Orthopedic Surgery

Simulation Skills Lab Skills Acquisition Competency Testing

How do we measure competence? Idea Face validity: Does the test is going to measure what it was intended to measure? Construct validity: Can the test differentiate between experts and novices? Reliability: Inter-rater and Intra-rater

OUTLINE Matching the “Right” educational level for resident’s level of learning 1. Upper Extremity Surgical Skills Competency Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Mandate

Upper Extremity Surgical Skills Competency Testing

Milestone Project Milestone Topics (n=16) ACL injury Ankle Arthritis Ankle Fracture Carpal Tunnel Syndrome Diabetic foot Shaft fx. Femur/tibia Distal Radius fracture Elbow fracture Medical Knowledge and Patient Care Hip and Knee OA Hip fracture Meniscal tear Metastatic bone Rotator Cuff Injury Septic arthritis (peds) Pediatric Supracondylar Humerus fracture Degenerative Spine

6th Lab Session: May 17, 2013 Carpal Tunnel Cadaveric Station 2 board-certified hand surgeon graders Detailed Checklist Global Rating Scale Pass / Fail Trigger Release Cadaveric Station 2 board-certified hand surgeon graders Detailed Checklist Global Rating Scale Pass / Fail Volar Approach Distal Radius Cadaveric Station board-certified hand surgeon grader Detailed Checklist Global Rating Scale Pass / Fail Biomechanical Testing Load to Failure Distal Radius Fixation with C-arm Radiographic Scores Debrief Pass/Fail Debrief Pass/Fail Post Test Evaluation Debrief

Motor Skills Testing: 3 Bench Stations 1.Global Rating Scale 2.Detailed Checklist 3.Adverse Events 4.Pass/Fail 5.Time

n = Program Year Global Rating – Carpal Tunnel p = 0.040

n = Program Year Carpal Tunnel Release Detailed Checklist (100pts) p = 0.002

Pass/Fail Assessment Did the surgeon achieve the goal of surgery?

n =0 / 21 / 75 / 07 / 04 / 02 / 0 Program Year Pass / Fail Assessment p < 0.001

Distal Radius Fracture Fixation

Conclusions This study reports that a surgeon’s ability to release a trigger finger does not correlate specifically to his or her ability to perform a carpal tunnel release or to perform plate fixation of a radius fracture. The results of this study would indicate that, for 3 different surgical simulations representing procedures of varying complexity, assessments by a single assessment tool is not adequate. To completely understand a resident’s abilities, assessment by checklist (understanding the steps of the surgery), global rating scales (assessment of basic surgical skills in light of lesser or greater complexity surgeries), and pass/fail assessment (examination of adverse events) are all necessary components.

Competence High Stakes Exam G3 level Pass vs Needs more practice Competency advancement, not social advancement DEMONSTRATING COMPETENCE IS GRADUATION REQUIREMENT

OUTLINE Matching the “Right” educational level for resident’s level of learning 1. Upper Extremity Surgical Skills Competency Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Mandate

Surgical Simulation Training Program for Articular Fracture Surgery Jenniefer Y. Kho, MD University of Iowa Hospitals and Clinics Karam MD, Ohrt GT, Thomas GW, Yehyawi TM, Lafferty PM, Anderson DD, Marsh JL

Aims Develop a comprehensive surgical simulation training program utilizing an articular fracture model Investigate whether simulator training can improve performance in junior residents

Methods 6 PGY-1 and 6 PGY-2 residents

Methods Interventio n (n=6) PretestTrainingPosttest Control (n=6) Pretest No Training Posttest 6 PGY-1 and 6 PGY-2 residents

Sawbones Fracture Model with Barium coated articular surface

Video capture

Methods - Intervention On-line cognitive module Video review with traumatologist Skills module Interventio n (n=6) PretestTrainingPosttest

Methods Outcome measures – OSATS global rating scale – Articular reduction – Hand motion – Fluoroscopy time, radiation dose Face Validity (Questionnaire)

Results OSATS GLOBAL RATING SCALE Control Δ OSATS global rating score p=0.06p= 0.68 Intervention

Results FLUOROSCOPY TIME Fluoroscopy time (s) Pretest Posttest Control Intervention p= 0.03p= 0.16

Results Control Intervention ARTICULAR STEP--OFF p= 0.22p= 0.69 Articular step-off (mm) Pretest Posttest

Results Pretest Posttest Control Intervention Cumulative hand distance (m) CUMULATIVE HAND DISTANCE p= 0.8p= 1.0

Results – Face validity Scale ranges from 1 (most) to 5 (least realistic) How realistic was this exercise overall? How realistic was the bone model visually? How realistic was the movement of the synthetic fracture fragments compared to that of real bony fragments? How realistic was the feel of the synthetic bone during instrumentation compared to that of real bone? Overall average score 2.75 Acceptable, but room for improvement

Discussion Improved OSATS score, decrease in fluoroscopy time and radiation exposure Articular step-off, hand motion no different Junior residents with little to no articular fracture experience may benefit from short-term dedicated training

Collaboration with University of Minnesota Incorporation into surgical skills curriculum at the University of Iowa Potential for expansion to other programs

OUTLINE Matching the “Right” educational level for resident’s level of learning 1. Upper Extremity Surgical Skills Competency Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Mandate

Low-Fidelity Arthroscopic Simulation Can Differentiate Between Experts and Novices Jon Braman MD, Robert Sweet, MD David Hananel, SimPortal Lab Paula Ludewig, Ph.D Ann VanHeest, MD

Panel of Experts Visualization Triangulation Object Manipulation

Triangulation 10 LED lights 10 switches Randomly order lights (training) Pre-determined order lights (testing) Time Errors: Alarm if touches metal VIDEO

Object Manipulation 3 bars on left with rings 3 bars on right with rings Transfer rings from left to right (Switch scope) Time Errors: Dropped rings VIDEO

Triangulation p = p =

Object Manipulation p = Errors: Drops p=1.0

Basic Arthroscopic Skills Construct Validity Face Validity G1 level skills: Readiness for OR Multi-Center Trial Planned Competency Testing: –By year in training –Translation to OR Scope Skills

ABOS Surgical Simulation Mandate “formal instruction in basic surgical skills, provided longitudinally or as a dedicated non-orthopaedic surgery rotation will be required. Skills training will need to be designed to integrate with subsequent post graduate years and should prepare the PGY 1 to participate in orthopedic surgery cases. Skills training will need a dedicated space and a curriculum which must include: –Goals and objectives and assessment metrics. –Skills used in the outpatient management of injured patients including splinting, casting, application of traction devices and other types of immobilization. –Basic operative skills including soft tissue management, suturing, bone management, arthroscopy, flouroscopy and use of basic orthopaedic equipment.

ABOS/AOA-CORD/AAOS Develop a modular outline of surgical skills curriculum 3 meetings (Sept, March, May) Posted on ABOS website with access to all residency programs.

1. Sterile Technique and Operating Room Set-Up 2.Knot Tying & Suturing 3.Basic Microsurgical Suturing 4.Soft Tissue Handling Techniques 5.Casting and Splinting 6.Traction 7.Compartment Syndrome 8.Bone Handling Techniques - Osteotomy 9.Fluoroscopic Knowledge and Skills 10.K-Wire Techniques 11.Techniques Basic to Internal Fixation of Fractures 12.Principles and Techniques of Fracture Reduction 13.External Fixation 14.Basic Arthroscopy Skills 15.Basic Arthroplasty Skills 16.Joint Sspiration and Injection 17.Patient Safety, Team Training, Obtaining Consent

Example of Module

How to measure Competency? 1.Upper Extremity Surgical Skills 2.Articular Fracture Reduction Model 3. Basic Arthroscopy Skills Box 4. G1 Basic Skills Modules Idea Face validity: Does the test measure what it was intended to measure? Construct validity: Can the test teach the desired operative skills? Reliability: Inter-rater and Intra-rater