Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein.

Slides:

Advertisements

Similar presentations

Use of a PDA to Monitor Surgery Student Work and Sleep Hours Susan Steinemann, MD, FACS Jill Omori, MD University of Hawaii School of Medicine.

Advertisements

EMILY W. ZANTOW, BSE, DOROTHY A. ANDRIOLE, MD, DONNA B. JEFFE, PHD, JULIE WOODHOUSE, RN, L. MICHAEL BRUNT, MD WASHINGTON UNIVERSITY SCHOOL OF MEDICINE.

The ERAS Application Can Predict ACGME Competency-Based Surgical Resident Performance Amy M. Tolan MD *, Amy H. Kaji MD PhD *, Chi Quach †, O.Joe Hines.

NYU School of Medicine CAN PROFESSIONALISM BE TAUGHT? FINALLY THERE IS EVIDENCE Department of Surgery New York University School of Medicine April 29,

Carlos S. Morales, MD; Foula Kontonicolas MD; Anita Volpe DNP; Pierre F. Saldinger MD, FACS Royd Fukumoto, MD, FACS Department of Surgery and Department.

Objective vs. subjective in assessment Jaime Correia de Sousa, MD, MPH Horizonte Family Health Unit Matosinhos Health Centre - Portugal Health Sciences.

Joan E. St. Onge, M.D. UMMSM At Holy Cross Hospital Internal Medicine Residency Faculty Development January 23, 2013 The Evaluation Toolkit.

Pilar Muñoz, José-Antonio González, Erik Cobo, Lluis Jover JORNADES D’INNOVACIÓ DOCENT A LA UPC: Presentació de resultats dels projectes MQD ICE 28/06/07.

CBES Essentials for Residents, Fellows, and Faculty A 10-minute primer on student performance assessment in required clerkships Stanford School of Medicine.

EVALUATION OF DR.MOHAMMED AL NAAMI, FRCSC, FACS, M Ed. Using O bjective S tructured C linical E xamination (OSCE)

American Board of Surgery Examinations: Can We Identify Surgery Residency Applicants Who Will Pass the Examinations on the First Attempt? John L. Shellito,

Assessment of clinical skills Joseph Cacciottolo Josanne Vassallo UNIVERSITY OF MALTA ANNUAL CONFERENCE - OSLO - MAY 2007.

UIUC College of Medicine: Teaching Curriculum

Barbara J Pettitt, MD Emory University Dept. of Surgery Atlanta GA April 24, 2015.

 Alterman DM, Jones TM, Daley BJ, Goldman MH Department of Surgery.

Assessment Tools. Contents Overview Objectives What makes for good assessment? Assessment methods/Tools Conclusions.

Seth D. Goldstein, MD 1, Brenessa Lindeman, MD 1, Jorie Colbert-Getz, PhD 3, Trisha Arbella 1, Robert Dudas, MD 2, Anne Lidor, MD, MPH 1, Bethany Sacks,

Portfolio Assessment in Clerkship Michelle Gibson - Geriatrics (thanks to Chris Frank and Melissa Andrew too)

Intersection of Surgical Outcomes and Medical Education The RRC Perspective APDS Panel Session IV Surgical Education Week March 21,2012 San Diego, CA James.

Education 793 Class Notes Welcome! 3 September 2003.

Assessment tool OSCE AH Mehrparvar,MD Occupational Medicine department Yazd University of Medical Sciences.

An Introduction to Measurement and Evaluation Emily H. Wughalter, Ed.D. Summer 2008 Department of Kinesiology.

Assessing clinical judgment using the script concordance test: The importance of using specialty-specific experts to develop the scoring key Petrucci AM.

An Introduction to Measurement and Evaluation Emily H. Wughalter, Ed.D. Summer 2010 Department of Kinesiology.

Is the Script-Concordance Test a Valid Instrument for Assessment of Intra-operative Decision-making Skills? Brent Zabolotny 1, Robert Gagnon 2, Bernard.

Background: As students complete their clerkships throughout their M3 year they gain in clinical experience and confidence, which may translate into improved.

Assessment in Education Patricia O’Sullivan Office of Educational Development UAMS.

Student assessment AH Mehrparvar,MD Occupational Medicine department Yazd University of Medical Sciences.

Forming the Method Subjects Instruments or apparatus Procedures Design and analysis The Method section explains how the study was conducted and should.

ScWk 242 Course Overview and Review of ScWk 240 Concepts ScWk 242 Session 1 Slides.

Is The ACGME Resident/Fellow Survey a Valid Tool to Assess General Surgery Residency Programs Compliance with Work Hours Regulations? Robert P. Sticca,

Developing an Assessment System B. Joyce, PhD 2006.

T HE R EVALIDA Ma. Graciela G. Gonzaga,M.D.,MSc Dean Faculty of Medicine and Surgery Ma. Graciela G. Gonzaga,M.D.,MSc Dean Faculty of Medicine and Surgery.

Assessment Tools.

USMLE Step 1 % Passed on All Attempts BSOM 1 st Attempt Examinees All Examinees 1 st Attempt Mean Score BSOM 1 st Attempt Mean Score All 1 st Attempt Repeat.

BOOT CAMP: Educational Outcomes After Four Successive Years of Preparatory Simulation-based Training at Onset of Internship Gladys L Fernandez MD, Richard.

CBEI Essentials for residents, fellows, and faculty A 10-minute primer on the Criterion-Based Evaluation Initiative and student performance assessment.

Assessing Learners The Teaching Center Department of Pediatrics UNC School of Medicine The Teaching Center.

Assessment of Course-Level Learning Outcomes in Psychology.

15 th May 2008 Virtual Patients as a Tool for Assessment Presented to Medbiquitous Conference 2008 Presented by Emily Conradi, e-Projects Manager Jonathan.

The use of OSCE to assess Patient Care, Professionalism and Interpersonal Communication Milestones in EM residents Miriam Kulkarni, MD, Harsh Sule, MD,

GRADING n n Grading that reflects the actual ability of the student n n Grading as a personal communication.

Students Assessment: Essay Type Questions. Dr. G. U. Kavathia Associate Professor in Microbiology, P. D. U. Govt. Medical college, Rajkot.

Family Medicine End of Clerkship Assessment (EOCA) [Eee-O-Kah] Irmanie Eliacin MD Christine Dalton, MD John Delzell Jr, MD Suzanne Minor, MD Marquita Samuels,

Scott Renshaw, MD Department of Family Medicine Third Year Clerkship.

Welcome to the M-3 World Office of Medical Student Programs S.K. Jacob, Ph.D. Rising M-3 meeting, 1/14/09.

Defining Grades in the Surgery Clerkship Jeremy M. Lipman, MD MetroHealth Medical Center Case Western Reserve University School of Medicine.

Practice CSE-3: A Formative, Low Stakes Experience for Third Year Family Medicine Clerkship Students Jeri R. Reid, MD Donna M. Roberts, MD Mary B. Carter,

Maria Gabriela Castro MD Archana Kudrimoti MBBS MPH David Sacks PhD

Troubleshooting 106 Use of oral exams and OSCEs for student assessment Jesse Moore, MD FACS Kathryn Butler, MD FACS Pam Rowland, PhD.

Introduction References Objectives Conclusions Results Faculty provision of performance feedback is critical for residents to improve their clinical skills.

PRIME+ Vicki Hayes, MD and Peggy Cyr, MD February 6, 2015.

Family Medicine Online Penn State College of Medicine Shou Ling Leong, MD Department of Family and Community Medicine Penn State College of Medicine.

Central Role of Physiology in the Professional Curriculum

An Alternative Certification Examination “ACE”, to assess the domains of professional practice. Morris M1*, Gillis AE2, Smoothey CO3, Hennessy M1, Conlon.

Melissa Smith-Phillips, MD* and Nigel Bourne, PhD**

Marquita Samuels, MBA Candidate

Developing a Student Learning Outcomes Assessment Plan and Report

Michael Henehan, DO San Jose-O’Connor Hospital

INSTRUCTIONAL MATERIALS AND METHODS USED

Objective vs Subjective

UMDNJ-New Jersey Medical School

Effective Feedback, Rubrics, and Grading

CBEI Essentials for Residents, Fellows, Advanced Practice Providers, and Faculty A 10-minute primer on student performance assessment in required clerkships.

Comparisons among methods to analyze clustered multivariate biomarker predictors of a single binary outcome Xiaoying Yu, PhD Department of Preventive Medicine.

Using an ‘Oral Board’ exam to assess for EPA 10 in

Statistical Inference for Managers

Weighted as 40% of final clerkship grade.

Kelley Morel, M. D. , Fidel Valea, M. D. , Allison Tegge, Ph. D

Gerald H. Sterling, Ph.D. Senior Associate Dean, Education

Presentation transcript:

Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein MEd PhD, R Nowygrod MD, A Fingeret MD

Agenda Needs assessment Exam Development Exam Design Methodology Results Discussion

Needs Assessment Historical assessment of students – Clinical evaluations – Unstructured oral exam – NBME surgery subject exam Goals of new evaluation tool – Objective and reliable – Congruent with educational objectives – Based on ACGME core competencies

Exam Design Novel Exam: Clinical Skills Exam (CSE) – 20+ clinical scenarios 5 randomly selected per exam – Short answer or essay response – Administered over 3 hours – ExamSoft computer software Grading – Rubric – Senior residents

Example 1 Identify operation Obtain surgical consent Determine order Describe steps shown

Example 2 Generate differential based on history Describe risk factors Interpret radiographic study Identify next step in management

Methodology July, February, 2013 – n = 262 Analysis – Descriptive statistics – Multivariate analysis Comparison data for validity assessment – USMLE Step 1 and 2CK – NBME Subject Exams in Medicine and Surgery – Clerkship clinical evaluations – Clerkship order

Results: Descriptive Statistics – CSE Content and Score Distribution

Results: Trend Analysis – CSE and NBME Surgery Subject Exam

Results: Multivariate Statistics - Validity Analysis CSENBME Correlation Linear RegressionCorrelation Linear Regression rp valuer NBME Surgery0.36*0.82 NBME Medicine Step 10.34* *0.02 Step 20.44* *0.004 Clerk - Clinical Reasoning 0.48* *0.09 Clerk - Fund of Knowledge 0.56* *< * = p < 0.001

Results: Multivariate Statistics – Validity Analysis CSE NBME Bivariate Analysis Logistic Regression p value Clerkship orderANOVA0.30< OB/Gyn priorX20.34< Medicine priorX20.69<

Results: Student Feedback on CSE

Results: Student Feedback - Word Cloud

Limitations – Study Generalizability Statistical analysis

Future Directions Future directions – New content – Open access – Computer algorithm based grading – Use as formative assessment mid-clerkship

Conclusions CSE is a useful adjunct to NBME surgery subject exam – Independent of clerkship order – Course goals and objectives – ACGME core competencies

Questions / Comments