1. Objective Methods for Assessment of Technical Skills in Otolaryngology–Head & Neck Surgery Residents: A Systematic Review
Érika MERCIER1, Ségolène CHAGNON-MONARQUE1, François LAVIGNE1, Tareck AYAD1
Centre hospitalier universitaire de Montréal (CHUM), Université de Montréal1
abstract
methods
results
Background: The format of residency is changing towards acquisition of specific competencies, which will require new methods of assessing surgical skills. Objectives: The primary goal is the identification of methods used to assess surgical competency in ORL-HNS residents. Secondary goals include assessment of the reliability and validity, as well as the identification of specific procedures or sub-specialities in ORL-HNS for which the development of additional surgical evaluation tools could be required. Methods: The PRISMA method of systematic reviewing was used to review digital and paper databases. Titles and abstracts were reviewed by two independent reviewers and retained articles were fully analysed in order to identify the types of evaluation tools used as well as the subspecialty of ORL-HNS involved. Reliability and validity of the instruments were also assessed and scored. Results: Five different evaluation tools for surgical competency assessment have been developed. Overall reliability of the tools was moderate to high while validity was generally high. Most surgical procedures from almost every subspecialty of ORL-HNS have a validated evaluation tool. Conclusions: Of the five types of evaluation tools used to assess the surgical skills of ORL-HNS residents, the combined global rating scale (GRS) and task-specific checklist (TSC) evaluation tool is the most used and studied. Reliability and validity of the different tools range from moderate to high. Specific procedures for which instruments have been developed are multiple and diversified, but further research is required to include oncologic procedures.
Database search
In line with PRISMA statement.
Search was conducted on January 18th 2016
Two independent reviewers
Databases : MEDBASE, OVID, Medline, CINAHL and EBM, printed literature available through the Université de Montréal library.
Inclusion criteria : Research written in English or French; published after 1990; containing information on assessment tools for resident evaluation in ORL-HNS.
Exclusion criteria : If the subject of the paper was not medical education; not ENT-specific; not specific to assessment of surgical skills; not a study/not the desired type of research paper; not related to or applied to residents; not available in full text.
Full manuscript analysis
General data : the type of article; the nature of the of tool described in the paper; the otolaryngology subspecialty involved; the surgical procedure for which the tool was developed; the overall risk of bias; the main conclusions of the authors.
Assessment methods and ENT subspecialties : Year and location of publication; type of article; evaluation method; number and type of participants; subspecialty of ORL-HNS; surgical procedure involved; general conclusions.
Reliability and validity :
Based on elements highlighted by Gail M. Sullivan1
Reliability criteria: internal consistency, inter-rater consistency and intra-rater consistency
Validity criteria: content, response process, relationship to other variables, consequences.
Scored as weak, moderate or strong
Indexation of surgical procedures on which residents are evaluated
Eleven different procedures
Procedures most frequently studied:
Temporal bone dissection (8 studies)
Endoscopic sinus surgery (7 studies)
Geographic location of publication
Assessment of reliability and validity
Reliability :
Internal consistency: serves to confirm that the tool tests for the concept of interest
Inter and intra-rater consistency: test respectively the outcomes of having only one versus many raters use the tool, in order to determine the reproducibility of the results.
Validity :
Content: encompasses the quality of the process and steps leading to the creation of the tool, including people and concepts involved.
Response process: evaluates if the subject’s test results reflect his actions. It also encompasses the quality of the rater’s training as well as the instructions given to the subjects prior to evaluation.
Relationship with other variables: relates to the way the evaluation tool compares to other similar tools or to a gold-standard.
Consequences: refers to the capacity of the evaluation tool to reflect better scores after additional training.
background
Reform in the medical education system
Competency-base learning :
New curriculum based on the objective acquisition of program specific competencies.2,3
An international movement :
These changes started in the United States and are being implemented in other countries, including Canada.4
Subjective to objective assessment :
In the current system, resident’s technical skills are evaluated informally during operating time or retrospectively at the end of a rotation.
Does not ensure reproducible, consistent and standardized feedback.
Various assessment tools have been developed and validated in order to provide a dependable way to evaluate surgical skills in otolaryngology residents.
conclusions
Figure 1: Search flow chart
objectives
results
Five major means of assessing surgical competency have emerged in this review: the combined global rating scale (GRS) and task-specific checklist (TSC), the automated analysis of the surgeon’s hand motions, simulation on animals.
The combined GRS and TSC tool was used in over 83% of all assessment studies.
These tools have overall demonstrated high reliability and validity scores.
No tool shave been used yet to assess oncologic procedures
Main objective : Review the available literature to identify methods currently available and validated to assess surgical competency in otolaryngology residents.
Secondary objectives :
Report on the reliability and validity of these assessment tools.
Identify specific procedures or areas of sub- specialities in Otolaryngology for which the development of surgical assessment tools might still be needed.
Study of surgical competency evaluation methods
Five different evaluation methods:
combined global rating scale (GRS) and task-specific checklist (TSC)
automated analysis of the surgeon’s motions
simulations on animal models
virtual reality technology
written practical examinations
References
Gail M. Sullivan. (2011). A Primer on the Validity of Assessment Instruments. J Grad Med Educ, 3(2), doi:doi: /JGME-D
Royal College of Physicians and Surgeons of Canada: Applying competency-based practices to residency education. (n.d.). Retrieved April 15, 2017, from
COMPETENCY-BASED MEDICAL EDUCATION. (n.d.). Retrieved April 15, 2017, from
Milestones. (n.d.). Retrieved April 15, 2017, from do/accreditation/milestones/overview