Clinical Skills Verification Rater Training MODULE 4 Strategies for Clinical Skills Assessment: Models and Best Practices Michael Jibson, M.D., Ph.D. David Kaye, M.D. Sandra Sexson, M.D.

Module 4 Pre-Test 1.The strength of a Clinical Skills Verification Examination stems from its ability to serve as a forum for all of the following except: A. A high stakes examination B.direct observation of clinical skills C.provision of immediate feedback on performance D.repeated administrations during residency

2.Which of the following models for CSV exam is the least standardized? A.Workshop Model B.Mock Board Model C.Clinical Evaluation Model Module 4 Pre-Test

3.Which of the following models for the CSV exam makes the greatest use of resident peer feedback? A.Workshop Model B.Mock Board Model C.Clinical Evaluation Model

Module 4 Pre-Test 4.Within a program, consistency of faculty expectations and rating of residents on the CSV exam can be maximized by: A. All raters using the same evaluation model B.Limiting assessment to the three competency areas outlined by the ABPN C.Using standardized patients D.Monitoring inter-rater reliability

Clinical Skill Verification (CSV) Strengths of Clinical Skills Verification Direct observation of clinical skill Multiple observations and observers Defined areas for evaluation Specific anchors for performance Immediate feedback Ample opportunity for remediation

Evaluation Model We recommend that the evaluation: Be done early in residency and repeated frequently Include immediate feedback Avoid the high-stakes exam model

Evaluation Model Mock Board Model Use the CSV for the current practice of Mock Board exams Prearranged patients Scheduled interviews

Mock Board Model Advantages Familiar format Highly standardized Disadvantages Retains high-stakes exam format Narrow patient selection Difficult to schedule

Evaluation Model Workshop Model Use the CSV form for a workshop or class on interviewing Involve peer resident observers Videotape the encounter

Workshop Model Advantages Multiple residents involved Highly standardized Excellent feedback (if videotaped) Disadvantages Few opportunities for each resident to interview Narrow patient selection

Evaluation Model Clinical Model Use the CSV form for any initial clinical encounter with supervising faculty Inpatient admission Emergency room evaluation Outpatient assessment

Clinical Model Advantages Broad range of patients Many opportunities to interview Easy to schedule Disadvantages Moderately standardized Faculty buy-in essential

Evaluation Model We recommend that the evaluation: Occur during regularly scheduled clinical rotations Involve the standard interview used in that clinical setting Involve randomly selected patients Be done as part of the routines of clinical care

Additional Issues Programs may want to consider: Using the exam to test additional competencies for their own purposes Requiring residents to pass the examination before advancing through training

Additional Competencies CSV requirements may be combined with other clinical competencies: Case formulation Diagnostic assessment Treatment planning

Faculty Training We recommend that programs: Offer training to faculty in how to conduct and score the examination Make an effort to standardize scoring on the examination Monitor inter-rater reliability internally Consider credentialing faculty as evaluators

Post-Test Module 4 1.The strength of a Clinical Skills Verification Examination stems from its ability to serve as a forum for all of the following except: A. A high stakes examination B.direct observation of clinical skills C.provision of immediate feedback on performance D.repeated administrations during residency

Post-Test Module 4 1.The strength of a Clinical Skills Verification Examination stems from its ability to serve as a forum for all of the following except: A. A high stakes examination The in –residency CSV exam allows programs to administer repeatedly and avoid the high-stakes aspect.

Post-Test Module 4 2.Which of the following models for CSV exam is the least standardized? A. Workshop Model B.Mock Board Model C.Clinical Evaluation Model

Post-Test Module 4 2.Which of the following models for CSV exam is the least standardized? C.Clinical Model The clinical model utilizes patients seen on regular clinical services without pre-selection of patients

Post-Test Module 4 3. Which of the following models for the CSV exam makes the greatest use of resident peer feedback? A.Workshop Model B.Mock Board Model C.Clinical Evaluation Model

Post-Test Module 4 3.Which of the following models for the CSV exam makes the greatest use of resident peer feedback? A.Workshop Model

Post-Test Module 4 4.Within a program, consistency of faculty expectations and rating of residents on the CSV exam can be maximized by: A. All raters using the same evaluation model B.Limiting assessment to the three competency areas outlined by the ABPN C.Using standardized patients D.Monitoring inter-rater reliability

Post-Test Module 4 4.Within a program, consistency of faculty expectations and rating of residents on the CSV exam can be maximized by: D.Monitoring inter-rater reliability Along with faculty training on how to conduct and score the examination, tracking inter-rater reliability among faculty after observing the same performance and giving feedback to faculty regarding their ratings compared to others is a very effective way of standardizing ratings.