1. Script Concordance Testing to Assess Clinical Reasoning
Mary Blumer, MD; Meenu Jindal, MD; Ihsan Elkhider, LL.B, MS. Ed
Tom Pace, MD; Peggy J Wagner, PhD
Greenville Health System, School of Medicine Greenville
Background and Significance
Teaching clinical reasoning is an educational challenge
Application of Script theory developed by cognitive psychologists is new in medical education
Professionals use “scripts” to process information and to confirm or eliminate diagnostic hypotheses or management options
Qualitative judgments can be measured and compared to expert practitioners allowing us to examine decision making in contexts of uncertainty.
Script Concordance Tests (SCTs) offer innovation and practicality as they can be scored automatically by the process of comparing the examinees score to a panel of experts using probabilities of response options.
Previous research has demonstrated internal reliability and the ability to differentiate medical learners at various levels
Intervention and Methods
We conducted one pilot of the SCT in a second year module in Hematology and Oncology in Our intention with this pilot was to learn about the feasibility of such an approach for teaching preclinical medical students clinical reasoning skills. We expanded on this effort by providing script concordance tests with each module in the year 2015.
6 Tests; 21 Cases; 150 questions
3 Approaches: Online at home; Large group setting using clicker questions; Online at OSCE station
Students’ perception survey
Students performance data
Example (1) Diagnostic Questions
A 45 year-old female presents to the office complaining of fatigue. No chest pain but some shortness of breath with exertion.
Choose the response that best reflects your opinion using the following options:
-2 - Highly Unlikely
-1 – Less likely than before
0 – Neither more nor less likely
+1 – More likely than before
+2 – Very likely
Example (2) Investigational Questions
A 30 year-old male presents with fever and shortness of breath. He has been feeling sick for the last 2-3 weeks.
Choose the response that best reflects your opinion using the following options:
-2 - Not indicated
-1 – Not useful at this point
0 – Neither more nor less useful
+1 – Indicated/useful
+2 – Will be very useful
Scoring
Scoring is based on the frequency of expert selection of each response item. The modal item checked receives full credit (1 point). Partial credit is awarded based on the # of expert responses choosing an option divided by the total number of modal expert responses. Outliers may be excluded.
Results
Students Performance
Students performance data in each SCT is collected and compared to the experts panel as well as to their summative exam score
Global SCT statistics are calculated for each test (Test reliability, mean , and SD). For example , ID module SCT: High reliability .65; Mean score 72, SD 11.9
Example Data from ID module 2015
Students’ Perception
N= 71
Educational Implications:
Potential to measure decision-making ability early in medical education and competency evaluation
Provide an efficient format that approximates the clinical uncertainty seen in medical practice
Next Steps
Continue to evolve in this process to support clinical reasoning education of our preclinical medical students
Consider comparison with other modalities of student assessment
If you were thinking of the following diagnosis. . .
. . . and you find the following evidence . . .
. . . the hypothesis becomes . . . 1
Iron deficiency anemia
History of uterine bleeding 2
B12 deficiency anemia
History of diabetes on glucophage for treatment 3
No koilonychia on physical exam
If you were considering asking for . . .
. . . and you find the following evidence . . .
. . . the investigation becomes . . . 4
Chest x-ray
Crackles on lung auscultation 5
Sputum stain
High Aa gradient 6
Bronchoscopy
Sputum positive for pneumocystis jirovecii
Q 1
Q 2 Q3 Q4 Q5 Q6
# exp CA -2 1 3 -1
.33 2
.67 6 +1 .5 +2