Script Concordance Testing to Assess Clinical Reasoning Mary Blumer, MD; Meenu Jindal, MD; Ihsan Elkhider, LL.B, MS. Ed Tom Pace, MD; Peggy J Wagner, PhD Greenville Health System, School of Medicine Greenville Background and Significance Teaching clinical reasoning is an educational challenge Application of Script theory developed by cognitive psychologists is new in medical education Professionals use “scripts” to process information and to confirm or eliminate diagnostic hypotheses or management options Qualitative judgments can be measured and compared to expert practitioners allowing us to examine decision making in contexts of uncertainty. Script Concordance Tests (SCTs) offer innovation and practicality as they can be scored automatically by the process of comparing the examinees score to a panel of experts using probabilities of response options. Previous research has demonstrated internal reliability and the ability to differentiate medical learners at various levels Intervention and Methods We conducted one pilot of the SCT in a second year module in Hematology and Oncology in 2014. Our intention with this pilot was to learn about the feasibility of such an approach for teaching preclinical medical students clinical reasoning skills. We expanded on this effort by providing script concordance tests with each module in the year 2015. 6 Tests; 21 Cases; 150 questions 3 Approaches: Online at home; Large group setting using clicker questions; Online at OSCE station Students’ perception survey Students performance data Example (1) Diagnostic Questions A 45 year-old female presents to the office complaining of fatigue. No chest pain but some shortness of breath with exertion. Choose the response that best reflects your opinion using the following options: -2 - Highly Unlikely -1 – Less likely than before 0 – Neither more nor less likely +1 – More likely than before +2 – Very likely Example (2) Investigational Questions A 30 year-old male presents with fever and shortness of breath. He has been feeling sick for the last 2-3 weeks. Choose the response that best reflects your opinion using the following options: -2 - Not indicated -1 – Not useful at this point 0 – Neither more nor less useful +1 – Indicated/useful +2 – Will be very useful Scoring Scoring is based on the frequency of expert selection of each response item. The modal item checked receives full credit (1 point). Partial credit is awarded based on the # of expert responses choosing an option divided by the total number of modal expert responses. Outliers may be excluded. Results Students Performance Students performance data in each SCT is collected and compared to the experts panel as well as to their summative exam score Global SCT statistics are calculated for each test (Test reliability, mean , and SD). For example , ID module SCT: High reliability .65; Mean score 72, SD 11.9 Example Data from ID module 2015 Students’ Perception N= 71 Educational Implications: Potential to measure decision-making ability early in medical education and competency evaluation Provide an efficient format that approximates the clinical uncertainty seen in medical practice Next Steps Continue to evolve in this process to support clinical reasoning education of our preclinical medical students Consider comparison with other modalities of student assessment If you were thinking of the following diagnosis. . . . . . and you find the following evidence . . . . . . the hypothesis becomes . . . 1 Iron deficiency anemia History of uterine bleeding -2 -1 0 +1 +2 2 B12 deficiency anemia History of diabetes on glucophage for treatment 3 No koilonychia on physical exam If you were considering asking for . . . . . . and you find the following evidence . . . . . . the investigation becomes . . . 4 Chest x-ray Crackles on lung auscultation -2 -1 0 +1 +2 5 Sputum stain High Aa gradient 6 Bronchoscopy Sputum positive for pneumocystis jirovecii Q 1 Q 2 Q3 Q4 Q5 Q6 # exp CA -2 1 3 -1 .33 2 .67 6 +1 .5 +2