1. Impact of Case-Vignette Test Items on National Board Scores Michael Oglesby, Ph.D. UNTHSC-FW

2. Assessing Clinical Competence Until recently, Licensing Board examinations assessed acquisition of medical knowledge (primarily via recall of isolated facts, definitions, etc) However, Licensing Boards have been under increasing pressure to produce examinations that: 1. Assess clinically relevant concepts and abilities 2. Document competence

3. Assessing Clinical Competence In response to that challenge, Licensing Boards adopted changes in the design of test item formats – most notably the introduction of case-vignette-based test questions, which now serve as the stem for the vast majority of licensure test items.

4. Assessing Clinical Competence Both the NBME and NBOME intend to increase the number of test items beginning with a case vignette to approximately 80% – 90% (a dramatic increase from an estimated 30% just a few years ago)

5. AAMC and Step 2 Impact of Complicated Vignettes Case et al. Verbosity, window dressing, and red herrings: do they make a better test item? Acad. Med. 71: 528-530, 1996. 3 variants of vignette formatting studied Point biserial analysis High/Low analysis (upper & lower 20% of exam takers)

6. First-Order Recall (Case et al., 1996) The most likely renal abnormality in children with nephrotic syndrome and normal renal function is A. Acute poststreptococcal glomerulonephritis B. Hemolytic-uremic syndrome C. Minimal change nephrotic syndrome D. Nephrotic syndrome due to focal and segmental glomerulosclerosis E. Shönlein-Henoch purpura with nephritis ABCDE High109900 Low819010

7. Focused Vignette (Case et al., 1996) A 2-year-old boy has a 1-week history of edema. His blood pressure is 100/60 mm Hg, and there is a generalized edema and ascites. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. The most likely diagnosis is (same option list). ABCDE High009820 Low528280

8. Extraneous Information (Case et al., 1996) A 2-year-old African American child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mm Hg, pulse 110/min, and respirations 28/min. In addition to swelling of his eyes and 2+ pitting edema of his ankles, he has abdominal distention with a positive fluid wave. Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ protein and no blood. (Same lead-in and option list followed) ABCDE High019810 Low10966105

9. et al.: Impact of Vignettes Summary of Case et al.: Impact of Vignettes ABCDE High019810 Low10966105 ABCDE High009820 Low528280 ABCDE High109900 Low819010 No Vignette Brief Vignette Full Case Vignette

10. Addition of Noise Complicates Pattern Recognition

11. Assessing Knowledge-Base Competence 1.Licensing Board examinations now utilize case vignettes extensively 2.Case vignettes are designed to assess comprehension and application skills - these are higher level cognitive skills than information acquisition (previously referred to as recall or memorization capabilities) 3.Case Vignettes are frequently “Second-Order” questions, i.e. they assess higher level cognitive skills (comprehension and application capabilities)

12. 2 nd Order Questions 2 nd Order Questions have two components. 1.A patient case-vignette that first requires a diagnosis 2.Then, a basic science (Level 1 Board) or management (Level 2 Board) component of the question that can only be answered if a.A correct diagnosis is made initially, and b.The candidate also knows the additional basic or clinical sciences concepts relevant to the disease represented by the case vignette

13. Typical 2 nd Order Question A 53-year-old male reports recurrent chest pain, typically of 10-20 minutes duration. He was not physically exerting during onset of the pain; indeed, the attacks most often started at near morning while sleeping. History is otherwise unremarkable. ECG reveals elevated ST segments during these attacks. If pharmacologic prophylaxis is ultimately adopted for this condition, which agent would be preferred for chronic treatment? A. Atenolol B. Hydralazine C. Isradipine D. Nitroglycerin E. Propranolol

14. Typical 2 nd Order Question This question requires that the candidate Recognize the diagnosis (Prinzmetal’s Angina) Recall that calcium channel blockers are appropriate prophylactic therapy for this condition (and select the correct medication from the list) TCOM students in 2005 estimated that more than 40% of COMLEX Level 1 questions were 2 nd -order (this number will increase dramatically in the near future)

15. 2 nd Order Questions Probability of getting this item correct is a function of two knowledge-bases 1.Diagnostic knowledge 2.Treatment (pharmacology) knowledge In a well-written 2 nd order question, if an incorrect diagnosis is made, the probability of getting the question correct is at best chance

16. 2 nd Order Questions Suppose a candidate knows both diagnosis and treatment with 0.8 probability. Because these probabilities are independent, the probability of getting the question correct is: 0.8* 0.8 = 0.64 The following graph assumes equal capability for each of these variables (diagnostic knowledge and scientific knowledge) 0.9 * 0.9 = 0.72 0.8 * 0.8 = 0.64 0.7 * 0.7 = 0.49 etc

17. Impact of 2 nd Order Questions Science Knowledge Base, 1 st -Order Question = 80% + 10% DDX Knowledge Base X Science Knowledge Base Assuming Science and DDX Knowledge Bases are Equal Probability

18. Impact of 2 nd Order Questions Flattens the distribution Broadens the distribution Generates much lower scores for the low end of test takers

19. Impact of 2 nd Order Questions Note what happens if DDX improves DDXScienceScore 0.8 =0.64 0.90.8=0.72

20. Impact of 2 nd Order Questions Students must be able to recognize (diagnose) the disease represented by the case vignette to answer the question associated with the case vignette The better the DDX capabilities, the better the score on Boards; the less skilled in DDX, the poorer the score on Boards

21. COMLEX Level 1 Averages TCOM ’01-’04 ScienceDDX

22. Curricular Change for Class Taking COMLEX in 2005 More emphasis on case-based teaching Active training of DDX Students were required to diagnose the common causes of patient problems utilizing a computer-based program (KBIT) that presented cases and corrected diagnostic errors

23. COMLEX Level 1 Averages TCOM ’01-’04 vs ‘05 ScienceDDX