1. Test question writing workshop—Going beyond simple recall Writing Real World or Patient Vignettes and Problem-Solving Questions
Michael Fennewald, PhD
Master Teacher Guild
Dept. of Microbiology & Immunology
Chicago Medical School
Rosalind franklin Univeristy
March 28, 2018

2. Outline: 1. Brief intro to writing test questions in general.
Session Objective: to be able to construct real world or patient vignettes and problem-solving questions
Outline: 1. Brief intro to writing test questions
in general.
2. Intro to writing real world or clinical vignette questions
3. Writing the questions in small groups
4. Reviewing the questions

3. The Basic Rules for One-Best-Answer Items
• Each item should focus on an important concept, typically a common or potentially catastrophic clinical problem.
• Each item should assess application of knowledge, not recall of an isolated fact.
• The stem of the item must pose a clear question, and it should be possible to arrive at an answer with the options covered.
To determine if the question is focused, cover up the options and see if the question is clear and if the examinees can pose an answer based only on the stem. Rewrite the stem and/or options if they could not.
• Each item should focus on an important concept, typically a common or potentially catastrophic clinical problem.
Don’t waste testing time with questions assessing knowledge of trivial facts. Focus on problems that would be
encountered in real life. Avoid trivial, “tricky,” or overly complex questions.
• Each item should assess application of knowledge, not recall of an isolated fact. The item stems may be relatively
long; the options should be short. Clinical vignettes provide a good basis for a question. For the clinical sciences,
each should begin with the presenting problem of a patient, followed by the history (including duration of signs and
symptoms), physical findings, results of diagnostic studies, initial treatment, subsequent findings, etc. Vignettes may
include only a subset of this information, but the information should be provided in this specified order. For the basic
sciences, patient vignettes may be very brief; “laboratory vignettes” are also appropriate.
• The stem of the item must pose a clear question, and it should be possible to arrive at an answer with the options
covered. To determine if the question is focused, cover up the options and see if the question is clear and if the examinees
can pose an answer based only on the stem. Rewrite the stem and/or options if they could not.
• All distractors (ie, incorrect options) should be homogeneous. They should fall into the same category as the correct
answer (eg, all diagnoses, tests, treatments, prognoses, disposition alternatives). Rewrite any dissimilar distractors.
Avoid using “double options” (eg, do W and X; do Y because of Z) unless the correct answer and all distractors
are double options. Rewrite double options to focus on a single point. All distractors should be plausible,
grammatically consistent, logically compatible, and of the same (relative) length as the correct answer. Order the
options in logical order (eg, numeric), or in alphabetical order.
• Avoid technical item flaws that provide special benefit to testwise examinees or that pose irrelevant difficulty.
Do NOT write any questions of the form “Which of the following statements is correct?” or “Each of the following statements
is correct EXCEPT.” These questions are unfocused and have heterogeneous options.
Subject each question to the five “tests” implied by the above rules. If a question passes all five, it is probably well-phrased
and focused on an appropriate topic.

4. These questions are unfocused and have
• All distractors (ie, incorrect options) should be homogeneous.
• Avoid technical item flaws that provide special benefit to testwise examinees or that pose irrelevant difficulty.
Do NOT write any questions of the form “Which of the following statements is correct?” or “Each of the following statements is correct EXCEPT.”
These questions are unfocused and have
heterogeneous options. But questions in these formats can be excellent practice questions

5. Let us now try the Pretest on Test-Wiseness

6. Guidelines for Item Content
• Test application of knowledge using experimental and real world or clinical vignettes
• Focus items on key concepts and principles that are essential information (without access to references) for all examinees to understand
• Test material that is relevant to learning in clinical clerkships, postgraduate medical education, and beyond
• Avoid items that only require recall of isolated facts
• Avoid esoteric or interesting topics that are not essential

8. Age, Gender (eg, A 45-year-old man)
The real world or patient vignettes may include some or all of the following components:
Age, Gender (eg, A 45-year-old man)
Site of Care (eg, comes to the emergency department)
Presenting Complaint (eg, because of a headache)
Duration (eg, that has continued for 2 days).
Patient History (with Family History ?)
Physical Findings
+/- Results of Diagnostic Studies
+/- Initial Treatment, Subsequent Findings, etc.
Many basic science questions can be presented within the context of a patient vignette.

9. Presenting Problem or Complaint
The real world or patient vignettes may include some or all of the following components:
Presenting Problem or Complaint
Duration of that problem (eg, that has continued for 2 days).
Background of the problem
Initial studies on the problem
Ask for further studies
Ask what the underlying problem is
Ask what is the treatment for the problem or management of the problem
Many basic science questions can be presented within the context of a patient vignette.

11. Let us look at some examples of clinical vignette style questions
Let us look at some examples of clinical vignette style questions. In my Medical Microbiology & Immunology course, we have electronic response devices that the students use during class. Let us look first at a test question in the clinical vignette format.

12. MMI case 9: A 52 year old man with fever and drainage from the right leg
A 52 year old man presents with a fever of up to C, erythema that is getting worse and purulent drainage from the lower medial aspect of his right leg. These symptoms began 4 days ago. He had injured this leg at the same area 5 years ago in an automobile accident.
Medical history: Not significant, denies diabetes mellitus
Medications: None
Allergies: None
Physical exam findings: Temperature 38.1; pulse 88; respirations 17; blood pressure 145/90. General: not in acute distress, well-nourished and well-developed adult. HEENT: normal; Chest: normal; Cardiac: normal. Extremities: medial aspect of right lower leg is erythematous, swollen, warm and tender with pus draining from a small opening.
Lab findings: WBC 15,200/microliter with 83% PMNs, 16% lymphocytes, 1% monocytes. ESR=75 mm/hr. Gram stain of sample of wound drainage shows gram positive cocci in clusters with culture pending. Blood cultures are negative. An x-ray of the leg shows evidence of necrotic bone.
What is the likely diagnosis?

13. What is the most likely diagnosis?
Cellulitis
Abscess
Staph toxic shock
Necrotizing fasciitis
Osteomyelitis 8

14. Is this chronic or acute osteomyelitis?
Neither 8

15. What is the most likely organism?
Non-coagulase staphylococci
Staphylococcus aureus
Streptococcus pyogenes (Group A)
Pseudomonas aeruginosa
Enterococcus
Group B strep 8

16. Diagnosis: osteomyelitis
What is the most likely organism?
Staphylococcus aureus

17. Let us look at some examples of clinical vignette style questions
Let us look at some examples of clinical vignette style questions. In my Medical Microbiology & Immunology course, we have electronic response devices that the students use during class. Let us look first at a hard test question in the clinical vignette format. Only 22% of students got this own correct.

18. A 30 year-old woman is requesting HIV testing
A 30 year-old woman is requesting HIV testing. She is asymptomatic and reports no known exposure to a person with HIV. She has had multiple lifetime sexual partners, all of whom are men, and none of whom were men known to have sex with other men. She reports no intravenous drug use by a sexual partner or herself. She does not use condoms and has been in a monogamous relationship for the past 6 months. Medical history is negative for any opportunistic infections or sexually transmitted infections. Her only medication is an oral contraceptive pill.
On physical examination, vital signs are normal. BMI is 23. The remainder of the evaluation is normal. The HIV-1/2 antigen/antibody combination immunoassay is reactive. Result on a subsequent HIV-1/HIV-2 antibody differentiation immunoassay is negative, as is result on an HIV-1 nucleic acid amplification test. Which of the following is the most appropriate next step in management?

19. Western blot for antibodies to HIV proteins
The HIV-1/2 antigen/antibody combination immunoassay is reactive. Result on a subsequent HIV-1/HIV-2 antibody differentiation immunoassay is negative, as is result on an HIV-1 nucleic acid amplification test. Which of the following is the most appropriate next step in management?  
Test for pg 160 antibodies
Western blot for antibodies to HIV proteins
T cell subsets testing
No further tests 9

20. Laboratory HIV Diagnosis Algorithm- CDC recommendation

21. Now, let us try to write our own questions.

22. THE END