Constructing Written Test Questions for Basic and Clinical Sciences Aaron McGuffin, M.D. Senior Associate Dean for Medical Education Adapted from NBME 3rd Edition Manual 2002 Susan M. Case, PhD and David B. Swanson, PhD http://www.nbme.org/publications/item-writing-manual-download.html

Purpose of Testing Communicate to students what is important information for them to know as practicing physicians Provide an outcome measure to verify students have achieved course objectives Identify areas of deficiency in students for purposes of remediation or to further learning Identify areas of deficiency in curriculum Teach students to interpret data and critically think to make decisions Motivate students to study

Multiple Choice Item Formats In order for a test question to be a good one, it must satisfy two basic criteria The test question must address important content Test question must be well structured and avoid flaws that benefit the test-wise examinee

What should be tested? Exam content should match course objectives Important topics should be weighted more heavily (more questions) than less important topics Congestive heart failure versus sarcoidosis Need to sample topics and sample skills (determining the diagnosis, deciding on the next best step in management) A test is a biopsy. You cannot ask everything! (at least not on the examination)

Where do I start? Start by writing an objective for the question: Ask yourself “What is it that I am wanting to see that the students know?” “Is it important that they know that?” “Does my question objective line up with the institutional objectives, course objectives, and my lecture objectives?” Focus on important concepts; don’t waste time testing trivial facts Review on-line question bank

Single Best Answer in Clinical Vignette Format Stem: A 32-year-old man has a four day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 37.80 C (1000 F), blood pressure is 130/80 mm Hg, pulse is 94/min, and respirations are 42/min and shallow. He has symmetric weakness on both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited and the plantar responses are flexor. Lead in: Which of the following is the most likely diagnosis? Options: A. Acute disseminated encephalomyelitis B. Guillain-Barre syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis Correct Answer: B

Stem: “A 32-year-old man” “has a four day history” Starts with a patient “has a four day history” Gives a time frame “of progressive weakness in his extremities.” Provides a chief complaint “He has been healthy except for an upper respiratory tract infection 10 days ago.” Provides some past medical history “His temperature is 37.80 C (1000 F), blood pressure is 130/80 mm Hg, pulse is 94/min, and respirations are 42/min and shallow. He has symmetric weakness on both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited and the plantar responses are flexor.” Provides some physical exam

Lead in: Which of the following is the most likely diagnosis? Should have a question with this stem for every disease covered, or make the diagnosis be part of a two step question, i.e. the student must figure out it’s Guillain-Barre syndrome, and then answer a question about the change in respiratory physiology that accompanies it. Options: A. Acute disseminated encephalomyelitis B. Guillain-Barre syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis Answers are capitalized, in alphabetical order, close to the same length, all “reasonable” answers, i.e. answer E was not “Tourette’s syndrome”

General Guidelines for Item Construction COVER TEST: Make sure the item can be answered without looking at the options Include as much of the item as possible in the stem. Stems should be long and options short Avoid superfluous information on Step 1 questions. Step 2 and 3 level questions should have some additional information to teach students to filter. Avoid “tricky” or overly complex items

General Guidelines for Item Construction Write options that are grammatically consistent and logically compatible with the stem Write plausible distracters Never use absolutes “always”, “never”, “all” Avoid negatively phrased items, excepts, all of the following are TRUE Which of the following is not… Which of the following is the least likely… All the following are true EXCEPT… Which of the following is true….

Lead-in examples Which of the following is the most likely diagnosis? Which of the following is the next best step in management? Which of the following enzymes does this drug inhibit? Which of the following nerves has most likely been damaged?

Two items written to assess same topic Question 1 Question 2 (Preferred) Acute intermittent porphyria is the result of a defect in the biosynthetic pathway for A. Collagen B. Corticosteroid C. Fatty acid D. Glucose E. Heme* F. Thyroxine A 33-year-old man presents with mild weakness and intermittent episodes of steady, severe crampy abdominal pain. He denies vomiting or diarrhea. One aunt and a cousin have had similar episodes. During an episode, his abdomen is distended, and bowel sounds are decreased. Neurologic examination shows mild weakness in the upper arms. These findings suggest a defect in the biosynthetic pathway for which of the following? A. Collagen B. Corticosteroid C. Fatty acid D. Glucose E. Heme* F. Thyroxine

How do I write questions about clinical entities students are not familiar with? 1. Use diseases they know Common cold, strep throat, pneumonia, appendicitis, gastroenteritis, diarrhea 2. If it involves a disease they do not know, describe it in the vignette and tell them the disease. By reading the vignette they are still learning. 3. Provide a short snapshot of the disease in the lecture material, even as “read on your own” material which begins to enhance their “life long learning” skills

How do I write questions about clinical entities I am not familiar with? 1. Read about them www.uptodate.com 2. Engage clinical faculty in actual course or a discussion about the topic In general we all love to talk about what we do Let them review the questions for clinical accuracy 3. Attend Grand Rounds presentations on topics you teach http://musom.marshall.edu/cme/cme-calendar.htm

Review of Submitted Questions

Immunology A 9-year-old boy is admitted to the hospital for an evaluation of a suspected underlying immune deficiency because of a life-threatening infection.  Careful history reveals that two years ago he underwent an emergency operation in which his spleen was removed following its rupture in a motor vehicle accident.  Which of the following is a major characteristic associated with splenectomy?   A.            Low IgM levels with increased mucosal IgA B.            Increased problems with blood-born pathogens C.            Progressive deterioration of the immune system D.            Transient depression of innate immunity Answer: B

Microbiology An 18-year-old college freshman presents to the emergency department with fever, rash and severe headache. A lumbar puncture is performed and shows numerous white blood cells and intracellular gram-negative diplococci. Antibiotic therapy is started. Which of the following organisms is most likely causing the patient’s symptoms?   A. Neisseria gonorrhoeae B. Neisseria meningitidis C. Haemophilus influenzae type b D. Mycoplasma pneumoniae E. Streptococcus pneumoniae

Pathology A 17-year-old woman presents to the emergency department with a 12 hour history of nausea, vomiting, right lower quadrant abdominal pain and an elevated white blood cell count. She is taken to surgery and her appendix is removed. Histologic examination of the appendix revealed each of the following features. Which of the following best indicates tissue necrosis? A. Dilation of blood vessels B. Interstitial edema C. Loss of cell nuclei D. Presence of neutrophils E. Swelling of cell cytoplasm

Physiology Which of the following would still be secreted in a patient who had his stomach removed as a treatment for gastric cancer? A. Insoluble mucus B. Intrinsic factor C. Pepsin D. Secretin E. Trypsin

A 64-year-old man presents to his primary care doctor complaining of three months of nausea, early satiety and persistent, dull epigastric abdominal pain. He has also noticed a 10 kg (22lb) weight loss. His primary care doctor refers him to a gastroenterologist who performs an esophagogastroduodenoscopy and finds a gastric mass. Biopsies of the mass reveal gastric carcinoma. A total gastrectomy is performed to remove the mass and chemotherapy is begun. The patient is lost to follow-up but returns to your office approximately 15 months later with numbness and weakness of his lower extremities. 34. Deficiency of which of the following substances explains this patient’s neurological symptoms?   A. Hydrochloric acid B. Intrinsic factor C. Pepsin D. Secretin E. Trypsin 35. Which of the following substances is still secreted in a patient who has undergone a complete gastrectomy?