1. Writing Multiple Choice Questions Amy S. Oxentenko, MD, FACP, FACG, AGAF Mayo Clinic, Rochester

2. Relevant Relationships:  Consulting work for MKSAP 14-17  On the IM-ITE exam committee Disclosures

3. Outline/Objectives Define the test taker level before writing List components of a multiple-choice question  Objective, stem, lead line, options  Annotated description of answers Evaluate your questions

4. Who Are You Writing For? Med school GI pathophysiology course?  More details; physiology on USMLEs Resident end-of-rotation exam?  Presentation, testing, management GI Fellow board review course?  More advanced understanding of above IM or GI CME course?  Very general vs very specific

5. What are the Stakes of the Questions? Board certification? Passing a class/rotation? Simple knowledge assessment? Solely for interaction?

6. The Stakes Reproducible Accurate Learning

7. What Should Be Tested? For a clerkship/CME course  Match goals/objectives For a standardized exam  Match the targeted learner level

8. How Often Will Learner Encounter Information Frequent  More relevant to test often  Atypical manifestations Infrequent  Test the “can’t miss” things  Life threatening  Typical manifestations

9. Types of Questions Single best answer  With clinical scenario Factoid questions All of the following EXCEPT Matching True/False A and B, A and C, all of the above

10. What are the Components of a Multiple Choice Question (MCQ)?

11. STEM OPTIONS LEAD LINE Question Components

12. EXPLANATION OBJECTIVE ANSWER Question Components

13. PLEASE TAKE THE PRETEST.

14. Steps to Writing the MCQ 1.Determine the objective 2.Create the lead line and options 3.Draft the stem 4.Write the explanation, if required

15. The Objective

16. Example: Objective Recall alcoholic hepatitis What is wrong with this?

17. Example: Objective Treat a patient with alcoholic hepatitis with corticosteroids in the absence of contraindications More focused

18. The Objective Succinct sentence Specifies a clinical skill to be learned Uses action verbs to delineate a goal  Recognize, diagnose, treat, manage Avoid vague action verbs  Remember, recall, know Allows you to match curriculum

19. VERB + TOPIC + MODIFIER Objective Format

20. Action Verbs for Objectives

21. Examples of Objectives: The Good, the Bad, the Ugly Good:  Diagnose niacin deficiency in a patient with carcinoid syndrome Bad:  Recall features of niacin deficiency Ugly:  Understand niacin deficiency

22. Other Objective Examples Select appropriate testing needed before beginning a biologic agent on an IBD patient. Evaluate for splenic vein thrombosis in a patient with isolated gastric varices. Manage a patient with celiac disease who has persistent symptoms on a GFD. Identify peptic ulcers with stigmata for rebleeding that require endoscopic therapy.

23. Lead Line

24. Example: Lead Line A 56-year old alcoholic stumbles into the ER, ataxic, with nystagmus on examination. Although he appears confused and intoxicated, his blood alcohol level is normal. You start IV glucose and symptoms worsen. The next step would be to administer A.Niacin B.CT scan of the head C.Thiamine D.Insulin E.Dismissal to home What is wrong with this?

25. Example: Lead Line A 56-year old alcoholic stumbles into the ER, ataxic, with nystagmus on examination. Although he appears confused and intoxicated, his blood alcohol level is normal. You start IV glucose and symptoms worsen. What is the next best step in management? A.Niacin B.CT scan of the head C.Thiamine D.Insulin E.Dismissal to home Keep lead line general; avoid syntax errors

26. Lead Line Most commonly used lead lines:  Which of the following is the most likely diagnosis?  What is the next best step in management?  Half treatment options, half diagnostic options Avoid negative lead lines:  All of the following EXCEPT  Which of the following is LEAST likely  Which would you NOT do

27. Lead Line Keep the lead line generic  “Next step” or “management”  Allows diagnostic & therapeutic options Avoid writing syntax or grammatical errors that help exclude answers “Cover the options” rule  Should be able to guess the answers

28. The Options

29. Example #1: Options What is the next best step in management? A.Add IV metronidazole B.Continue po vancomycin C.Start cholestyramine D.Begin rifaximin E.Stop the po vancomycin What is wrong with this?

30. Example #1: Options What is the next best step in management? A.Add IV metronidazole B.Continue po vancomycin C.Start cholestyramine D.Begin rifaximin E.Stop the po vancomycin Mutually exclusive

31. Example #2: Options What is the next best step in management? A.Metronidazole 500 mg po TID x 14 days B.Vancomycin 250 mg IV QID x 10 days C.Metronidazole 500 mg IV QID x 14 days D.Vancomycin 125 mg po QID x 14 days What is wrong with this?

32. Example #2: Options What is the next best step in management? A.Metronidazole 500 mg po TID x 14 days B.Vancomycin 250 mg IV QID x 10 days C.Metronidazole 500 mg IV QID x 14 days D.Vancomycin 125 mg po QID x 14 days Testing too many concepts! Drugs, dose, route, duration

33. Example #3: Options The cause of diarrhea in most patients with Zollinger-Ellison syndrome is: A. Antacid use B. High glucagon levels C. Fat malabsorption from pancreatic lipase inactivation by acid production D. Bacterial overgrowth What is wrong with this?

34. Example #3: Options The cause of diarrhea in most patients with Zollinger-Ellison syndrome is: A. Antacid use B. High glucagon levels C. Fat malabsorption from pancreatic lipase inactivation by acid production D. Bacterial overgrowth Correct answer longer, detailed

35. The Options 4 or 5 options (less may be better) One is not right and the rest 100% wrong C E A B D Wrong Right

36. The Options 4 or 5 options (less may be better) One is not right and the rest 100% wrong One is correct, the rest plausible C E ABD Least correct Most correct

37. Options Avoid mutually exclusive options  Increases/decreases/same  Always/never; stop/continue Each option should test one concept  Drug OR dose OR route OR duration Keep options brief; similar in length Options should be homogeneous  Length, complexity

38. Stem

39. The Stem Chronologic order of presentation  Age, gender (avoid race unless needed)  Site of visit (ER, clinic, hospital)  Chief complaint (add features)  PMH/Meds/FH/SH (relevant or distracter)  Vitals/Exam/Labs/Tests (pertinent)

40. The Stem Fine to have details to make the other options attractive; avoid extraneous info It takes time to read; make sure it is fitting in with time to administer test Organize labs if many

41. The Explanation

42. Explain why right answer correct first Explain why wrong answers incorrect – Do this chronologically Example for explanation order: – If correct answer is B – B, A, C, D, E

43. How Do You Analyze Your Questions?

44. Question Performance Goal for correct answer 50-70%  40-90% may be allowed at times  Difficulty score of > 90% (too easy)  Difficulty score of < 40% (too hard) See if it discriminates:  Top test takers  Bottom test takers

45. Performs well; difficulty ok and discriminating (A)(B)(C)(D) Answer* Top 50%78094 Bottom 50%1944289 Total1362187 Option0.62

46. Performs poor; difficulty ok, non-discriminating Option(A)(B)(C)(D) Answer* Top 50%620731 Bottom 50%800713 Total710722 Difficulty0.71

47. Performs poor; difficulty low (hard), non-discriminating Option(A)(B)(C)(D) Answer* Top 50%345250 Bottom 50%651538 Total548344 Difficulty0.44

48. http://www.nbme.org/publications/item-writing-manual-download.html

50. The Pretest: How Did You Do?

51. Avoid Duplicating Terms 1.The primary role of the denter is to move the a)orme b)latten c)denter pin d)stagle

52. Avoid Repeat Digits, Look at Pattern 2.How many steps are in an average carrum? a)3.7 b)370 c)240 d)37

53. Avoid Absolute Terms: Never, Always, Every, All, None 3. The grotto is frequently humid because a)every ictal is fabliterous b)langers are always gantering c)reslins are often dewinest d)culers are never liblis

54. Avoid Inhomogeneous Options 4. Rigniter is a a)form of glatnick b)prestigious subset of the family of honorary gloutnins c)signal by a sellant d)crown of slessings

55. Avoid Repeating the Correct Option More Than Others 5. Which of the following pairs contributed the Sterlik track? a)Smith & Johnson b)Jones & Smith c)Johnson & Thomson d)Smith & Davis

56. Avoid Syntax or Grammatical Errors 6. The tarnon is proceeded by a a)ofler b)argling c)isterin d)glantern

57. Questions?

58. Summary Know your learner level Writing order: objective, lead line, options, stem Avoid errors in MCQ writing Evaluate the effectiveness of your MCQs

59. Thank you! oxentenko.amy@mayo.edu