1. MCQ CONSTRUCTION & DESIGN Dr. Nadia Al Wardy College of Medicine & Health Sciences Medical Education Unit College of Nursing Workshop March 2014

2. 2 Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7. Knows Shows how Knows how Does A Simple Model of Competence 2 Written, Oral or Computer based assessment Performance or hands on assessment

3. 3 Knows Shows how Knows how Does Knows Factual tests: MCQ, essay type, oral….. Shows how Performance assessment in vitro: OSCE, SP-based test….. Does Performance assessment in vivo: Masked SPs, Video, Audits….. Knows how (Clinical) Context based tests: MCQ, essay type, oral….. 3

4. Instruments of Written Assessment  Open-ended question types  Modified Essay Questions (MEQs)  Short-answer questions (SAQs)  ….  Multiple-choice question types  Single-best answer  True/false  Extended-matching items (EMIs)  Key features  …. 4

5. Multiple-Choice Question Formats Multiple-choice Questions Standard True-false-type R-type Extended matching A-type True-false Questions One-best-answer Questions K-type Complex True False 5

6. Chapter 2 Multiple-Choice Item Formats Page 13 From Case & Swanson: Constructing Written Test Questions for the Basic and Clinical Sciences http://www.NBME.ORG/about/itemwriting.asp 6

7. MULTIPLE-CHOICE QUESTIONS A-Type (Single Best Answer)

8. A 35-year old man has had a stomach ache all afternoon. He ate the following lunch: two big McDonalds hamburgers, an ice cream shake, and large fries. What is the most likely diagnosis? A. Abdominal aneurysm B.Appendicitis C.Bowel obstruction D.Cholecystitis E. Colon cancer F. Pancreatitis G. Too much lunch Components of an A-type Question Lead-in Vignette, Scenario, Or Stem Options A, B, C, D, E & F are distractors Option G is the key 8

9. Sample Good Question & ‘Cover The Options’ Rule A 32-year old man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. What is the most likely diagnosis? A. Acute disseminated encephalomyelitis B. Guillain-Barré syndrome C. Myasthenia gravis D. Poliomyelitis E. Polymyositis 9

10. D C AE B Least Most Correct Answer Note: Options must be homogeneous (eg, all diagnoses, all muscles). Diagram of Rank-Ordered Options for a Good One-Best-Answer Item 10

11. A 40 year old man is admitted to the Emergency Room with renal calculi and is complaining of moderate to severe flank pain and nausea. His temperature is 38.2 0 C. What is the priority nursing goal for this patient? A. maintain fluid and electrolyte balance B. Control nausea C. Manage pain D. Prevent urinary tract infection 11 Sample Good Question & ‘Cover The Options’ Rule

12. Which of the following is true about pseudogout? A.It occurs frequently in women. B.It is seldom associated with acute pain in a joint. C.It may be associated with a finding of chondrocalcinosis. D.It is clearly hereditary in most cases. E.It responds well to treatment with allopurinol. Sample Flawed Question & ‘Cover The Options’ Rule 12

13. D Inheritance Rx E Gender A Associations False True B C Sample Flawed One-Best-Answer Item Which of the following is true about pseudogout? A.It occurs frequently in women. B.It is seldom associated with acute pain in a joint. C.It may be associated with chondrocalcinosis. D.It is clearly hereditary in most cases. E.It responds well to treatment with allopurinol. 13

14. The #1 Rule For One-Best-Answer Questions A 24-year old man comes to the urgent care unit….. What is the most likely cause of his pain? A.cholecystitis B.diverticulitis C.kidney stone D.pancreatitis E.peptic ulcer disease 14

15. If you want your students to go beyond memorizing facts, your tests must require more 15

16. Two Rules 1. Ensure that each question satisfies the ‘cover the options’ rule. 2. Assess examinee ability to apply their knowledge rather than simply their ability to recall isolated facts. 16

17. Item Assessing Recall of an Isolated Fact What arterial blood gas analysis is most likely to show in patients with cardiogenic shock? A. Hypoxemia with normal pH B. Metabolic acidosis C. Metabolic alkalosis D. Respiratory acidosis E. Respiratory alkalosis 17

18. Alternate Item Assessing Application of Knowledge: Diagnosis A 74-year-old woman is brought to the emergency department because of crushing chest pain. She is restless, confused, and diaphoretic. On admission, temperature is 36.7 C, blood pressure is 148/78 mm Hg, pulse is 90/min, and resp are 24/min. During the next hour, she becomes increasingly stuporous, blood pressure decreases to 80/40 mm Hg, pulse increases to 120/min, and respirations increase to 40/min. Her skin is cool and clammy. An ECG shows sinus rhythm and 4 mm of ST segment elevation in leads V2 through V6. Arterial blood gas analysis is most likely to show 18

19. A 74-year-old woman is brought to the emergency department because of crushing chest pain. She is restless, confused, and diaphoretic….. Arterial blood gas analysis is most likely to show A.Hypoxemia with normal pH B.Metabolic acidosis C.Metabolic alkalosis D.Respiratory acidosis E.Respiratory alkalosis Alternate Item Assessing Application of Knowledge: Diagnosis 19

20. Sample MCQ Items: Isolated Fact vs Case-Based Question What is the most likely renal abnormality in children with nephrotic syndrome and normal renal function? OR A 2-year-old boy developed swelling of his eyes and ankles over the past week. BP 100/60; pulse 110; R 28. PE shows eyelid swelling; 2+ pitting edema of his ankles; abdominal distension with positive fluid wave. Serum: creatinine 0.4; albumin 1.4; cholesterol 569 mg/dL. Urinalysis: 4+ protein and no blood. What is the most likely diagnosis? 20

21. How To Develop Better Multiple-Choice Questions What are the signs and symptoms of multiple sclerosis? A 29-yr-old woman has a one-day history of (insert description of presenting complaint, history, physical exam here). What is the most likely diagnosis? 21

22. What is the best drug to treat Otitis media? A 6-yr-old boy has a one-day history of (insert description of presenting complaint, history, physical exam here). What is the most appropriate therapy? How To Develop Better Multiple-Choice Questions 22

23. What immunizations should be given to adults? A 37-yr-old man comes to the physician for a routine health maintenance visit (insert relevant history, physical exam here). What immunizations should be administered? How To Develop Better Multiple-Choice Questions 23

24. Structure of a Patient Vignette Age, Gender (eg, A 45-year-old man) Site of Care (eg, comes to the emergency dept.) Presenting Complaint (eg, because of headache) Duration (eg, that has continued for 2 days) Patient History (with Family History ?) Physical Findings +/- Results of Diagnostic Studies +/- Initial Treatment, Subsequent Findings, etc. 24

25. Stages in writing MCQs 1. Plan the question 2. Write the question 3. Test the question 25

26. 1.Plan the question

27. Ensure question deals with an important and useful aspect of the subject  Avoid trivia  Include:  Common issues  Serious conditions  Important or frequent misunderstandings  Appropriate to expected level 27

28. Test knowledge around a central theme  Be clear about exactly what fact, concept or generalization is being tested  This central theme should be clearly defined in the stem  All options should refer to this theme 28

29. Decide on the ability you wish to test  Recall or recognition  Interpretation  Problem-solving 29

30. Consider how the information should be presented  Word description  Photograph  Radiograph  Tracing (e.g. ECG, visual field)  Diagram (e.g. anatomical)  Movie film  Amplified sound (e.g. heart sounds) 30

31. 2.Write the Question

32. Use simple and clear language  Avoid unfamiliar technical or unusual words  Check for ambiguity amongst fellow examiners or students  Anxious examinees imagine traps in questions 32

33. Write stem as a question or incomplete statement  If a question, each option must be a possible answer  If an incomplete statement, each response must follow grammatically and logically from the stem; i.e. stem + each option must be a complete statement  Avoid stems with just one or two words: Hair : A. growth in man shows the phenomenon of asynchronous cyclic growth. B.…………… 33

34. Keep reading time to a minimum  Include only material or explanations necessary for answering the question In considering your management of a 28-year-old fashion designer with bleeding from the nose, you would be aware that the most frequent cause of epistaxis at any age is A. ……………… 34

35. Make the responses clear and concise  If a response requires a large number of words to qualify what is meant by it, the questions is probably too complex for an MCQ  Include all common elements in the stem The third part of the duodenum A. is anterior to the uncinate process of the pancreas B. is anterior to the common bile duct C. is anterior to the right ureter D. is anterior to the superior mesenteric artery E. is anterior to the inferior mesenteric vein 35

36. Avoid clues to the correct answer  Use of stereotyped or standard phraseology “the pituitary gland is the master of the endocrine orchestra”  Unequal length of alternatives  Obvious inconsistencies between stem and a response An intercostal tube may be placed in a patient with a flail chest but without a pneumothorax or haemothorax to A. remove air B. remove blood C. prevent onset of pulmonary contusion D. prevent development of a tension pneumothorax when the patient is placed on positive pressure ventilation E. eliminate paradoxical movement of the flail segment 36

37. Avoid clues to the correct answer [contd]  Using opposite of the correct answer as a distractor Paralysis of a vocal cord in a patient with bronchogenic carcinoma is most likely to be: A. bilateral B. on the right side C. on the left side D. due to metastasis to the larynx E. due to cigarette smoking  Use of absolutes (e.g. “always”, “only”, “never”) Adamantinoma of the mandible A. never metastasize B. usually metastasize to lymph nodes C. ………… 37

38. Avoid clues to the correct answer [contd]  Use of synonyms or overlapping alternatives  Use of similar distractors, making the correct response more conspicuous Antidiuretic hormone is synthesized in A. the anterior lobe of the pituitary gland B. the posterior lobe of the pituitary gland C. the pars intermedia of the pituitary gland D. the pars nervosa of the pituitary gland E. the hypothalamus 38

39. Use plausible and logical distractors  Obvious distractors increase chances of guessing  Derive distractors from common errors made by students  Derive distractors from common misconceptions held by students  If legitimate distractors cannot be found, recast question in a different format 39

40. Use eponyms, acronyms and abbreviations only if you expect students to know them  Eponym: Named after a person (e.g. pouch of Douglas; Meckel’s diverticulum)  Acronym: Combination of first letters (e.g. ACTH, CK)  Abbreviation: Shorter form of a word (e.g. ht. sd.) 40

41. Specify units  Particularly when using laboratory units, as they can vary from one laboratory to another  The normal range for a given laboratory may have to be quoted 41

42. Indicate opinion or authority if dealing with unestablished facts  Indicate whose opinion or what authority the question is based on According to the passive equilibrium model of Kokko (1975) the main factor responsible for the concentration of urine in the loop of Henle is A. the relative impermeability of the descending limb of the loop of Henle to sodium chloride B. the movement of urea out of the apex of the loop of Henle C. ………… 42

43. Minimize negatively phrased questions  Avoid double negatives  They confuse examinees  If absolutely necessary, emphasize the word(s) conveying a negative aspect by using capitals or underlining Haematemesis is not unexpected in each of the following, except A. ……… B. ……… C. ……… 43

44. Beware of unintentionally asking two questions in the same option  Avoid especially if the answer to one is ‘true’ and to the other is ‘false’. Psychological process and gastrointestinal function are A. clearly separable and rarely correlated B. ……… 44

45. Avoid “all of the above” and beware of “none of the above”  After partial gastrectomy, gastric emptying is abnormal because A. the pylorus is removed B. there is interference with the pacemaker C. the gastric remnant is smaller D. all of the above E. none of the above 45

46. 3.Test the question

47. Get agreement from fellow examiners that:  the question is worth asking  it is clearly and unambiguously stated  distractors are reasonable and plausible 47

48. Test the question on students if possible  may represent a different interpretation of the question  may reveal what the question was actually testing  may have an opinion on importance and validity  can indicate their answering strategy  may point out errors in the question 48

49. MULTIPLE-CHOICE QUESTIONS R-Type (Extended-Matching Items, EMIs)

50. Chapter 6 Extended Matching Page 69 From Case & Swanson: Constructing Written Test Questions for the Basic and Clinical Sciences http://www.NBME.ORG/about/itemwriting.asp 50

51. Theme:Cardiovascular drugs Options:A. Alpha agonists B. Alpha blockers C. ACE inhibitors D. Beta agonists……. K. Vasoconstrictors Lead-in:For each patient, select the most appropriate drug class Item stems: 1. A 40-year-old man with sudden onset of severe headache, dizziness and vomiting, BP 260/130 mmHg; encephalopathy and grade IV retinopathy 2. A 55-year-old man with mild, chronic hypertension, tachycardia and other symptoms of mild congestive cardiac failure. Previous history of asthma and gout. 51 Components of an R-type Question

52. 1.Decide on the theme of your set (eg, Establishing a diagnosis of patients with abdominal pain) 2.Develop the lead-in for your set 3.Write the options for your set (eg, appendicitis, cholecystitis) 4.Write a patient description for each of the diagnoses in the list 5.Ask a colleague to review your items (patient descriptions) without the correct answers noted Writing R-Set Extended Matching Items 52

53. Topic: Diagnosis of Patient With Abdominal Pain Lead In: For each patient with abdominal pain, select the most likely diagnosis. 53

54. A.Abdominal aneurysm K. Kidney stone B.Appendicitis L. Mesenteric adenitis C.Bowel obstruction M. Mesenteric artery thrombosis D.Cholecystitis N. Ovarian cyst - ruptured E.Colon cancer O. Pancreatitis F.Constipation P. Pelvic inflammatory disease G.Diverticulitis Q. Peptic ulcer disease H.Ectopic pregnancy - ruptured R. Perforated peptic ulcer I.Endometriosis S. Pyelonephritis J.Hernia T. Torsion For each patient with abdominal pain, select the most likely diagnosis. Options and Lead-in: Diagnosis 54

55. Structure of a Patient Vignette Age, Gender (eg, A 45-year-old man) Site of Care (eg, comes to the emergency dept.) Presenting Complaint (eg, because of headache) Duration (eg, that has continued for 2 days) Patient History (with Family History ?) Physical Findings +/- Results of Diagnostic Studies +/- Initial Treatment, Subsequent Findings, etc. 55

56. 1. A 25-year-old woman has sudden onset of persistent right lower abdominal pain that is increasing in severity. She has nausea without vomiting. She had a normal bowel movement just before onset of pain. Examination shows exquisite deep tenderness to palpation in right lower abdomen with guarding but no rebound; bowel sounds are present. Pelvic examination shows a 7-cm, exquisitely tender right sided mass. Hematocrit is 32%. WBC is 18,000/mm 3. Serum amylase activity is within normal limits. Test of the stool for occult blood is negative. Ans: B Sample Diagnosis Items: Abdominal Pain 56

57. 2. A 27-year-old woman has had abdominal pain for the past 18 hours. The pain was first noted in epigastrium and has moved to the right lower quadrant of the abdomen. She has mild nausea and anorexia, but no vomiting. She has a low- grade fever. Her last bowel movement was normal yesterday. Her last menstrual period was 4 weeks ago. Examination shows marked right lower quadrant abdominal tenderness with guarding and rebound. Pelvic exam reveals marked tenderness on right side without masses. Hematocrit is 37%. WBC is 14,500/mm 3. Serum amylase activity is within normal limits. Test of the stool for occult blood is negative. Sample Diagnosis Items: Abdominal Pain Ans: C 57

58. 3. An 84-year-old man in a nursing home has increasing poorly localized lower abdominal pain recurring every 3-4 hours over the past 3 days. He has no nausea or vomiting; the last bowel movement was not recorded. Examination shows a soft abdomen with a palpable, slightly tender, lower left abdominal mass. Hematocrit is 28%. WBC is 10,000/mm 3. Serum amylase activity is within normal limits. Test of the stool for occult blood is positive. Ans: E Sample Diagnosis Items: Abdominal Pain 58

59. Rules for Use in Reviewing Sets Options should be single terms, or very short phrases. Make sure the options are homogeneous. A lead-in must be specified for each set to make the examinee task explicit by specifying the relationship between the items and the options. Items should assess application of knowledge, not just one-word associations. Distractors should be plentiful. Add distractors that would be used in a traditional item if necessary. 59

60. A.is motion sickness I. are completely controlled B.have no effects on people J. cause plant & eye damage C.indirectly increase CO 2 K. are negligible D.cause death L. increase risk of skin cancer E.increased odor sensitivity M. cannot be controlled F.is a reduction in visibility N. excess acute respiratory G.esthetics, economics, health illness among children H.products of fossil fuel O. contrary to public opinion combustion 1.Factors that people consider when evaluating air quality 2.A principal effect of particulate matter in air 3.The products of photochemical smog Sample Extended Matching Set: Flawed 60

61. Options and Lead-in: Microbiology A.AdenovirusL.Haemophilus influenzae B.Aspergillus fumigatusM.Histoplasma capsulatum C.Bacillus anthracis N.Mycobacterium tuberculosis D.Candida albicansO.Mycoplasma pneumoniae E.Chlamydia psittaciP.Neisseria gonorrhoeae F.Coccidioides immitisQ.Neisseria meningitidis G.CoronavirusR.Pneumocystis carinii H.Corynebacterium diphtheriaeS.Rhinovirus I.Coxiella burnetiiT.Streptococcus pneumoniae J.CoxsackievirusU.Streptococcus pyogenes K.Epstein-Barr virus(group A) For each description, select the associated pathogen. 61

62. An encapsulated gram-positive organism that usually grows in pairs or short chains. This stem assesses recall of isolated facts, rather than application of knowledge. It is more like a crossword puzzle question than an item for a medical school examination. Sample Poor Item: Microbiology T 62

63. 1. A 7-year-old girl has a high fever and a sore throat. There is pharyngeal redness, a swollen right tonsil with creamy exudate, and painful right submandibular lymphadenopathy. Throat cultureon blood agar yields numerous small b-hemolytic colonies that are inhibited by bacitracin. 2. For the past week, an 18-year-old man has had fever, sore throat, and malaise with bilaterally enlarged tonsils, tonsillar exudate, diffuse cervical lymphadenopathy, and splenomegaly. There is lymphocytosis with atypical lymphocytes. The patient tests positive for heterophil antibodies. Sample Good Items: Microbiology UK 63

64. Defining Content Domain Using Chief Complaints WheezingPalpitationsChest Pain ItchingDiarrheaEasy Bruising FatigueAnemiaLymphadenopathy JaundiceBack PainConfusion AscitesJoint Pain Hematemesis/Melena CoughWeight LossShortness of Breath FeverDizzinessAbdominal pain SyncopeGI Bleeding 64

65. Are There Advantages of Extended Matching R- Sets over Traditional 5-Option Single Items? 65

66. Percentage of Students Selecting Each Option When Item Presented in a 5-Option and 15-Option Format AB CDEFGHIJKLMN O 5-option 81 *40 14*1 15-option 259 213 1210 1 72280 0 66

67. Frequency Distribution of Scores Comparing Sets with 5 or 15 -Options Percent Correct Score Percent of Students Case SM, Swanson DB. Evaluating diagnostic pattern recognition: a psychometric comparison of items with 15, 5, and 2 options. Paper presentation, Annual Meeting of the AERA. San Francisco, CA: March 1989. 67

68. ACTIVITY

69. Checklist  Does it assess application of knowledge?  Does it assess key concepts and principles that are essential information?  Does it assess relevant material?  Does it satisfy the “cover the options rule”? 69

70. 1.Decide on the theme of your set (eg, Establishing a diagnosis of patients with abdominal pain) 2.Develop the lead-in for your set 3.Write the options for your set (eg, appendicitis, cholecystitis) 4.Write a patient description for each of the diagnoses in the list 5.Ask a colleague to review your items (patient descriptions) without the correct answers noted Checklist 70