1

2 Dr. Shama Mashhood Dr. Shama Mashhood Medical Educationist Medical Educationist & Coordinator Coordinator Question Review Committee Question Review Committee

3 “Multiple Choice Questions- One best type”

4 Overview of the Presentation Strengths and Limitations of one best type MCQs. Characteristics of good MCQs Items. Checklist for review of MCQs.

5 STRENGTH OF MCQs Valid Valid Reliable Reliable Standardized Standardized Objective Objective Sample broad content Sample broad content Easy to score & analyze. Easy to score & analyze. Item analysis, difficulty & discrimination Item analysis, difficulty & discrimination Can test various levels of cognition Can test various levels of cognition

6 LIMTATIONS OF MCQs Difficult to prepare. Difficult to prepare. Time consuming. Time consuming. Assess recall (If carefully not constructed) Assess recall (If carefully not constructed)

7 Higher order problem solving and decision making ability Higher order problem solving and decision making ability Scoring can be affected by reading ability. Scoring can be affected by reading ability. Difficult to determine why candidate select wrong response Difficult to determine why candidate select wrong response

8 Components A 68 years old man presents with colicky abdominal pain. Within three days of admission to the hospital, he becomes jaundiced and develops high-grade fluctuating pyrexia. The most likely diagnosis in this patient is A. infective hepatitis B. cirrhosis of liver C. a stone impacted in the common bile duct D. carcinoma of the head of pancreas E. an intra hepatic abscess KEY: C Questio n Stem Lead in statemen t Option

9 Item Shape? Long stem with all essential information Short Option

10 Item Shape? Short item stem Long Option

11 Item Shape? Item Stem Uneven option

12 Characteristics of good Item Identify Objectives, level and domain of learning Identify Objectives, level and domain of learning Formation of table of specification. Formation of table of specification. Simple recall to problem solving questions. Simple recall to problem solving questions.

13 GOOD ITEM STEM GOOD ITEM STEM 1.Must be clear, concise, straightforward and a complete question. 1.Must be clear, concise, straightforward and a complete question. E.g. Multiple choice question is Valid Valid Reliable Reliable Objective Objective Feasible Feasible

14 Recall Item: Recall Item: e.g. What area is supplied with blood by the posterior inferior cerebellar artery? e.g. Which of the following presents as Chronic (longer than 3 months) airspace disease on a chest radiograph? Streptococcal pneumonia Streptococcal pneumonia Adult respiratory distress syndrome Adult respiratory distress syndrome Pulmonary edema Pulmonary edema Pulmonary alveolar proteinosis Pulmonary alveolar proteinosis

15 Item with a vignette/case/scenario Item with a vignette/case/scenario measuring application of knowledge measuring application of knowledge e.g. A 35 year woman presented with a 4 month history of dysnea, low grade fever, cough and fatigue. Given the following chest radiograph, what is the most likely diagnosis? Streptococcal pneumonia Adult respiratory distress syndrome Adult respiratory distress syndrome Pulmonary edema Pulmonary edema Pulmonary alveolar proteinosis Pulmonary alveolar proteinosis Streptococcal pneumonia Streptococcal pneumonia

16 3. Worded positively. e.g. Which of the following best distinguishes small airway disease from interstitial lung disease on chest CT?

17 4. No negative phrasing b/c negative stem less effective and difficult to understand. e.g. “Which is not true" or “all of the following except” e.g. Which of the following is NOT a characteristic CT finding of small airway disease?

18 6. Must have a central idea which fits grammatically with the option. e.g. If the stem is in past tense,all options should be in past tense. If answer demands for plural, then all options should be in plural.

19 Avoid Avoid 7. Absolute terms e.g. ‘Always’, ‘Never’, ‘all’, ‘none’ b/c few ideas or situations are absolute or universally true. b/c few ideas or situations are absolute or universally true. 8. Terms cues for test wise examinees e.g. ‘may’, ‘could’, ‘can’. (b/c anything is possible) 9. ‘Seldom’, ‘rarely’, ‘occasionally’, ‘sometimes’, ‘few’,& ‘many'. (not uniformly understood) 10. Excessive verbiage 11. Pronoun ‘it, he, she & you’.

20 GOOD LEAD IN Focus on one aspect of the disease of the condition. Focus on one aspect of the disease of the condition. e.g. Diagnosis, Management

21 GOOD OPTIONS Grammatically consistent with stem Grammatically consistent with stem Similar to correct answer in construction and length Similar to correct answer in construction and length Homogenous in content e.g. all diagnosis, test, treatments. Homogenous in content e.g. all diagnosis, test, treatments. Avoid ‘none of the above’ or ‘all of the above’. Avoid ‘none of the above’ or ‘all of the above’.

22 Keep options in logical order e.g. numerical,chronological,alphabetical, ascending order. Keep options in logical order e.g. numerical,chronological,alphabetical, ascending order. Change position of key responses. Change position of key responses. Keep options independent and not overlapping. Keep options independent and not overlapping.

23 Check list Reviewing multiple choice items

24 1. Is this the most appropriate type of item to use? 2. Does each item stem present a meaningful problem? 3. Are the item stems free of irrelevant material? 4. Are the item stems stated in positive term (if possible)?

25 5.If used has the negative wording been given special emphasis (e.g. capitalized) 6.Are the alternatives grammatically consistent with the item stem? 7.Are the alternative answers brief and free of unnecessary words? 8.Are the alternatives similar in length and form?

26 9.Is there only one correct or clearly best answer? 10.Are the distracters plausible to low achievers? 11.Are the items free of verbal clues to the answer? 12.Are the verbal alternatives in alphabetical order?

27 13.Are numerical alternatives in numerical order? 14.Have none of the above and all of the above been avoided (or used sparingly and appropriately)? 15.If revised, are the items still relevant to the intended learning outcomes? 16.Have the items been set aside for a time before reviewing them?

28 QUESTIONS ?