1. MCQ’s 1: Construction of an MCQ MCQ’s 1: Construction of an MCQ

2. MCQ Check-list  This presentation will focus on: Why we write them Why we write them What type is best What type is best  What flaws are to be avoided  What content to test How to construct it How to construct it

3. Why do we test? Formative assessment Formative assessment Communicate to the students what material is important Communicate to the students what material is important Identify areas of deficiency in need of remediation or further learning Identify areas of deficiency in need of remediation or further learning Summative assessment Summative assessment Determine final grades Determine final grades Make promotion decisions Make promotion decisions Both Both Motivate students to study Motivate students to study Identify areas where the course/curriculum is weak Identify areas where the course/curriculum is weak

4. What Should Be Tested? Exam content should match course objectives Exam content should match course objectives Important topics should be weighted more heavily than less important topics Important topics should be weighted more heavily than less important topics The testing time devoted to each topic should reflect the relative importance of the topics The testing time devoted to each topic should reflect the relative importance of the topics

5. What Should Be Tested? Sampling - Performance on the sample provides a basis for estimating achievement in the broader domain that is actually of interest Sampling - Performance on the sample provides a basis for estimating achievement in the broader domain that is actually of interest Questions sample: Questions sample: TOPICS – eg MCC presentations TOPICS – eg MCC presentations SKILLS – eg Course Objectives – determining diagnosis, next step in management, etc. SKILLS – eg Course Objectives – determining diagnosis, next step in management, etc.

6. Anatomy of an MCQ Item Stem Stem Lead-in Lead-in Options Options One correct answer One correct answer Distractors Distractors

7. Types of MCQ’s True/False family Assess recall of isolated facts Assess recall of isolated facts Stems must be clear and unambiguous Stems must be clear and unambiguous Options must be absolutely true or false or examinee must decide how true options are Options must be absolutely true or false or examinee must decide how true options are Generally not recommended – usually have subtle flaws Generally not recommended – usually have subtle flaws

8. Types of MCQ’s: True/False Which of the following statements is true regarding contraception? A) oral contraceptives should not be initiated until six weeks after therapeutic abortion to avoid the risk of hypercoagulability B) an intrauterine device can provide good contraceptive protection 5 years after insertion C) oral contraceptives should not be prescribed to individuals under the age of 15 without parental consent D) lesbian women do not require counseling about contraception or sexually transmitted diseases E) oral contraceptives should not be prescribed for women over the age of 35

9. Types of MCQ’s One-Best-Answer family (A type) Options may be diagrammed: Options may be diagrammed: D C A E B Correct answer is the most likely correct Correct answer is the most likely correct Better application of knowledge, integration, synthesis and judgment Better application of knowledge, integration, synthesis and judgment Least correctMost correct

10. Types of MCQ’s: One-Best-Answer family A 24 year old university student came to see you because she is missing several days of school each month because of severe dysmenorrhea unresponsive to the OCP and NSAIDS. She also complains of deep dyspareunia and has a tender uterosacral nodule on pelvic examination. Which one of the following conditions is most likely responsible for her symptoms? A 24 year old university student came to see you because she is missing several days of school each month because of severe dysmenorrhea unresponsive to the OCP and NSAIDS. She also complains of deep dyspareunia and has a tender uterosacral nodule on pelvic examination. Which one of the following conditions is most likely responsible for her symptoms? A) Chronic pelvic inflammatory disease B) Endometriosis C) Adenomyosis D) Fibroids

11. What to test? Items can be classified by cognitive processes required to answer the question: Recall (Memory)– tests knowledge of isolated facts Recall (Memory)– tests knowledge of isolated facts Interpretation (Comprehension)– review information to reach conclusion, eg diagnosis Interpretation (Comprehension)– review information to reach conclusion, eg diagnosis Problem solving (Reasoning) – situation requires action, eg next step in management Problem solving (Reasoning) – situation requires action, eg next step in management Process depends on background of trainee Process depends on background of trainee

12. What to test? Simpler classification based on task: Application of knowledge item ** Application of knowledge item ** Reach conclusions Reach conclusions Make predictions Make predictions Select course of action Select course of action Recall item Recall item Test knowledge of isolated facts Test knowledge of isolated facts

13. What to test? Test application of knowledge using clinical vignettes Test application of knowledge using clinical vignettes Adds ‘face-validity’ Adds ‘face-validity’ Usually selects more important, less trivial Usually selects more important, less trivial Identifies student who’ve memorized but do not understand Identifies student who’ve memorized but do not understand BUT experience in patient care should not be necessary

14. What to test - summary Focus item on key concepts and principles that are essential information (without access to references) for all examinees to understand Focus item on key concepts and principles that are essential information (without access to references) for all examinees to understand Test material that is relevant to learning in pre- clinical courses and beyond Test material that is relevant to learning in pre- clinical courses and beyond Avoid items that only require recall of isolated facts Avoid items that only require recall of isolated facts Avoid esoteric or interesting topics that are not essential Avoid esoteric or interesting topics that are not essential

15. What to test - summary Test application of knowledge using clinical vignettes to pose medical decisions in patient care situations Test application of knowledge using clinical vignettes to pose medical decisions in patient care situations Focus items on common or potentially catastrophic problems Focus items on common or potentially catastrophic problems Pose clinical decisions that would be expected of a successful examinee Pose clinical decisions that would be expected of a successful examinee ie Avoid clinical situations that would be handled by a (sub)specialist ie Avoid clinical situations that would be handled by a (sub)specialist

16. Anatomy of an MCQ Item Stem Stem Lead-in Lead-in Options Options One correct answer One correct answer Distractors Distractors

17. Writing the Questions Construct stem –include all necessary information to arrive at the right answer Construct stem –include all necessary information to arrive at the right answer Choose lead-in – pose a clear question Choose lead-in – pose a clear question Choose distractors Choose distractors

18. Stem* Usually a clinical scenario Usually a clinical scenario Clear, unambiguous Clear, unambiguous Should be long relative to options Should be long relative to options Includes all pertinent information Includes all pertinent information Patient’s age/gender Patient’s age/gender Clinical setting Clinical setting Complaint Complaint Other important info – history, physical findings, test results Other important info – history, physical findings, test results

19. Lead-in* Follows stem Follows stem In the form of a question In the form of a question Should relate to the stem Should relate to the stem Should be answerable without looking at options Should be answerable without looking at options “Cover the Options test” “Cover the Options test” Should clearly communicate the learning objective Should clearly communicate the learning objective

20. Distractors Each should be selected by some, therefore all plausible, none obviously incorrect Each should be selected by some, therefore all plausible, none obviously incorrect 3 or 4 choices 3 or 4 choices Common misconceptions Common misconceptions Faulty reasoning Faulty reasoning

21. Distractors Homogeneous in content to the correct answer Homogeneous in content to the correct answer Should be in same category as correct answer– diagnosis, treatment, diagnostic test Should be in same category as correct answer– diagnosis, treatment, diagnostic test Clearly incorrect or inferior to the correct answer Clearly incorrect or inferior to the correct answer Plausible and attractive to the uninformed Plausible and attractive to the uninformed Similar to the correct answer in construction and length Similar to the correct answer in construction and length Grammatically consistent and logically compatible with the stem Grammatically consistent and logically compatible with the stem

22. General Guidelines for Item Construction Include as much of the item as possible in the stems: Long stems, short options Include as much of the item as possible in the stems: Long stems, short options Avoid negatively phrased items (except or not in the lead-in). If you must, use only with short options Avoid negatively phrased items (except or not in the lead-in). If you must, use only with short options Avoid writing questions of the form: Avoid writing questions of the form: Which of the following statements is correct? Which of the following statements is correct? Each of the following statements is correct EXCEPT Each of the following statements is correct EXCEPT Unfocussed Unfocussed Heterogeneous options Heterogeneous options

23. “Top-down” versus “Bottom-up” Top-Down Knowledge or recall question Knowledge or recall question Cite a disease and then ask what patient findings are expected Cite a disease and then ask what patient findings are expected Structured like textbooks Structured like textbooks Clinically backward

24. “Top-down” versus “Bottom-up” Bottom-up Application of knowledge Application of knowledge Gives findings and asks examinee to indicate the disease Gives findings and asks examinee to indicate the disease Examinees need to be able to synthesize information from several pages of a textbook (or notes) to answer these questions Examinees need to be able to synthesize information from several pages of a textbook (or notes) to answer these questions Clinically realistic

25. Basic Rules for MCQ’s (A type)* Focus item on important concept Focus item on important concept Item should assess application of knowledge, not recall of an isolated fact Item should assess application of knowledge, not recall of an isolated fact Stem must pose clear question – ‘Cover the options’ test Stem must pose clear question – ‘Cover the options’ test All distractors should be homogeneous All distractors should be homogeneous Avoid technical item flaws Avoid technical item flaws

26. Reference Constructing Written Test Questions For the Basic and Clinical Sciences Case & Swanson, NBME 1996 Case & Swanson, NBME 1996

27. Basic Rules for MCQ’s (A type)* Focus item on important concept Focus item on important concept Item should assess application of knowledge, not recall of an isolated fact Item should assess application of knowledge, not recall of an isolated fact Stem must pose clear question – ‘Cover the options’ test Stem must pose clear question – ‘Cover the options’ test All distractors should be homogeneous All distractors should be homogeneous Avoid technical item flaws Avoid technical item flaws

28. One last step… Get a colleague to read over your questions Get a colleague to read over your questions Detects more than one right answer Detects more than one right answer Discovers the ‘what am I thinking?’ question Discovers the ‘what am I thinking?’ question Identifies not enough information in the stem Identifies not enough information in the stem …