PREPARING EFFECTIVE MULTIPLE CHOICE EXAMINATION QUESTIONS PATRICK DUFF, M.D.

LEARNING OBJECTIVES Describe the levels of complexity of an examination question Describe the levels of complexity of an examination question Describe the types of examination questions Describe the types of examination questions Describe desirable and undesirable features of examination questions Describe desirable and undesirable features of examination questions

KEY REQUIREMENTS FOR AN EXAMINATION REQUIREMENTDEFINITION Validity Measures what it is supposed to measure Reliability Provides reproducible and consistent results Discrimination Distinguishes between the excellent, very good, and fair student

COMPLEXITY OF EXAMINATION QUESTIONS CLASSIFICATIONDESCRIPTION Recall Recall of a specific fact Interpretation Identification and interpretation of illustrative material Problem solving Recall, interpretation, and practical application

TYPES OF EXAMINATION QUESTIONS TYPEREMARKS C – either, or, both, neither Unacceptable K – select among several alternatives Unacceptable True – false Unacceptable

TYPES OF EXAMINATION QUESTIONS TYPEREMARKS A – select the one best answer Ideal question – appropriate for recall, interpretation, problem solving B – matching Appropriate for recall, interpretation, and simple management problems

EXAMPLE RECALL Which of the following is the most important risk factor for squamous cell carcinoma of the lung? Which of the following is the most important risk factor for squamous cell carcinoma of the lung? A. Age > 50 B. Diabetes C. Smoking D. Obesity

EXAMPLE INTERPRETATION A 24 year old nulliparous woman at 16 weeks gestation had the following values on her serum analyte screening test: A 24 year old nulliparous woman at 16 weeks gestation had the following values on her serum analyte screening test: AFP …below the mean HCG…above the mean Estriol…below the mean Inhibin…above the mean

EXAMPLE INTERPRETATION Which of the following chromosome abnormalities is her baby most likely to have? Which of the following chromosome abnormalities is her baby most likely to have? A. Triploidy B. Trisomy 21 C. 45, X D. Trisomy 18

EXAMPLE INTERPRETATION A term neonate has hepatosplenomegaly, thrombocytopenia, and the diffuse rash noted in the photograph. A term neonate has hepatosplenomegaly, thrombocytopenia, and the diffuse rash noted in the photograph.

EXAMPLE INTERPRETATION Which congenital infection is most likely? Which congenital infection is most likely? A. Parvovirus A. Parvovirus B. CMV B. CMV C. HIV C. HIV D. Mumps D. Mumps

EXAMPLE PROBLEM SOLVING A 22 year old male has a 48 hour history of malaise, fever (39 deg C), shaking chills, and cough productive of rust- colored sputum. On physical examination, there is dullness to percussion in the lower left lung field, and tubular breath sounds can be heard in this area. A 22 year old male has a 48 hour history of malaise, fever (39 deg C), shaking chills, and cough productive of rust- colored sputum. On physical examination, there is dullness to percussion in the lower left lung field, and tubular breath sounds can be heard in this area.

EXAMPLE PROBLEM SOLVING Which of the following is most appropriate for treatment of this patient? Which of the following is most appropriate for treatment of this patient? A. Isoniazid B. Fluconazole C. Ceftriaxone D. Acyclovir

APPROPRIATE MIX OF EXAMINATION QUESTIONS

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION Determine a single clear objective prior to writing the question Determine a single clear objective prior to writing the question Have only one objective per question Have only one objective per question Make sure the question is at an appropriate level of difficulty for the target audience Make sure the question is at an appropriate level of difficulty for the target audience

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION Correct response should be obvious to the well-informed examinee before the answers are read Correct response should be obvious to the well-informed examinee before the answers are read If the examinee has to read all of the answers before deciding, the question is probably a K-type “disguised” as an A-type If the examinee has to read all of the answers before deciding, the question is probably a K-type “disguised” as an A-type

TYPE A QUESTION GOOD EXAMPLE Which of the following is the most common complication of multiple pregnancy? Which of the following is the most common complication of multiple pregnancy? A. Gestational diabetes B. Placenta previa C. Postpartum hemorrhage D. Preterm delivery

BEWARE THE PSEUDO-A QUESTION All of the following statements about preterm delivery are true except All of the following statements about preterm delivery are true except The frequency is 10 %. The frequency is 10 %. It can be prevented by use of progesterone. It can be prevented by use of progesterone. It is more common in whites than in African-Americans. It is more common in whites than in African-Americans. It may result in neonatal RDS. It may result in neonatal RDS.

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION Include sufficient information in the stem, but avoid unnecessary verbiage Include sufficient information in the stem, but avoid unnecessary verbiage Be certain the question has a clear, unambiguous answer Be certain the question has a clear, unambiguous answer

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION Include 4 to 5 possible answers Include 4 to 5 possible answers Be certain the incorrect answers are reasonably plausible Be certain the incorrect answers are reasonably plausible

TYPE A QUESTION BAD EXAMPLE What is the risk of pregnancy loss after a second trimester amniocentesis? What is the risk of pregnancy loss after a second trimester amniocentesis? A. 0.5% B. 10 % C. 25 % D. 50 %

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION Do not use “none of the above” or “all of the above” as distractors Do not use “none of the above” or “all of the above” as distractors Confusing Confusing Rewards guessing Rewards guessing

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION Avoid clues in the responses such as answers of unequal length or repetition of key words Avoid clues in the responses such as answers of unequal length or repetition of key words

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION Avoid compound answers, ie, answers that include multiple actions Avoid compound answers, ie, answers that include multiple actions

TYPE A QUESTION BAD EXAMPLE TYPE A QUESTION BAD EXAMPLE Which of the following is the most appropriate treatment for community- acquired pnemonia? Which of the following is the most appropriate treatment for community- acquired pnemonia? A. Clindamycin A. Clindamycin B. Vancomycin B. Vancomycin C. Valacyclovir C. Valacyclovir D. Administer oxygen, check blood gases, and administer levofloxacin D. Administer oxygen, check blood gases, and administer levofloxacin

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION Be wary of “except” type questions, especially when a “double negative” is present Be wary of “except” type questions, especially when a “double negative” is present

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION In certain instances, an “except” type question may be appropriate and may accurately assess the breadth of an examinee’s knowledge In certain instances, an “except” type question may be appropriate and may accurately assess the breadth of an examinee’s knowledge

TYPE A QUESTION GOOD EXAMPLE All of the following are risk factors for gestational diabetes except All of the following are risk factors for gestational diabetes except A. Obesity B. Positive family history C. Age < 25 y D. History of a macrosomic child

GUIDELINES FOR AN EFFECTIVE TYPE-A QUESTION Be certain that illustrative materials are clear and that they reproduce well Be certain that illustrative materials are clear and that they reproduce well Vary the correct response among the letters a - d Vary the correct response among the letters a - d

GUIDELINES FOR AN EFFECTIVE TYPE-B QUESTION Clearly identify the nature of the items in column A and column B Clearly identify the nature of the items in column A and column B Include at least 1 or 2 unused distractors Include at least 1 or 2 unused distractors

GUIDELINES FOR AN EFFECTIVE TYPE-B QUESTION Indicate if answers in one of the columns may be used only once, more than once, or not at all Indicate if answers in one of the columns may be used only once, more than once, or not at all Be certain the matching items are reasonably related to one another Be certain the matching items are reasonably related to one another

GUIDELINES FOR AN EFFECTIVE TYPE-B QUESTION Matching items are particularly useful for testing interpretation skills Matching items are particularly useful for testing interpretation skills X-ray X-ray ECG ECG Sonogram Sonogram Histology Histology

TYPE B QUESTION GOOD EXAMPLE COLUMN A Antibiotics COLUMN B Target Organisms 1. Clindamycin a. Bacteroides species 2. Penicillin b. Staphylococci 3. Gentamicin c. E. coli d. Group B streptococci

TYPE B QUESTION BAD EXAMPLE DRUG PRINCIPAL SIDE EFFECT 1. Nifedipine a. Marrow suppression 2. Ampicillin b. Decreased libido 3. Fluoxetine c. Rash d. Hypotension

CONCLUSIONS Use only type-A and type-B questions Use only type-A and type-B questions Minimize recall questions and maximize questions that test interpretive and problem-solving skills Minimize recall questions and maximize questions that test interpretive and problem-solving skills

CONCLUSIONS Be certain that each question has a single and specific objective Be certain that each question has a single and specific objective Be certain the questions can be answered in the time allotted Be certain the questions can be answered in the time allotted