Multiple Choice Test A multiple choice test is a form of assessment in which examinees are instructed to choose one or more from several options from a list. Multiple choice tests are widely used, especially in higher education, because they are seen as being objective, and they are popular even with students (see Furnham et al., 2008). © POSbase 2008Contributor

Formats of MC Questions There are basically two formats for MC questions (1):  True-false: Please state for each statement whether it is true or false: [T] [F]Sydney is the capital of Australia [T] [F]Sydney is the biggest city of Australia [T] [F]Auckland is the capital of New Zealand [T] [F]Auckland is the biggest city of New Zealand © POSbase 2008

Formats of MC Questions There are basically two formats for MC questions (2):  One best answer: Which of the following statements about Australia is true? [ ]Sydney is the capital of Australia [ ]Sydney is the biggest city of Australia [ ]Melbourne is the capital of Australia [ ]Melbourne is the biggest city of Australia © POSbase 2008

Formats of MC Questions One best answers format seems to be preferable:  True/false items are more often unclear. Indeed, items sometimes have to rescored.  Examiners often wish to test the understanding of procedures or explanations. However, true/false items often allow assessment of isolated facts only. As a result, the NBME (National Board of Medical Examiners) has completely stopped using true/false formats on its examinations. © POSbase 2008

Formats of MC Questions Research has shown that two formats complicate questions and hamper test reliability:  “Each of the following is correct EXCEPT”  “Which of the following statements is NOT correct?” These formats should be avoided © POSbase 2008

Formats of MC Questions Other format issues:  To prevent cheating, some universities used different paper colors  performance best on white paper (Skinner, 2004).  Performance on MC tests with increasing difficulty is superior to MC tests with decreasing difficulty. © POSbase 2008

Guessing In theory, the probability of guessing in one- best-answer options is 1 divided by number of options:  Five options: 20%  Four options: 25% However, this is certainly an underestimation. Two factors contribute to an increase in this probability:  Exclusion of options as being accurate  Testwiseness © POSbase 2008

Exclusion of Distracters All of the Above Option: Who discovered classical conditioning? [A]Sigmund Freud [B]B.F. Skinner [C ]Ivan Pavlov [D]James D. Watson [E]Mickey Mouse Even the most knowledgeable student can exclude option E, and the student who knows little, option A. © POSbase 2008

Testwiseness All of the Above Option: Which of the following statements is true? [ ]Sydney is the capital of Australia [ ]Sydney is the biggest city of Australia [ ]Auckland is the capital of New Zealand [ ]Wellington is the biggest city of New Zealand [ ]All of the above If the examinee knows that one of the options is wrong  can conclude that “All of the Above” is wrong © POSbase 2008

Testwiseness All of the Above Option (1): Which of the following statements is true? [ ]Canberra is the capital of Australia [ ]Sydney is the biggest city of Australia [ ]Wellington is the capital of New Zealand [ ]Auckland is the biggest city of New Zealand [ ]All of the above If the examinee knows that two of the options are right  can conclude that “All of the Above” is right © POSbase 2008

Testwiseness All of the Above Option (2): Which of the following statements is true? A.[ ]Sydney is the capital of Australia B.[ ]Sydney is the biggest city of Australia C.[ ]Auckland is the capital of New Zealand D.[ ]A and B E.[ ]None of the above If the examinee knows that one of the options is right  can conclude that “None of the Above” is wrong © POSbase 2008

Testwiseness In the following question, options A, B, and E contain terms that are less absolute than C and D  testwise students exclude C and D In patients with advanced dementia, Alzheimer’s type, the memory defect A. can be treated adequately with phosphatidylcholine (lecithin) B. could be a sequela of early parkinsonism C. is never seen in patients with neurofibrillary tangles at autopsy D. is never severe E. possibly involves the cholinergic system © POSbase 2008

Testwiseness Exhaustive categories may give a hint that one of those answers if correct: Usually, performance in free recall tests is … A. better than performance in recognition tests B. the same as performance in recognition tests C. worse than performance in recognition tests D. independent of levels-of-processing © POSbase 2008

Reliability of Tests How many items are necessary to accurately assess a topic? A usual number given is 100 MC items to cover a topic. For problems (in medicine), a number of 12 is given. In educational practice, often fewer questions are given, which hampers reliability of the examinations. © POSbase 2008

Conclusion The presented findings suggest the following amendments (take this as starting point for discussion):  Assess performance repeatedly (different coursework, but with repeated feedback?)  100 MC questions and 12 essay questions  Higher threshold for pass (underestimation of probability)  Not only MC tests  Computerize tests in order to randomize some otherwise systematic factors like option order  Quality control of tests: Do they obey quality criteria? © POSbase 2008