1. Designing and evaluating good multiple choice items Jack B. Monpas-Huber, Ph.D. Director of Assessment & Student Information

2. Parts of a multiple-choice item Stem Responses What are the poles of a magnet called? a.Anode and cathode b.North and south c.Strong and weak d.Attract and repel

3. Strengths of multiple-choice items* Easy for students to take Easy to score Can quickly assess a broad range of content Distractors can have diagnostic (formative) value Efficient *Stiggins, R. J., Arter, J. A., Chappuis, J., & Chappuis, S. (2006). Classroom assessment for learning: Doing it right-using it well. Educational Testing Service.

4. Limitations of multiple-choice items Guessing can skew score “With three options, the probability of randomly guessing the correct answer is 0.33. However, the probability of getting two of these three-option test items right by random guessing alone is 0.11, three correct is 0.04, four correct is 0.01 and five correct is 0.004” (Downing, 2006, p. 291-2) Can be hard and time-consuming to craft plausible distractors Not as useful for measuring other kinds of proficiencies we value Downing, S. M. (2006). Selected-response item formats in test development. Pp. 287-301 in Handbook of Test Development, edited by Downing, S.M. and Haladyna, T. M. (Eds.). Lawrence Erlbaum. Stiggins, R. J., Arter, J. A., Chappuis, J., & Chappuis, S. (2006). Classroom assessment for learning: Doing it right-using it well. Educational Testing Service.

5. When to use multiple-choice items* To assess mastery of knowledge Recall of factual knowledge Some procedural knowledge To assess some reasoning skills i.e., compare/contrast, inference, cause and effect *Stiggins, R. J., Arter, J. A., Chappuis, J., & Chappuis, S. (2006). Classroom assessment for learning: Doing it right-using it well. Educational Testing Service.

6. Some general guidelines for multiple-choice items* State the whole question in the item stem Eliminate repetition of material in response options Be sure there is only one correct or best answer Keep response options brief and parallel Make all response options the same length Limit use of “all of the above” or “none of the above” Use “always” and “never” with caution *Stiggins, R. J., Arter, J. A., Chappuis, J., & Chappuis, S. (2006). Classroom assessment for learning: Doing it right-using it well. Educational Testing Service.

7. Multiple-choice items on the Reading MSP* *Office of the Superintendent of Public Instruction. (2010). Test and item specifications for grades 3-high school Washington state reading assessment. Retrieved on October 31, 2010 from http://k12.wa.us/Reading/Assessment/default.aspx.http://k12.wa.us/Reading/Assessment/default.aspx.

8. Multiple-choice items on the Math MSP* *Office of the Superintendent of Public Instruction. (2010). Measurements of student progress test and item specifications: Grades 6-8 mathematics. Retrieved on October 31, 2010 from http://k12.wa.us/Mathematics/TestItemSpec.aspxhttp://k12.wa.us/Mathematics/TestItemSpec.aspx

9. Let’s do one.... Kehoe, J. (1995). Writing multiple-choice test items. Practical Assessment, Research & Evaluation, 4(9). Retrieved October 30, 2010 from http://PAREonline.net/getvn.asp?v=4&n=9.http://PAREonline.net/getvn.asp?v=4&n=9 Stiggins, R. J., Arter, J. A., Chappuis, J., & Chappuis, S. (2006). Classroom assessment for learning: Doing it right-using it well. Educational Testing Service. 1. Start with a learning objective A single point or problem that can be stated as a proposition (a statement of the most important facts, concepts, or understanding we think every student should have at the end of instruction): Knowledge propositions (statements of learned fact) Manifest Destiny was first used as a justification for annexing Texas to the United States. Manifest Destiny represents a belief that was used to justify the taking of Native American lands. Reasoning propositions (states the result of reasoning applied to info at hand) Identify knowledge to be applied, apply the pattern of reasoning, and state result as a declarative sentence: Example: “Compare/Contrast” reasoning about “Reasons settlers went west” The Mormons went west to practice their religion without persecution, whereas the settlers in Texas went west because land was cheap or free and they wanted a place to start over Both Mormons and settlers in Texas were searching for a better life In order to assess reasoning, must provide with a context different than that in which the students practiced. Otherwise we are assessing memory rather than transfer.

10. Let’s do one.... Kehoe, J. (1995). Writing multiple-choice test items. Practical Assessment, Research & Evaluation, 4(9). Retrieved October 30, 2010 from http://PAREonline.net/getvn.asp?v=4&n=9.http://PAREonline.net/getvn.asp?v=4&n=9 2. Write a good stem Identify one point or one problem Try to make it an incomplete statement or a direct question, and avoid familiar phraseology Try not to use negatives Try to avoid giving clues to the correct answer Don’t offer superfluous information as an introduction to a question.

11. Let’s do one.... Downing, S. M. (2006). Selected-response item formats in test development. Pp. 287-301 in Handbook of Test Development, edited by Downing, S.M. and Haladyna, T. M. (Eds.). Lawrence Erlbaum. Kehoe, J. (1995). Writing multiple-choice test items. Practical Assessment, Research & Evaluation, 4(9). Retrieved October 30, 2010 from http://PAREonline.net/getvn.asp?v=4&n=9.http://PAREonline.net/getvn.asp?v=4&n=9 3. Write some response options Three good ones is sufficient. Based on a meta analysis of 80 years of research, “MC items should consist of three options, one correct option and two distractors. Using more options does little to improve item and test score statistics and typically results in implausible distractors” (Rodriguez, 2005, p.11, quoted in Downing, 2006, 292) The correct answer should be unequivocal to students who know the material and do well on the test overall Construct distractors that are similar in length, complexity, and grammatical form to the correct answer. Don’t use “all of the above.” Recognition of one wrong option eliminates “all of the above” and recognition of two right options identifies it as the answer.

12. Evaluating the quality of multiple-choice items: Judgment Content considerations What exactly is the purpose of this item? Does it accurately represent the content? Is it essential to measuring the learning target? Wording / Bias Is there anything in the wording (apart from the learning objective) that could affect how the students responds? How can we word the item more clearly and succinctly?

13. Evaluating the quality of multiple-choice items: Item analysis data Item difficulty (or easiness) Percent of students scoring item correctly (or “p-value”). What is your expectation for an item’s easiness? What if 90% of students score an item correctly? 10%? Depending on purpose, might want to include mostly items of moderate difficulty (40-60% correct), with just a few easy and difficult items included. Item discrimination Items should discriminate those who understand from those who don’t. Item- total correlation is one tool for seeing item discrimination. A high correlation would mean high achieving students (on total test score) are getting item correct, low achieving students are missing the item. Want items to correlate with total score at least.30. Definitely don’t want really low or negative correlation (means kids are mostly guessing). Possibly Scantron machine provides these or similar statistics; if not, they are easy enough to do in Excel.

14. Downing, S. M. (2006). Selected-response item formats in test development. Pp. 287-301 in Handbook of Test Development, edited by Downing, S.M. and Haladyna, T. M. (Eds.). Lawrence Erlbaum. Frary, R. B. (1995). More multiple-choice item writing do’s and don’ts. Practical Assessment, Research & Evaluation, 4(11). Retrieved October 30, 2010 from http://PAREonline.net/getvn.asp?v=4&n=11. http://PAREonline.net/getvn.asp?v=4&n=11 Kehoe, J. (1995). Writing multiple-choice test items. Practical Assessment, Research & Evaluation, 4(9). Retrieved October 30, 2010 from http://PAREonline.net/getvn.asp?v=4&n=9. http://PAREonline.net/getvn.asp?v=4&n=9 Office of the Superintendent of Public Instruction. (2010). Measurements of student progress test and item specifications: Grades 6-8 mathematics. Retrieved on October 31, 2010 from http://k12.wa.us/Mathematics/TestItemSpec.aspxhttp://k12.wa.us/Mathematics/TestItemSpec.aspx Office of the Superintendent of Public Instruction. (2010). Test and item specifications for grades 3-high school Washington state reading assessment. Retrieved on October 31, 2010 from http://k12.wa.us/Reading/Assessment/default.aspx.http://k12.wa.us/Reading/Assessment/default.aspx. Stiggins, R. J., Arter, J. A., Chappuis, J., & Chappuis, S. (2006). Classroom assessment for learning: Doing it right-using it well. Educational Testing Service. References