Selected Response Tests From Classroom Assessment: What Teachers Need to Know, (2004) Ch. 6, by W.J. Popham

What are selected response tests?

What are selected-response tests? Binary choice items Multiple binary choice items Multiple choice items Matching items Make question-relay cards for each with definition on back

5 General Item-Writing Commandments

5 general commandments Thou shalt not provide opaque directions regarding how to respond to your assessment . Opaque in this context means: A) not transmitting heat B) impenetrable to light C) hard to understand D) stupid, unintelligent

5 general commandments Thou shalt not provide opaque directions regarding how to respond to your assessment . Opaque in this context means: A) not transmitting heat B) impenetrable to light C) hard to understand D) stupid, unintelligent

5 general commandments (cont.) 2. Thou shalt not employ ________________ statements in your assessment items. Example: Indicate if the following statement is true or false: ____Leaders of developing nations have tended to distrust leaders of developed nations due to their imperialistic tendencies.

5 general commandments (cont. 2. Thou shalt not employ __ambiguous__ statements in your assessment items.

5 general commandments (cont.) 3. Thou shalt not provide students with an __________________clue regarding the appropriate response. A. context B. deceptive C. unintentional (i.e., not purposeful) D. problematic

5 general commandments (cont.) 3. Thou shalt not provide students with an __________________clue regarding the appropriate response. A. context B. deceptive C. unintentional (i.e., not purposeful) D. problematic

5 general commandments (cont.) 4. Thou shalt not employ complex syntax in your assessment items. Mark the following directions as complex (C) or simple (S) syntax, appropriate for a beginning level ELL: ___ After reading the article and identifying the cause and effect relationships, read the lists of causes and effects below, and draw a line between each main action and its resulting event.

5 general commandments (cont.) 4. Thou shalt not employ complex syntax in your assessment items. Mark the following directions as complex (C) or simple (S) syntax, appropriate for a beginning level ELL: _C_ After reading the article and identifying the cause and effect relationships, read the lists of causes and effects below, and draw a line between each main action and its resulting event.

5 general commandments (cont.) 5. Thou shalt not use vocabulary that is more advanced than required . Which of the following is a concise paraphrase of the above statement. a) Employ scintillating syntax. b) Eschew obfuscating verbiage. c) Imitate exhilarating phonology.

5 general commandments (cont.) 5. Thou shalt not use vocabulary that is more advanced than required . Which of the following is a concise paraphrase of the above statement. a) Employ scintillating syntax. b) Eschew obfuscating verbiage. c) Imitate exhilarating phonology.

Binary Choice Items Greatest strength - can cover large amount of content in a brief time Greatest weakness – 50% chance of guessing correctly. (A large number of items reduces impact of this probability)

Binary choice examples (See handout) – Look over these true/false reading questions and identify how the binary choice guidelines have been violated.

Binary Choice Guidelines Phrase items to elicit thoughtfulness Minimize negative statements Avoid double-concept items Balance response categories Maintain item length similarity Not blatantly true or false, a superficial analysis might suggest a wrong answer, want students to think! Causes difficulty for many students Confusing Approximately same number of t/f or whatever categories you’re using Tendency is to throw in qualifying clauses on the true statements to make it clear it’s true – students catch on that longer answers are true

Multiple Binary-Choice items A cluster of items is presented to the students , requiring a binary response to each item in the cluster. Usually the items are related to an initial statement or set of statements.

Multiple Binary Choice Items Research (Frisbie ‘92) shows these are Highly efficient for gathering data More reliable than other selected-response items Able to measure same skills/abilities Slightly more difficult than multiple choice Perceived by students are more difficult but more efficient

Multiple Binary-choice items Other reasons to use them: If you use stimulus material that is new to the students, the item can be more intellectually demanding Students can respond to more M B-C than M-C items in the same amount of time – greater reliability

Multiple Binary-Choice Items Read handout example – how could it be improved?

Multiple Binary-Choice Guidelines Separate item clusters with obvious formatting Make sure all items in a cluster are linked in a meaningful way to the stem (stimulus material)

Multiple Choice items Can measure knowledge OR ability to think One advantage - answers can differ in relative correctness – students choose the best answer A weakness – students need only recognize correct answer (not generate) Stem can be a direction question or an incomplete statement – direction question best for younger students 4,5 alternative answers best 3 alternativs = greater chance of guessing correctly > 5 = heavy reading load

Multiple choice examples (See handout)

Multiple Choice guidelines The stem should consist of a self-contained question or problem Avoid negative statements Attend to alternative length (no unintended clues!) Randomly assign correct answers to alternative positions Never use “all of the above”/use “none of the above” to increase difficulty IF it helps you make the test-inference you want

Matching items Advantage: compact form/little space on page – easy to tap into lots of information efficiently Disadvantage: sometimes encourages memorization of low-level factual information

Matching item examples (See handout)

Matching item guidelines Each list should be homogenous Use brief lists; place shorter words/phrases on the right Employ more responses than premises Order the responses logically Make explicit the basis for matching and the # of times a response can be used Place all premises and responses on a single page

Conclusion Guidelines will help, but there’s no substitute for lots of PRACTICE Run your tests by a colleague Variety in testing procedures = greater validity Why? A well-learned cognitive skill can be displayed in many ways. We should be promoting generalizable mastery

Discuss Why are selected-response tests often criticized? Someone makes this statement: “Good teachers shouldn’t ever use selected-response type questions on a test. They don’t test higher order thinking skills.” What would you say?