1. Selected Response Tests
From Classroom Assessment: What Teachers Need to Know, (2004) Ch. 6, by W.J. Popham

2. What are selected response tests?

3. What are selected-response tests?
Binary choice items
Multiple binary choice items
Multiple choice items
Matching items
Make question-relay cards for each with definition on back

4. 5 General Item-Writing Commandments

5. 5 general commandments
Thou shalt not provide opaque directions regarding how to respond to your assessment .
Opaque in this context means:
A) not transmitting heat
B) impenetrable to light
C) hard to understand
D) stupid, unintelligent

6. 5 general commandments
Thou shalt not provide opaque directions regarding how to respond to your assessment .
Opaque in this context means:
A) not transmitting heat
B) impenetrable to light
C) hard to understand
D) stupid, unintelligent

7. 5 general commandments (cont.)
2. Thou shalt not employ ________________ statements in your assessment items.
Example:
Indicate if the following statement is true or false:
____Leaders of developing nations have tended to distrust leaders of developed nations due to their imperialistic tendencies.

8. 5 general commandments (cont.
2. Thou shalt not employ __ambiguous__ statements in your assessment items.

9. 5 general commandments (cont.)
3. Thou shalt not provide students with an __________________clue regarding the appropriate response. A. context B. deceptive C. unintentional (i.e., not purposeful) D. problematic

10. 5 general commandments (cont.)
3. Thou shalt not provide students with an __________________clue regarding the appropriate response. A. context B. deceptive C. unintentional (i.e., not purposeful) D. problematic

11. 5 general commandments (cont.)
4. Thou shalt not employ complex syntax in your assessment items.
Mark the following directions as complex (C) or simple (S) syntax, appropriate for a beginning level ELL:
___ After reading the article and identifying the cause and effect relationships, read the lists of causes and effects below, and draw a line between each main action and its resulting event.

12. 5 general commandments (cont.)
4. Thou shalt not employ complex syntax in your assessment items.
Mark the following directions as complex (C) or simple (S) syntax, appropriate for a beginning level ELL:
_C_ After reading the article and identifying the cause and effect relationships, read the lists of causes and effects below, and draw a line between each main action and its resulting event.

13. 5 general commandments (cont.)
5. Thou shalt not use vocabulary that is more advanced than required .
Which of the following is a concise paraphrase of the above statement.
a) Employ scintillating syntax.
b) Eschew obfuscating verbiage.
c) Imitate exhilarating phonology.

14. 5 general commandments (cont.)
5. Thou shalt not use vocabulary that is more advanced than required .
Which of the following is a concise paraphrase of the above statement.
a) Employ scintillating syntax.
b) Eschew obfuscating verbiage.
c) Imitate exhilarating phonology.

15. Binary Choice Items
Greatest strength - can cover large amount of content in a brief time
Greatest weakness – 50% chance of guessing correctly.
(A large number of items reduces impact of this probability)

16. Binary choice examples
(See handout) – Look over these true/false reading questions and identify how the binary choice guidelines have been violated.

17. Binary Choice Guidelines
Phrase items to elicit thoughtfulness
Minimize negative statements
Avoid double-concept items
Balance response categories
Maintain item length similarity
Not blatantly true or false, a superficial analysis might suggest a wrong answer, want students to think!
Causes difficulty for many students
Confusing
Approximately same number of t/f or whatever categories you’re using
Tendency is to throw in qualifying clauses on the true statements to make it clear it’s true – students catch on that longer answers are true

18. Multiple Binary-Choice items
A cluster of items is presented to the students , requiring a binary response to each item in the cluster. Usually the items are related to an initial statement or set of statements.

19. Multiple Binary Choice Items
Research (Frisbie ‘92) shows these are
Highly efficient for gathering data
More reliable than other selected-response items
Able to measure same skills/abilities
Slightly more difficult than multiple choice
Perceived by students are more difficult but more efficient

20. Multiple Binary-choice items
Other reasons to use them:
If you use stimulus material that is new to the students, the item can be more intellectually demanding
Students can respond to more M B-C than M-C items in the same amount of time – greater reliability

21. Multiple Binary-Choice Items
Read handout example – how could it be improved?

22. Multiple Binary-Choice Guidelines
Separate item clusters with obvious formatting
Make sure all items in a cluster are linked in a meaningful way to the stem (stimulus material)

23. Multiple Choice items Can measure knowledge OR ability to think
One advantage - answers can differ in relative correctness – students choose the best answer
A weakness – students need only recognize correct answer (not generate)
Stem can be a direction question or an incomplete statement – direction question best for younger students
4,5 alternative answers best
3 alternativs = greater chance of guessing correctly
> 5 = heavy reading load

24. Multiple choice examples
(See handout)

25. Multiple Choice guidelines
The stem should consist of a self-contained question or problem
Avoid negative statements
Attend to alternative length (no unintended clues!)
Randomly assign correct answers to alternative positions
Never use “all of the above”/use “none of the above” to increase difficulty IF it helps you make the test-inference you want

26. Matching items
Advantage: compact form/little space on page – easy to tap into lots of information efficiently
Disadvantage: sometimes encourages memorization of low-level factual information

27. Matching item examples
(See handout)

28. Matching item guidelines
Each list should be homogenous
Use brief lists; place shorter words/phrases on the right
Employ more responses than premises
Order the responses logically
Make explicit the basis for matching and the # of times a response can be used
Place all premises and responses on a single page

29. Conclusion
Guidelines will help, but there’s no substitute for lots of PRACTICE
Run your tests by a colleague
Variety in testing procedures = greater validity
Why? A well-learned cognitive skill can be displayed in many ways. We should be promoting generalizable mastery

30. Discuss Why are selected-response tests often criticized?
Someone makes this statement:
“Good teachers shouldn’t ever use selected-response type questions on a test. They don’t test higher order thinking skills.”
What would you say?