Writing Selection Items EDU 385 Session 8 Writing Selection Items Multiple Choice

Bell Work How to not to prepare a Selection Item Test

Content Objectives Understand the characteristics of multiple- choice items Understand the strengths and limitations of multiple-choice items

Language Objectives Distinguish between well-stated and poorly stated multiple-choice items Identify and correct faults in poorly state multiple-choice Match multiple-choice items to intended learning outcomes Construct multiple-choice items that are well stated, relevant to important learning outcomes, and free of defects

Vocabulary Stem of the Item Distractor Items

Multiple-Choice Items Strengths 1. Learning outcomes from simple to complex can be measured 2. Highly structured and clear tasks are provided 3. A broad sample of achievement can be measured 4. Incorrect alternatives provide diagnostic information 5. Scores are less influenced by guessing than true-false items 6. Scoring is easy, objective, and reliable Limitations 1. Constructing good items is time consuming 2. It is frequently difficult to find plausible distractors 3. This item in ineffective for measuring some types of problem solving and the ability to organize and express items 4. Score can be influenced by reading ability

Nature of Multiple-Choice Items Consists of A Stem, which represents a problem situation Several Alternatives (Options or Choices), which provide possible solutions to the problem The Stem may be a question or incomplete statement Alternatives include the correct answer and several plausible wrong answers called Distractors (for students uncertain of correct answer)

Example (Question Form & Incomplete Sentence Form) Which one of the following item types is an example of a supply-type test item? A. multiple-choice item B. true-false item C. matching item *D. short answer item 2. An example of a supply-type item is the {Though stated differently both stems pose the same problem.} NOTE: The incomplete sentence is more concise

Best Answer Multiple Choice Best answer form all answers are partially correct but one is clearly the best answer Example Which item is best for measuring computational skill A. Multiple-choice item B. True-false item C. Matching item *D. Short answer item

Rules for Writing Multiple-Choice Items Effective multiple-choice questions present students with tasks that are important and clearly understood and can be answered by anyone who has achieved the intended learning outcome Nothing in the content should prevent the informed student from responding correctly Nothing in the content should give clues to enable the uninformed student to select the correct answer ˜

1. Design each item to measure an important learning outcome The problem situation should be important and related to the intended learning outcome to be measured Avoid obscure, irrelevant, and unimportant material Remember the purpose is help determine the extent to which the intended learner outcomes have been achieved

Rules for Writing Multiple-Choice Items 2. Present a single clearly formulated problem in the STEM of the item Example Poor” A table of specifications A. indicates how a test will be used to improve learning *B. Provides more balanced sampling C. Arranges the instructional objectives in order of their importance D. Specifies the method of scoring to be used on a test Better What is the main advantage of using a table of specification when preparing for an achievement test A. It reduces the amount of time required *B. It improves the sampling of content C. It make the construction of test items easier D. It increases the objectivity of the test

Rules for Writing Multiple-Choice Items 3. State the stem of the item in simple, clear language Example Poor: The paucity of plausible, but incorrect, statements that can be relate to a central idea poses a problem when constructing which one of the following items? A. Short answer B. True-false *C. Multiple choice D. Essay Better The lack of plausible, but incorrect, alternatives will cause the greatest difficulty when constructing A. short-answer items B. true-false items *C. multiple-choice items D. essay items

Rules for Writing Multiple-Choice Items 4. Put as much of the wording as possible in the stem of the item Example Poor In objective testing, the term objective A. refers to the method of identifying the learning outcomes B. refers to the method of selecting the test content C. refers to the method of presenting the problem *D. refers to the method of scoring answers Better In objective testing, the term objective refers to the method of A. identifying the learning outcomes B. selecting the test content C. presenting the problem *D. scoring the answers

Some times it requires more than moving common words to the STEM Poor Example Instructional objectives are more apt to be useful for test-construction purposes when they are stated in such a way that they show A. the course content to be covered during the instructional period *B. the kinds of performance students should demonstrate reaching the goal C. the things the teacher will do to obtain maximum student learning D. the types of activities to be participated in during the course Good Example Instructional objectives are most useful for test-construction when they are stated in terms of A. course content *B. student performance C. teacher behavior D. learning activities

Rules for Writing Multiple-Choice Items 5. State the stem of the item in positive form, whenever possible *A positively stated item tends to measure more important learning outcomes than negatively stated items *Knowing the most relevant argument has greater education value than knowing least relevant arguments. *Positively stated are harder to write because the item needs more distractors, than negatively stated items

Rules for Writing Multiple-Choice Items 5. State the stem of the item in positive form, whenever possible Example Item one: Which one of the following is a category in the revised taxonomy of cognitive domain? *A. Understand B. (distractor needed) C. (distractor needed) D. (distractor needed) Item two: Which one of the following is not a category in the revised taxonomy of cognitive domain? A. Understand B. Apply C. Analyze *D. (answer needed)

When it is difficult to find enough distractors, a more desirable procedure is a modified true-false form Example Directions: Indicate which of the following are categories in the taxonomy of the cognitive domain by circling Y for yes and N for no *Y N Understand Y N* Critical Thinking Y N* Reasoning *Y N Create

Rules for Writing Multiple-Choice Items 6. Emphasize NEGATIVE wording whenever it is used in the stem of an item Example Poor: Which one of the following is not desirable practice when preparing for multiple-choice items? A. Stating the stem in positive form B. Using the stem that could function as a short-answer item C. Underlining certain words in the stem for emphasis *D. Shortening the stem by lengthening the alternatives Better: All of the following are desirable practices when preparing multiple-choice items EXCEPT

7. Make certain the intended answer is correct or clearly the best Example Poor: What is the best method of selecting course content for test items? Better: Which one of the following is the best method of selecting course content for test items? Example of proper phrasing of the stem What is the purpose of classroom testing? One purpose of classroom testing is (or) The main purpose of classroom testing is

8. Make all alternatives grammatically consistent with the stem of the item and parallel in form Example Poor The recall of factual information can be measured best with a A. matching item B. multiple-choice item *C. short-answer item D. essay question Better The recall of factual information can be measured best with A. matching items B. multiple-choice items *C. short-answer items D. essay questions The indefinite article “a” in the first version makes the last distractor obviously wrong (correct form would be “an” essay

a) similarity of wording in both the stem and the correct answer 9. Avoid verbal clues that might enable students to select the correct answer or to eliminate an incorrect alternative a) similarity of wording in both the stem and the correct answer Which one of the following would you consult first to locate research articles on achievement testing A. Journal of Educational Psychology B. Journal of Educational Measurement C. Journal of Consulting Psychology *D. Review of Educational Research b) Stating the correct answer in textbook language or stereotyped phraseology Learning outcomes are most useful in preparing test when the are *A. clearly stated in performance terms B. developed cooperatively by teachers and students C. prepared after the instruction has ended D. stated in general terms

c) Stating the correct answer in greater detail Lack of attention to learning outcomes during test preparation A. will lower the technical quality of the items B. will make the construction of test items more difficult C. will result in greater use of essay questions. *D. may result in a test that is less relevant to the instructional program d) Including absolute terms in the distracter Achievement tests help students improve their learning by A. encouraging them all to study hard *B. informing them of their progress C. giving them all a feeling of success D. preventing any of them from neglecting their assignments

e) Including two response that are all inclusive Which of the following types of test items measures learning outcomes at the recall level? *A. Supply-type items B. Selection-type items D. Matching items E. Multiple-choice items f) Including two responses that have the same meaning Which one of the following is the most important characteristic of achievement-test results? A. Consistency B. Reliability *C. Relevance D. Objectivity

10. Make the distracters plausible and attractive to the uninformed Ways to increase plausibility and attractiveness of distracters a. Use the common misconceptions of errors of students as distracters b. State the alternative in the language of the students c. Use “good-sounding” words (“accurate,” “important,”) in the distracters d. Make the distractors similar to the correct answer in both length and complexity of wording e. Use extraneous clues in the distracters, such as stereotyped phrasing, scientific-sounding answers, and verbal associations with the stem of the item. But don’t overuse these clues to the point where the become ineffective f. Make the alternatives homogeneous, but in doing so beware of fine discriminations that are educationally insignificant

11. Vary the relative length of the correct answer to eliminate length as a clue There is a tendency for the correct answer to be longer than alternatives because of the need to qualify statements to make them unequivocally correct The relative length of the correct answer can be removed as a clue by varying it in such a manner that no apparent pattern is provided

12. Avoid using the alternative “all of the above,” and use “none of the above” with extreme caution *Lessens the effectiveness of the test *On “all the above” questions - even a student with a partial knowledge of the topic knows one of the responses is false they know the correct answer *Same with “none of the above” - if they know one response is true then know the correct answer *They also have a 50 percent chance of guessing the right answer

13. Vary the position of the correct answer in a random manner Avoid following a pattern Randomly select where to put the correct answer Page # strategy - open any page in a book look at the last digit on the right hand page (since the right page alway ends in an “odd number” A. = 3 B. = 5 C. = 7 D. = 9

To increase the difficulty of a test item 14. Control the difficulty of the item by varying the problem in the stem or changing the alternatives To increase the difficulty of a test item - Increase the level of knowledge called for to by making the item more complex - Make the alternatives more homogeneous

15. Make certain each item is independent of the other items in the test - Occasionally information given in the stem of another question help students answer another one - Avoid by carefully reviewing all the question before administering the test - Avoid interlocking questions where the answer to a question relies on correctly knowing the answer to the previous question

16. Use an efficient item format - Alternatives (distracters) listed below each other - Place answers on the left side of test (makes scoring easier - Use letters not numbers on items (less confusing for he students

17. Follow the normal rules of grammar

18. Break (or bend) any of these rules if it will improve the effectiveness of the item The previous rules are set forth to help design multiple- choice tests As teachers gain more experience they may want to vary ignore or modify some of these suggestions

Checklist for Evaluating Multiple-Choice Items 1. Is the type of item appropriate for measuring the intended learner outcome? 2. Does the item task match the learning task to be measured? 3. Does the stem of the item present a single, clearly formulated problem? 4. Is the stem stated in simple, clear language? 5. Is the stem worded so that there is no repetition of material in the alternative? 6. Is the stem stated in positive form whenever possible? 7. If negative wording is used in the stem, is it emphasized (by underlining or caps? 8. Is the intended answer correct or clearly best?

Checklist for Evaluating Multiple-Choice Items 9. Are all alternatives grammatically consistent with the stem and parallel in form? 10. Are the alternatives free from verbal clues to the correct answer? 11. Are the distracters plausible and attractive to the uniformed? 12. To eliminate length as a clue, is the relative length of the correct answer varied? 13. Has the alternative “all the above” been avoided and “none of the above” used only when appropriate? 14. Is the position of the correct answer varied so that there is no detectable pattern? 15. Does the item format and grammar usage provide for efficient test taking?

Summary 1. The multiple-choice item is the most highly regarded and useful selection-type item 2. The multiple-choice item consists of a stem and a set of alternative answers (Options or Choices) 3. The multiple-choice item can be designed to measure various intended learning outcomes, ranging from simple to complex 4. Items designed to measure achievement beyond the knowledge level must contain some novelty

Summary 5. The stem of a multiple-choice item should present a single, clearly formulated problem that is related to an important learning outcome 6. The intended answer should be correct or clearly best, as agreed upon by authorities 7. The distractors (incorrect alternatives) should be plausible enough to lead the uninformed away from the correct answer 8. The items should be written in simple, clear language that is free of nonfunctioning content

Summary 9. The items should be free from irrelevant sources of difficulty (e.g., ambiguity) that might prevent an informed examinee from answering correctly 10. The items should be free of irrelevant clues (e.g., verbal associations) that might enable an uniformed examinee to answer correctly 11. The item format should provide for efficient responding and follow the normal rules of grammar 12. The item format should provide a framework for preparing effective multiple-choice items, but experience in item writing may result in modifications to fit particular situations

Next Class Writing Selection Type Items True-False Matching Interpretive Exercise

Now Go Forth and Do Good Things