1. Assessment Item Writing Workshop Ken Robbins FDN-5560 Classroom Assessment Click HERE to return to the Documentation HERE

2. Introduction  Classroom Assessment activity  The activity demonstrates how improperly designed assessments leads to improper student inferences

3. Textbook Tests  Frisbie, Miranda, and Baker ’s 1993 research study –Textbook publishers’ assessments  90% of assessments at knowledge level  Gateway 2005 research study –Teacher/textbook assessments  96% at lower three levels of taxonomy  68% at knowledge level

4. Bloom’s Taxonomy of the Cognitive Domain  Bloom’s taxonomy of 1956  Updated in 2001 by Anderson et al. –Original study defined six categories  Knowledge  Comprehension  Application  Analysis  Synthesis  Evaluation  NC uses Dimensions of Thinking –Adaptation of Marzano’s research

5. Bloom’s Taxonomy Key Words  Knowledge (facts, recall, recognition) –Define, list, select, match, who, what, when, where  Comprehension (translate, interpret) –Classify  Application (apply knowledge in new situation) –Solve, identify results

6. Bloom’s Taxonomy Key Words  Analysis (breaking down into parts) –Relationships, main ideas, relevance, conclusions  Synthesis (combine elements into something new) –Create, develop, formulate  Evaluation (evaluate by a set form of criteria) –Defend, criticize, judge

7. Assessment Guidelines  Define Learning Target for assessment –Knowledge or ability desired –Objective of instruction  Timeframe for assessment creation –When do you normally create test –Create assessment before instruction (Popham) –Create immediately after instruction (Kehoe) –Recommendation-Create first, review after

8. Multiple-Choice Item-Writing Guidelines  Content Concerns –1. Every item should reflect specific content and a single behavior –2. Base each item on important content, no trivial content –3. Use novel material to test for higher-learning –4. Keep content of each item independent from content of other items on test

9. Multiple-Choice Item-Writing Guidelines  Content Concerns –5. Avoid over specific or over generalized content –6. Avoid opinion based items –7. Avoid trick items –8. Keep vocabulary simple for student group

10. Multiple-Choice Item-Writing Guidelines  Formatting Concerns –9. Use the question, completion, and best answer versions of the conventional MC item –10. Format the item vertically instead of horizontally

11. Multiple-Choice Item-Writing Guidelines  Style Concerns –11. Edit and proof items –12. Use correct grammar, punctuation, capitalization, and spelling –13. Minimize the amount of reading in each item

12. Multiple-Choice Item-Writing Guidelines  Writing the stem –14. Ensure directions in stem are clear –15. Include the central idea in the stem –16. Avoid window dressing (excess verbiage) –17. Word the stem positively, avoid negatives such as NOT or EXCEPT if at all possible. If negatives must be used, BOLD/CAPITALIZE

13. Multiple-Choice Item-Writing Guidelines  Writing the choices –18. Develop effective choices (3 is adequate) –19. Ensure only one right answer –20. Vary the location of the right answer –21. Place choices in logical or numerical order –22. Keep choices independent

14. Multiple-Choice Item-Writing Guidelines  Writing the choices –23. Keep choices homogeneous in content and grammar –24. Keep the length of the choices about equal –25. None-of-the-above should be used carefully –26. Avoid All-of-the-above –27. Phrase choices positively, avoid negatives such as NOT

15. Multiple-Choice Item-Writing Guidelines  Writing the choices –28. Avoid giving clues to the right answer, such as  a. specific determiners including always, never, ect  b. choices identical to or resembling the stem  c. grammatical inconsistencies that cue correct choice  d. conspicuous correct choice  e. pairs of options that cue correct choice  f. absurd, ridiculous options

16. Multiple-Choice Item-Writing Guidelines  Writing the choices –29. Make all distractors plausible –30. Use typical errors of students to write distractors –31. Use humor if it is compatible with the teacher and the learning environment

17. Final Assessment Guidelines  To increase knowledge level of students, use upper levels of Bloom’s taxonomy –Analysis –Synthesis –Evaluation  Develop discriminating items –Items every misses or answers correctly does not discriminate between knowledge levels  Review items after each administration –Ensure items assess learning targets –Ensure items provide proper student inferences

18. Conclusion TTTTextbook assessments (NOT GOOD) BBBBloom’s taxonomy (all 6 areas are important, but strive for the top 3) IIIItem-writing guidelines (your guidebook for success in assessment)

19.  Measure your students’ learning against your learning targets, not the targets of some textbook publisher  Questions? (Click Back to return to the Documentation)

20. Reference List  Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: Longman.  Bloom, B. S. (1976). Human characteristics and school learning. New York: McGraw-Hill.  Bloom, B. S., & et al. (eds) (1956). Taxonomy of educational objectives. New York: David McKay Company.  Frisbie, D. A., Miranda, D. U., & Baker, K. K. (1993). An evaluation of elementary textbook tests as classroom assessment tools. Applied Measurement in Education, 6(1), 21-36.  Haladyna, T. M., & Downing, S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied measurement in education, 2(1), 37-50.  Haladyna, T. M., Downing, S. M. & Rodriquez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309-334.  Kehoe, J. (1995). Writing multiple-choice test items. Practical Assessment, Research & Evaluation, 4(9).  Kehoe, J. (1995). Basic item analysis for multiple-choice tests. Practical Assessment, Research & Evaluation, 4(10).  North Carolina Department of Public Instruction. (1999). Understanding North Carolina tests: Thinking skills level. Assessment brief, Vol 6, No 4.  Office Port. (n.d.). Bloom’s taxonomy. Retrieved February 28, 2005, from http://www.officeport.com/edu/blooms.htm http://www.officeport.com/edu/blooms.htm  Popham, W. J. (2005). Classroom assessment: What teachers need to know. 4th ed. Boston: Pearson Education.  University of Texas Learning Center. (n. d.). Bloom’s taxonomy’s model questions and key words. Retrieved February 28, 2005, from http://www.utexas.edu/student/utlc/handouts/1414.html http://www.utexas.edu/student/utlc/handouts/1414.html