1. 1 Writing Test Blueprints and Test Items “Software for Creating and Delivering Assessments With Powerful Reports”

2. 2 Objectives 210/18/2015 Develop test items Survey the test blueprint 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. At the end of this presentation, you will understand how to: Document the test blueprint

3. 3 Best Practices for Developing Tests 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. 1. Analyze the job 2. Document the test blueprint 3. Survey the blueprint 4. Develop items 5. Alpha test and revise items 6. Beta test the exam 7. Select items and set the cut-score 8. Go live with the test

4. 4  Define objectives in behavioral terms Action verb Condition(s) Standard(s)  Use concrete verbiage  Meet key criteria Sample Work Process for Writing Test Objectives 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

5. 5  Major test objectives or topics (job tasks)  Subtopics (subtasks) within each major performance area  Rank test objectives and topics (tasks) in order of importance How Do You Document the Test Blueprint? 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

6. 6  Select contributors  Re-prioritize tasks (objectives)  Query time spent How Do You Survey the Test Blueprint? 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

7. 7 Question Writing Process  Start from your surveyed blueprint  Select a level of difficulty guideline  Write question stems  Write correct answers  Write distractors  Edit 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

8. 8 What Does a Level of Difficulty Guideline Give You?  Consistency  Set performance standards  Yardstick to gauge level of difficulty 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

9. 9 Level of Difficulty Guideline  Evaluation  Synthesis  Analysis  Application  Comprehension  Knowledge 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

10. 10 Knowledge and Comprehension Evaluation Synthesis Analysis Application  Comprehension - explain, predict, paraphrase  Knowledge - define, label, list 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

11. 11 Application and Analysis Evaluation Synthesis  Analysis – compare, contrast, relate  Application – apply, solve, compute Comprehension Knowledge 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

12. 12 Synthesis and Evaluation  Evaluation – appraise, evaluate, judge  Synthesis – formulate, construct, create Analysis Application Comprehension Knowledge 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

13. 13 Apply Appropriate Test Standards  Formal standards  Administrative policies  Item writing guidelines 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

14. 14 Question (Response) Types Perception offers eight question types:  Multiple choice  True/false  Multiple response  Numeric item  Word response  Selection item  Matrix item  Hot spot item 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

15. 15 Preferred Item Types  Easily understood  Minimize guessing  Avoid testing trivial information  Avoid candidate scrolling 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

16. 16 How to Write Stems  Present a single, definitive problem  Use a question format  Objective congruence  Appropriate level of difficulty  Active voice 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

17. 17 What to Avoid in Stems  Questions that build  Negatively phrased questions  “Always” and “Never”  “Corporate-centric” questions  Humor  Trick questions  “Trainer-centric” questions 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

18. 18 Guidelines to Write Options  Comparable in content, length, & logic  Non-repeating  Parallel language  Logically or numerically ordered  Avoid “All of the above” and “None of the above”  Plausible distractors 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

19. 19 Avoid Grammatical Clues 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. Option A is the only option that makes grammatical sense. The best way to increase the reliability of a test is to: A. Increase the test length. [Correct] B. Removing poor quality items. C. Test should be readable for all test takers.

20. 20 Avoid Association Clues 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. Option B includes the word “seven.” Who are the “Magnificent Seven?” A. A pro basketball team [Correct] B. A group of seven fictional, heroic, Western figures [Correct] C. A protest group made famous during the 1960s D. A vegetarian rock group

21. 21 What is the cost of an item that normally sells for 9.99 USD that is discounted 25%? Order Options Numerically 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. Wrong A. 5.00 B. 7.50 C. 2.50 D. 6.66 Right A. 2.50 B. 5.00 C. 6.66 D. 7.50

22. 22 What are the three important concerns in achievement testing? V=validity R=reliability E=efficiency O=objectivity Order Options Logically 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. Wrong A. V, R, E B. O, V, R C. R, O, E D. E, O, V Right A. V, R, O B. V, R, E C. V, E, O D. R, E, O

23. 23  Objective statement by item  Item naming conventions  Item difficulty or weight Evolving the Test Blueprint 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

24. 24 Thank you for your attention Thank you for your time and attention. We hope you have enjoyed learning about how to write test blueprints and test items. We at Question Mark strive to improve our products and services. To let us know how you felt about this presentation, visit our North American home page and complete the Learning Center Survey: http://www.questionmark.com/surveys.htm http://www.questionmark.com/surveys.htm Question Mark provides numerous presentations to help our customers, and prospective customers, successfully deploy computerized assessments. We provide public “web-seminars” that use a combination of conference calls and web technology to discuss computerized assessment issues. You can register for a public seminar by going to: www.questionmark.com/seminars/index.asp www.questionmark.com/seminars/index.asp For other material that you might find useful, please visit our Learning Center at: www.questionmark.com/learningcenter/www.questionmark.com/learningcenter/ 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.