1. 1 Producing Your Assessment Question Mark “Software for creating and delivering assessments with powerful reports”  Copyright 2000 QuestionMark. All rights reserved.

2. 2 Objectives 210/18/2015 Compute your passing score Define your passing score 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. At the end of this presentation, you will understand how to: Beta test Alpha test

3. 3 Best practices for developing tests 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. 1. Analyze the job 2. Document the test blueprint 3. Survey the blueprint 4. Develop items 5. Alpha test and revise items 6. Beta test the exam 7. Select items and set the cut score 8. Go live with the test

4. 4  Costly to revise  Inexperienced test writers  New test content or program  New test design Why test your test? 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

5. 5 What does a stepped evaluation process give you?  Verify technical accuracy  Ensure implement-ability  Test content effectiveness  Gauge organizational or industry acceptance 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

6. 6 Stepped evaluations for alpha & beta testing 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. Self evaluation Expert review Small group review- alpha test Large group review – beta test

7. 7 Inside evaluation steps 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. Self Rough version ExpertSm.GroupLg.Group Polished version High interaction Low interaction Content & clarity Usability & performance

8. 8 What do you want to know? 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. Reaction ? Customer or trainee satisfaction Learning ?? Knowledge learned Skills developed Attitudes changed Results ?!??! Productivity, quality, profits increased Certify !!!!!!! Mastery achieved

9. 9 Types of measurements 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. Reaction ?“Smile” sheet Attitude survey Learning ??Self-assessment Skill/knowledge check Results ?!??!Training ROI Behavioral impact Certify !!!!!!!Certification exams Legal defensibility Confidentiality policies Low Stakes High Stakes

10. 10  Relative scores  Absolute scores  Compromised scores  Failure rates  Arbitrary, but not capricious Scoring terminology 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

11. 11  Contrasting groups method  Experts set score arbitrarily  Experts’ consensus opinion Ways to set passing scores 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

12. 12 Contrasting Group method  Expert performers  Poor performers  Consider the overlap  Pick a score in the zone of acceptance Low stakes score setting 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

13. 13 Contrasting group sample 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. ScoreHi GroupLow Group 24+xxxxxx 24x 23x 22x 21xx 20x 19xxx <+=18xxxxx Zone of Acceptance

14. 14  Use your judgment to assign points to test items  Estimate a percentage to pass  Variations on estimating a percentage  Compromise on a passing score  Some practical hints High stakes score setting 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

15. 15  Identify what it would take to “just” pass  Rate & categorize characteristics  Set required number of characteristics  Compute minimum points Assigning points 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

16. 16  Agree on “borderline” characteristics  Estimate the percentage of candidates  Compute average Estimating a percentage 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

17. 17  Supplying correct answers  Omitting result data  Working individually Common variations on estimating 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

18. 18  Both relative and absolute  Define acceptable standards  Compute curves  Compute cut score Compromise on your passing score 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

19. 19  Minimum failure rate  Maximum failure rate  Minimum passing point  Maximum passing point Acceptable standards for the compromise method 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

20. 20 50 40 30 20 10 0 <45 50 55 60 65 70 Compromise curve 10/18/2015  Copyright 2000 QuestionMark. All rights reserved. %Correct Score % Fail 20% acceptable failure rate Minimum fail score Maximum passing score

21. 21  Multiple judges  Provide correct answers to items  Provide result data  Collect relevant demographics  Avoid abstraction  Perform additional surveying Common themes & practical hints 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.

22. 22 Thank you for your attention Thank you for your time and attention. We hope you have enjoyed learning about how to write test blueprints and test items. We at Question Mark strive to improve our products and services. To let us know how you felt about this presentation, visit our North American home page and complete the Learning Center Survey: http://www.questionmark.com/home.htm#Surveys http://www.questionmark.com/home.htm#Surveys Question Mark provides numerous presentations to help our customers, and prospective customers, successfully deploy computerized assessments. We provide public “teleseminars” that use a combination of conference calls and web technology to discuss computerized assessment issues. You can register for a public seminar by going to: www.questionmark.com/seminars/index.asp www.questionmark.com/seminars/index.asp For other material that you might find useful, please visit our Learning Center at: www.questionmark.com/learningcenter/www.questionmark.com/learningcenter/ Thank you once again. This is the end of the presentation. 10/18/2015  Copyright 2000 QuestionMark. All rights reserved.