1. Effective Communication of Exam Results: What Should (or Shouldn't) be Included in the Candidate's Score Report Elizabeth A. Witt, Ph.D. American Board of Emergency Medicine 2006 Annual ConferenceAlexandria, Virginia Council on Licensure, Enforcement and Regulation Expect the Unexpected: Are We Clearly Prepared?

2. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Validity Degree to which evidence supports the interpretation and proposed use of test scores

3. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Certified Chefs of America Examination Results Exam: South American Cuisine Date: August 15, 2006 Name: Jane Smith Result: Pass Your certificate will be mailed within 10 days.

4. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

5. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

6. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia What information do examinees want? Pass/Fail status Actual score 78 Comparison to passing score Areas of strength/weakness Comparison with others

7. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia What else do examinees want? Timely notification Explanations – graphs, technical terminology Graphics, visual presentation Simplicity

8. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Timely notification Why do we not just give them what they want? Instant notification possible with CBT BUT Not with paper exams AND Requires relinquishing control INCLUDING Elimination of poorly performing items

9. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Simplicity Why do we not just give them what they want? We lose sight of the purpose/audience. This is not a technical report!

10. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Explanations Explanations – graphs, technical terminology Why do we not just give them what they want? Not always necessary Familiarity – lose sight of the audience May require psychometric services

11. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Graphics, visual presentation Why do we not just give them what they want? Not always necessary Small staff, small program

12. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Why do we not just give them what they want? Actual score Actual score – may need explanation Raw score: number correct, percent correct Logits Scaled score Scores may not be precise away from the cut

13. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Why do we not just give them what they want? Comparison to passing score Comparison to passing score – Scores may not be precise except at the cut – May need explanation (SEM, for example) – May be self-explanatory (if a simple raw is reported, for example)

14. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Why do we not just give them what they want? Areas of strength/weakness - Need a subscore for each area - Scores based on few items are not reliable - Subscores may not sum to total test score - confusing to examinees

15. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Why do we not just give them what they want? Comparison with others - Scores are normally criterion-referenced. - The group testing may differ by exam administration.

16. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Recommendations DO include: Name, exam title, exam date Pass/Fail Status An interpretable score Score required to pass Same results by section, if appropriate Simple, clean graphics

17. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Recommendations DO include: Simple, clear explanations for any graphs or technical terminology Brief, simple explanation of scoring Diagnostic information for those who fail How to request score verification, appeal result

18. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Recommendations Do NOT include: Decimals Mean score = 275.358 Mean score = 275 Excessive detail Jargon SEM Coefficient Alpha Logit Personal information (SSN, for example) Teeny, tiny font s iz e s

19. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Recommendations Candidate score reports should be: Timely Simple Clear Brief Easy to read and understand

20. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia

21. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Bottom Line Who is the audience and what do they need to know?

22. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia References/Resources American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association Ryan, J. M. (2006). Practices, issues, and trends in student test score reporting. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of Test Development (pp. 677-710). Mahwah, NJ: Lawrence Erlbaum. Schroeder, L. L. & Castle, R. Technology and testing: Reporting scores. (Winter 2006). CLEAR Exam Review, 27(1), 8-10. http://www.ets.org/Media/Products/IDMS/pdf/51588_idmsRpt.pdf http://www.realestatesimulator.com/pages/sample_reports/ Sample3.htm http://www.visualcertexam.com/screenshots.html

23. Presented at the 2006 CLEAR Annual Conference September 14-16 Alexandria, Virginia Speaker Contact Information 517-332-4800, ext. 328 ewitt@ABEM.org www.ABEM.org Elizabeth A. Witt, Ph.D. American Board of Emergency Medicine 3000 Coolidge Rd. East Lansing, MI 48823