Effective Communication of Exam Results: What Should (or Shouldn't) be Included in the Candidate's Score Report Elizabeth A. Witt, Ph.D. American Board of Emergency Medicine 2006 Annual ConferenceAlexandria, Virginia Council on Licensure, Enforcement and Regulation Expect the Unexpected: Are We Clearly Prepared?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Validity Degree to which evidence supports the interpretation and proposed use of test scores

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Certified Chefs of America Examination Results Exam: South American Cuisine Date: August 15, 2006 Name: Jane Smith Result: Pass Your certificate will be mailed within 10 days.

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What information do examinees want? Pass/Fail status Actual score 78 Comparison to passing score Areas of strength/weakness Comparison with others

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What else do examinees want? Timely notification Explanations – graphs, technical terminology Graphics, visual presentation Simplicity

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Timely notification Why do we not just give them what they want? Instant notification possible with CBT BUT Not with paper exams AND Requires relinquishing control INCLUDING Elimination of poorly performing items

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Simplicity Why do we not just give them what they want? We lose sight of the purpose/audience. This is not a technical report!

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Explanations Explanations – graphs, technical terminology Why do we not just give them what they want? Not always necessary Familiarity – lose sight of the audience May require psychometric services

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Graphics, visual presentation Why do we not just give them what they want? Not always necessary Small staff, small program

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Why do we not just give them what they want? Actual score Actual score – may need explanation Raw score: number correct, percent correct Logits Scaled score Scores may not be precise away from the cut

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Why do we not just give them what they want? Comparison to passing score Comparison to passing score – Scores may not be precise except at the cut – May need explanation (SEM, for example) – May be self-explanatory (if a simple raw is reported, for example)

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Why do we not just give them what they want? Areas of strength/weakness - Need a subscore for each area - Scores based on few items are not reliable - Subscores may not sum to total test score - confusing to examinees

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Why do we not just give them what they want? Comparison with others - Scores are normally criterion-referenced. - The group testing may differ by exam administration.

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Recommendations DO include: Name, exam title, exam date Pass/Fail Status An interpretable score Score required to pass Same results by section, if appropriate Simple, clean graphics

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Recommendations DO include: Simple, clear explanations for any graphs or technical terminology Brief, simple explanation of scoring Diagnostic information for those who fail How to request score verification, appeal result

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Recommendations Do NOT include: Decimals Mean score = Mean score = 275 Excessive detail Jargon SEM Coefficient Alpha Logit Personal information (SSN, for example) Teeny, tiny font s iz e s

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Recommendations Candidate score reports should be: Timely Simple Clear Brief Easy to read and understand

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Bottom Line Who is the audience and what do they need to know?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia References/Resources American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association Ryan, J. M. (2006). Practices, issues, and trends in student test score reporting. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of Test Development (pp ). Mahwah, NJ: Lawrence Erlbaum. Schroeder, L. L. & Castle, R. Technology and testing: Reporting scores. (Winter 2006). CLEAR Exam Review, 27(1), Sample3.htm

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Speaker Contact Information , ext Elizabeth A. Witt, Ph.D. American Board of Emergency Medicine 3000 Coolidge Rd. East Lansing, MI 48823