You’ve Been Shown: Now It’s Your Turn to Ask! CLEAR 2004Kansas City, Missouri Responding to Your Questions About Testing

Credentialing and Exam Issues: Sessions at CLEAR 2004 Ten Year Review of CBT When Certification Replaces Licensure: Show me Accreditation ADA: Let Us Show You What Works Show Me What Globalization Means for Credentialing Programs Multiple Tracks to Licensure: Show me the Way Test Piracy and Security in a Wired World CLEAR 2004Kansas City, Missouri

Show Me the Validity Show Me How to Get Past MCQs: New Opportunities in Measurement Show me an Evidential Approach to Assessment Design The Case of the Repeating Licensure Examination Candidate: Psychometric, Legal, and Policy Issues CLEAR 2004Kansas City, Missouri Credentialing and Exam Issues: Sessions at CLEAR 2004

Panelists: Carol Morrison Assistant Vice President National Board of Medical Examiners Elizabeth Witt Senior Statistician American Board of Emergency Medicine Rose McCallin Director of Office of Examination Services State of Colorado CLEAR 2004Kansas City, Missouri Moderator: Chuck Friedman Assistant Vice President, ACT

Speakers You... the audience!!! CLEAR 2004Kansas City, Missouri

Validity Reliability Item Analysis Item Review Item Writing Passing ScoreScoring Job Analysis Test Blueprint Standard Administration

Validity Reliability Item Analysis Item Writing Passing Score Scoring Test Blueprint Standard Administration Job Analysis Item Review

Should state licensing boards require continuing education? CLEAR 2004Kansas City, Missouri

How do I find out if my test has been compromised? If so, what should I do? CLEAR 2004Kansas City, Missouri

CLEAR 2004Kansas City, Missouri What should I do if I do or not do if I think a candidate or candidates have cheated?

Test Compromise? Missing test book –When, who? –Statistical analyses Memorization of items –Memorize and reconstruct items –Study guides –Internet postings –Statistical analyses

Is a self assessment exam good enough for continuing competency? CLEAR 2004Kansas City, Missouri

Four Components of Continuous Certification Professional Standing Professional Standing Lifelong Learning and Self Assessment Lifelong Learning and Self Assessment Assessment of Cognitive Expertise Assessment of Cognitive Expertise Assessment of Practice Performance Assessment of Practice Performance

Considerations in Determining the Value of a Self-Assessment How is the exam developed and scored? Can you be certain who took the exam? How high are the stakes for maintaining the credential? Does the self-assessment process foster learning? Is it easy to participate as part of one’s normal professional activities?

Is performance testing useful in state licensure? CLEAR 2004Kansas City, Missouri

Performance Assessment Does performance assessment measure something different? Reliability, validity How do you score it? Cost, complexity Answer: It depends!

CLEAR 2004Kansas City, Missouri What is your opinion about open book exams?

What should I do if a candidate receives an incorrect score report? CLEAR 2004Kansas City, Missouri

How do you determine if a request for accommodations is appropriate? If a test is administered under special conditions, should the score report be “flagged”? CLEAR 2004Kansas City, Missouri

Is Request Appropriate? Use expert consultant Level the playing field, don’t tilt it too far the other way

To Flag or Not to Flag? Many testing companies have stopped flagging Some claim that flagging may lead to unintended consequences Standards for Educational and Psychological Testing (APA/AERA/NCME) –Flag if do not have evidence of score comparability

Do we have to equate our tests? Is there one best way? ONE BEST WAY? CLEAR 2004Kansas City, Missouri

Reasons to Consider NOT Equating Small sample size Multidimensional exam Changing content outline

What If We Can’t Equate? Create a detailed blueprint Write the best possible items Build forms that are closely parallel in both content and statistics Conduct frequent standard setting studies Include common items across forms; study differences qualitatively

CLEAR 2004Kansas City, Missouri An examinee complains that he failed the test by one point/question. What do you tell him?

Other Sources of Information CLEAR Exam Review Other CLEAR Publications –ADA Handbook –Development, Administration, Scoring and Reporting of Credentialing Exams CLEAR web site: Frequently Asked Questions Joint Standards: AERA, NCME, APA Professional psychometricians CLEAR 2004Kansas City, Missouri