Experts Respond to Your Questions 2006 Annual ConferenceAlexandria, Virginia Council on Licensure, Enforcement and Regulation Expect the Unexpected: Are We Clearly Prepared? Are You Ready?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Moderator/Coordinator Chuck Friedman Assistant Vice President ACT

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Panelist David Foster Chief Executive Officer Kryterion, Inc.

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Panelist Ron Rodgers President Employment Research Institute Director of Measurement Services Continental Testing Services

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Panelist Barbara Showers Director, Education and Examinations Wisconsin Department of Regulation and Licensing

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Speakers You... the audience!!!

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Developing and Implementing ADA Accommodations and Preparing for ADA Litigation Credentialing & Exam Issues Sessions Early Bird Sessions Thursday, September 14 Psychometrics of Testing with Small Numbers Friday, September 15

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Demystifying Testing: Guide to Managing Exam Programs Leveraging the Internet in Exam Development and Delivery When the Unexpected Happens: Logistical and Technical Challenges Credentialing & Exam Issues Sessions

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Physical Ability Testing and Practical Examinations: They Fought the Law and the Law Won Reporting Examination Results to Stakeholders: Issues and Best Practices Test Compromise and Security: Be Prepared Credentialing & Exam Issues Sessions

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Responding to YOUR Testing Questions

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Test Blueprint Job Analysis Reliability Scoring Validity Passing Score Item Writing Standard Administration Item Analysis Item Review

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Test Blueprint Job Analysis Reliability Scoring Validity Passing Score Item Writing Standard Administration Item Analysis Item Review

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia How often should my program conduct a job analysis? Job analysis

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Job analysis required to support test Link content outline to test references Survey representative sample of profession Support test specifications with job analysis results Document each question with test references identified and available to all candidates

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Job analysis should be updated when: References supporting test have changed (new editions, out of print, etc.) Emerging technology changes practice Changes in practice redefine critical knowledge & skills needed to protect public & practitioners

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Should my Board or program sponsor allow candidates to review test results including test questions?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Considerations: Credibility/defensibility of test process Content reviews/expert input Verification measures for scoring History of scoring errors

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Considerations: Test exposure/security issues Extended study time with test content Limit to questions wrong? Provide correct answers?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Considerations: Current law or legislative climate Law on books? Transparency of government Fairness Remedies – is retake enough?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia How do we increase security for both paper-and-pencil test delivery and computerized test delivery in the face of new technologies for cheating and test theft?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Be informed; know the threats and risks

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Question Do you know what this is?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Question Do you know what this is? Two- Way Radio

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Question What is the threat here?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Question What is the threat here? Web Camera to capture items

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Question What is the threat here? Display to show answers

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Be informed; know the threats and risks Detecting and monitoring the threats will reduce the risks: Examples include Better training of site and program staff Data forensics Watch the Web Protect your items, tests and test results with better access controls, authentication and identification, encryption, etc. Take decisive action when a breach occurs

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What is the relationship between the examination process and reciprocity between states? Between countries?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Between states: Uniform national exams aid reciprocity State-unique exams can create barriers Authority to regulate professions is given to state government, but reciprocity is gaining importance

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Between countries: Global delivery of exams is increasingly possible Evaluations of foreign credentials have been aided by supplemental exams – ECFMG, ECFVG CGFNS – will they remain necessary?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia When it is appropriate to translate or adopt and exam? See you soon

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Public Health & Safety Issues Are safety sources (MSDS, etc.) and product labels available to practitioners in the target language? Are comparable training materials and textbooks available in English & in target language? How are target language candidates trained? In what language? With comparable health/safety resources?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia If translation/adaptation is appropriate: Are practice standards compromised in any way by training as delivered in target language? Follow standards & guidelines for adapting test into new language Review CLEAR Resource Brief on translating or adapting tests

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia What new technologies can we expect in the next five years that will affect the testing process?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia New security technologies will enable us to prevent fraud and catch test thieves, cheaters and proxy test takers in the act The Internet combined with test administration technologies will make possible truly ubiquitous high-stakes testing where people work, learn and live Better measurement with innovations at the item and test levels (performance testing using simulations, for example) Faster, virtually immediate, test revision and worldwide distribution

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia How should my program deal with an error: for example an incorrect score report, setting of an incorrect passing score, incorrect scoring on an item?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Guiding Principles Accept responsibility for the problem and the resolution Apologize promptly and sincerely Remediate with candidate and with internal or external systems

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Steps to Take First, investigate who is impacted and how it happened Most important in licensing is pass/fail status and licensing errors Identify remedies – Rescore? Free retest? Other? Request back the license?

Presented at the 2006 CLEAR Annual Conference September Alexandria, Virginia Thank you and Enjoy Lunch!!!!!!!!!