1. Recovering from the Unexpected
Presented at
The 22nd Annual CLEAR Conference
September 12 – 14, 2002
Moderator
I. Leon Smith, PhD
President & CEO
PES
Panel
Ronald C. Rodgers, PhD
Consulting Psychometrician
Continental Testing Services
President, Employment Research Institute
Randolph P. Reaves, JD
Executive Officer and General Counsel
Association of State and Provincial Psychology Boards
Elizabeth A. Witt, PhD
Senior Measurement Analyst
Promissor (formally CAT*ASI)

2. Scenario Summary
The “unexpected” arrived in the form of an to the licensure program sponsor to “look at the website of a test preparation company.” A review of the site revealed a number of pages containing the full text of items presumably recalled from the examination by some candidates. A check of the text of the items revealed that approximately 5% of the items were identical to those on the four current forms of the examination being delivered through the computer delivery vendor. Most of the exposed items also appeared on the four new forms already at the computer delivery vendor to be deployed shortly as replacements for the four current forms. Each form contains operational (scored) items, and several pre-test (unscored) blocks. All forms are pre-equated. To ensure the timely implementation of the four new forms, any replacement of items must be limited to those items already on the compute delivery vendor’s system. Since the test preparation website has been operational for several years, the magnitude of exposure probably went beyond these few items. In addition, approximately 70% of the items on the four new forms overlap with the current forms being administered.

3. Objective
The licensure program sponsor wants to accomplish the following goals:
Ensure the continuing validity of examination scores and the public protection function of licensure examination forms
Close down the test preparation website that is posting the exposed items
Have the Internet host provider terminate the service to the test preparation website
What to do? And what are the issues?