1. “Putting a Luster on a Good Testing Program” Barbara Showers, PhD Wisconsin Department of Regulation and Licensing Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

2. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Low-Tech Indicators of Potential Exam Problems Frequent Score Adjustments Pass Rate Changes “I Can’t Find it on Your Web Site”

3. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Frequent Score Adjustments Indicators Regular grade adjustments on practical or oral exams before release or on appeal Increasing numbers of multiple-keyed or omitted multiple choice questions

4. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Frequent Score Adjustments Practical/Oral Issues Are examiners unclear on the criteria? Are examiners given too much flexibility? Are the criteria themselves solid (one correct answer)?

5. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Frequent Score Adjustments Practical/Oral Solutions Clarify scoring criteria Reduce flexibility – checklists Develop a bank of pre-approved questions with correct answers Train new examiners/board members Pair new examiners with experienced ones Use multiple examiners

6. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Frequent Score Adjustments Multiple Choice Issues Questions are out of date Questions are ambiguous

7. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Frequent Score Adjustments Multiple Choice Solutions Multiple question sources and writers Committee review before use Update on a regular schedule Allow candidate comments after exam If sufficient volume, run item analysis statistics – ID problems Don’t keep the multi-keyed or omits in the test – reflects poorly on quality

8. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Pass Rate Changes Indicators Gradual pass rate increase over time, or Pass rates swing up or down in short time

9. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Pass Rate Changes Issues Reasonable Explanations for Pass Rate Change Exam content change Exam cut score change Candidate qualifications change

10. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Pass Rate Changes Issues If no reasonable explanation, consider Security breach Standard setting issues – are the forms equivalent in difficulty?

11. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Pass Rate Change Issues What is a “reasonable” pass rate? Example: WI Hearing Instrument Specialist Exam Audiologist Pass Rate = 70% Apprentice Pass Rate = 38%

12. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Pass Rate Changes Solutions Security solutions Multiple forms New content as often as possible Avoid repeating highly memorable scenarios

13. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Pass Rate Changes Solutions Unexplained Variations Equate forms if possible If statistical equating is not possible, use at least subject matter expert ratings of difficulty to create similar forms Define and document the passing performance standard and use the definition over time for consistency

14. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona “I can’t find it on your web site.” Put the luster on your testing program by polishing your web site

15. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Polishing the Web Site Is the candidate bulletin prominent? How deeply buried is the fee information?

16. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Polishing the Web Site How to apply What to expect in content and administration Study resources Links to other relevant sites

17. Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Speaker Contact Information Barbara Showers, PhD Wisconsin Department of Regulation and Licensing P.O. Box 8935 Madison, WI 53708 608-266-7703, fax 608-267-1809 barbara.showers@drl.state.wi.us http://drl.wi.gov