“Putting a Luster on a Good Testing Program” Barbara Showers, PhD Wisconsin Department of Regulation and Licensing Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Low-Tech Indicators of Potential Exam Problems Frequent Score Adjustments Pass Rate Changes “I Can’t Find it on Your Web Site”

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Frequent Score Adjustments Indicators Regular grade adjustments on practical or oral exams before release or on appeal Increasing numbers of multiple-keyed or omitted multiple choice questions

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Frequent Score Adjustments Practical/Oral Issues Are examiners unclear on the criteria? Are examiners given too much flexibility? Are the criteria themselves solid (one correct answer)?

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Frequent Score Adjustments Practical/Oral Solutions Clarify scoring criteria Reduce flexibility – checklists Develop a bank of pre-approved questions with correct answers Train new examiners/board members Pair new examiners with experienced ones Use multiple examiners

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Frequent Score Adjustments Multiple Choice Issues Questions are out of date Questions are ambiguous

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Frequent Score Adjustments Multiple Choice Solutions Multiple question sources and writers Committee review before use Update on a regular schedule Allow candidate comments after exam If sufficient volume, run item analysis statistics – ID problems Don’t keep the multi-keyed or omits in the test – reflects poorly on quality

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Pass Rate Changes Indicators Gradual pass rate increase over time, or Pass rates swing up or down in short time

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Pass Rate Changes Issues Reasonable Explanations for Pass Rate Change Exam content change Exam cut score change Candidate qualifications change

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Pass Rate Changes Issues If no reasonable explanation, consider Security breach Standard setting issues – are the forms equivalent in difficulty?

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Pass Rate Change Issues What is a “reasonable” pass rate? Example: WI Hearing Instrument Specialist Exam Audiologist Pass Rate = 70% Apprentice Pass Rate = 38%

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Pass Rate Changes Solutions Security solutions Multiple forms New content as often as possible Avoid repeating highly memorable scenarios

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Pass Rate Changes Solutions Unexplained Variations Equate forms if possible If statistical equating is not possible, use at least subject matter expert ratings of difficulty to create similar forms Define and document the passing performance standard and use the definition over time for consistency

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona “I can’t find it on your web site.” Put the luster on your testing program by polishing your web site

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Polishing the Web Site Is the candidate bulletin prominent? How deeply buried is the fee information?

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Polishing the Web Site How to apply What to expect in content and administration Study resources Links to other relevant sites

Presented at the 2005 CLEAR Annual Conference September Phoenix, Arizona Speaker Contact Information Barbara Showers, PhD Wisconsin Department of Regulation and Licensing P.O. Box 8935 Madison, WI , fax