CLEAR 2008 Annual Conference Anchorage, Alaska “Problems and Priorities in Pretesting Pondered” Beth Noeller Grady Barnhill Carol O’Byrne

CLEAR 2008 Annual Conference Anchorage, Alaska Pretesting – why all the fuss? With today’s emphasis on computer-based testing – many programs provide instant results to candidates Item analysis often done once/year on item bank Generally not an opportunity for review of item analysis, comments and scoring changes

CLEAR 2008 Annual Conference Anchorage, Alaska Thus, much greater pressure for items to be right (therefore more need for pretesting) Larger, more sophisticated programs have subroutines to control item exposure to prevent overutilization Pretesting – why all the fuss?

CLEAR 2008 Annual Conference Anchorage, Alaska So what if there’s a small n? Combine data from multiple administrations (but be careful of “data expiration dates”) ….and?…

CLEAR 2008 Annual Conference Anchorage, Alaska Small n Item Analysis In interpreting item analysis for small n… –Focus primarily on p value –Largely ignore discrimination – often spurious –Can look at pattern of response, if dramatic –Rely much more on SMEs. –Review more items than you would normally (set review parameters more conservatively than you would normally)

CLEAR 2008 Annual Conference Anchorage, Alaska …So….. What else? So now we may have to get creative –Sometimes…. Something is better than nothing… but not always….

CLEAR 2008 Annual Conference Anchorage, Alaska Multiple exams-common items Although rare, some exams with multiple levels (Class 1, 2 & 3), use common items. Combine data for common items from different classes. Beware of violations of assumptions – is pretest meaningful?

CLEAR 2008 Annual Conference Anchorage, Alaska Multiple item reviews… In a way – reviews by SMEs can considered pretesting Especially important that reviewers not be original writers

CLEAR 2008 Annual Conference Anchorage, Alaska Cut Score Studies 1 Use Ebel or Nedelsky… Estimating probability of response for each choice provides critical review Also simulates responses from candidates Potential problem: difference in ability levels of SMEs and candidates

CLEAR 2008 Annual Conference Anchorage, Alaska Cut Score Studies 2 When doing an Angoff – have SMEs take test 1 st Data gathered can be used as pretest information (and also useful to use as a limit on standard setting)

CLEAR 2008 Annual Conference Anchorage, Alaska “Floating” pretest items Don’t delineate items as pretest initially Use item analysis to identify most flawed items Declare worst performing items to be Pretest Can be continued through life of examination Care must be taken to still meet blueprint

CLEAR 2008 Annual Conference Anchorage, Alaska Specialty Exams PMPs – Patient Management Problems Complicated – essentially redeveloped with each deployment Multiple item reviews/edits, help preserve quality Secure, online review and conferencing facilitates review

CLEAR 2008 Annual Conference Anchorage, Alaska Sooooooooo…. ---Good Luck with that!!!