1. CLEAR 2008 Annual Conference Anchorage, Alaska “Problems and Priorities in Pretesting Pondered” Beth Noeller Grady Barnhill Carol O’Byrne

2. CLEAR 2008 Annual Conference Anchorage, Alaska

3. CLEAR 2008 Annual Conference Anchorage, Alaska All Items are not Created Equal

4. CLEAR 2008 Annual Conference Anchorage, Alaska Purposes of Pretesting Gather performance information Add new content

5. CLEAR 2008 Annual Conference Anchorage, Alaska What Information do you Gather? Performance data ♦Classical and IRT ♦Difficulty ♦Discrimination Cognitive Level Data Content

6. CLEAR 2008 Annual Conference Anchorage, Alaska Difficulty How hard is the item? Usually expressed as a %. What percent of the candidates get the item correct? IRT b-value

7. CLEAR 2008 Annual Conference Anchorage, Alaska What is a “good” difficulty? What is your cut score? The best difficulty is right around the cut score. Too easy and too hard provide little information. Between 40 & 90 for many testing programs

8. CLEAR 2008 Annual Conference Anchorage, Alaska Discrimination How well an item differentiates between those who perform well on the test and those who get the item correct. Correlation Values from -1.0 to +1.0 IRT a-value

9. CLEAR 2008 Annual Conference Anchorage, Alaska Positive Discrimination Values Two people walking together ‘in step’ Get an item correct and receive a high score on the examination Perfect positive relationship +1.0 Item correctness exam score

10. CLEAR 2008 Annual Conference Anchorage, Alaska Negative Discrimination Values Two people walking in opposite directions Get an item correct and receive a low score on the examination Perfect negative relationship -1.0 Item correctness exam score

11. CLEAR 2008 Annual Conference Anchorage, Alaska What is a “good” discrimination? A positive relationship The higher the better Diversity of the candidate population Greater than 0.15 exam score Item correctness

12. CLEAR 2008 Annual Conference Anchorage, Alaska Combining Difficulty and Disc. Each quality by itself is linear Together they can be depicted on a two- dimensional graph

13. CLEAR 2008 Annual Conference Anchorage, Alaska

14. CLEAR 2008 Annual Conference Anchorage, Alaska

15. CLEAR 2008 Annual Conference Anchorage, Alaska

16. CLEAR 2008 Annual Conference Anchorage, Alaska

17. CLEAR 2008 Annual Conference Anchorage, Alaska

18. CLEAR 2008 Annual Conference Anchorage, Alaska

19. CLEAR 2008 Annual Conference Anchorage, Alaska

20. CLEAR 2008 Annual Conference Anchorage, Alaska

21. CLEAR 2008 Annual Conference Anchorage, Alaska

22. CLEAR 2008 Annual Conference Anchorage, Alaska

23. CLEAR 2008 Annual Conference Anchorage, Alaska

24. CLEAR 2008 Annual Conference Anchorage, Alaska

25. CLEAR 2008 Annual Conference Anchorage, Alaska

26. CLEAR 2008 Annual Conference Anchorage, Alaska

27. CLEAR 2008 Annual Conference Anchorage, Alaska

28. CLEAR 2008 Annual Conference Anchorage, Alaska

29. CLEAR 2008 Annual Conference Anchorage, Alaska

30. CLEAR 2008 Annual Conference Anchorage, Alaska Summary

31. CLEAR 2008 Annual Conference Anchorage, Alaska How do you use pretesting to add new content? Get the pulse of your item bank Inventory of acceptable scored items Inventory of unused items

32. CLEAR 2008 Annual Conference Anchorage, Alaska Scored Items and Test Specs

33. CLEAR 2008 Annual Conference Anchorage, Alaska Pretest Needs

34. CLEAR 2008 Annual Conference Anchorage, Alaska Beth’s Contact Information Beth Noeller, PhD 319-339-3095 beth.noeller@act.org