1. In the Science and Practice of Standard Setting, Where is the Science? Barbara S. Plake, University of Nebraska-Lincoln, Emeritus

2. Purpose of Presentation Standard setting involves applications of Social science Psychometrics Content expertise Politics economics Decisions based on scientific studies but many based on human judgment or for streamlining process Presentation will focus on a few areas where additional research is needed to support many of these practical decisions

3. Overview of topics Selection of panelists Panelists’ training Panelists’ operational ratings Feedback to panelists during rounds of ratings Cross panel/vertical articulation of cutscores

4. Selection of Panelists What is the impact of involving “outside” stakeholders on panels? What is the efficacy of using crowd-sourcing to create panel of raters?

5. Panelist Training What is the effect of providing pre-meeting materials to panelists?

6. Panelists’ Training How critical is it for the panelists to “take the test”? Is it sufficient for them just to “experience the test”? Should panelists have a fuller practice in the method during training, including practice with feedback?

7. Panelists’ Operational Ratings Does it matter which performance category is rated first? Should the ratings be sequential or hierarchical? Should the same RP value be used across content areas? What is the best strategy for setting cutscores on computerized adaptive tests?

8. Feedback to Panelists How does the type and timing of feedback affect panelists’ ratings? How do graphical displays impact panelists’ interpretation of feedback data?

9. Cross-grade/vertical articulation of cutscores What are the optimal ways to conduct cross- grade/vertical articulation of cutscores? How much flexibility should the cross-grade panels be allowed in adjusting cutscores?

10. Conclusion Multiple areas for research to support validity of cutscore interpretations and uses Standard setting plays a crticial role in assessment practices Results provide meaning to test scores Validity of score interpreations relies directly on the quality of the standard setting process. Research is needed to help ensure validity is a priority