1. Standard Setting
Zagreb, July 2009

2. The Problem: Setting cut scores for criterion referenced tests
Not having many test takers
Having no concurrent validity

3. What is a standard setting process?
The step where the policy descriptions and elaborated descriptions are transformed into a different language, the numerical language of the test score
There is an intention in the standard that is specified by the policy of the agency.
(Reckase, 2009)

4. Purpose
The purpose of the standard setting process must be made clear.
We must be clear about what we want to achieve in making pass/fail decisions,
Do we want to exclude or include, for example

5. Procedure Select a large and representative panel
Choose a standard setting method
Be sure to have good descriptors of performance categories
Train participants to use decriptors and standard setting method
Compile item ratings from participants
Facilitate discussion among participants about first round of rating, Individual feedback and results of pretesting
Make new round of item ratings
Let participants review again – then arrive at final recommended cut score
Assemble documentation af the standard setting process

6. About the procedure
Three types of feedback to participants after first round
Normative
Reality
Impact
Hvordan deres egne resultater svarer til andres standard ceviation ets
Hvordan items virkede i test population (kan selvfølgelig ikke have masters og non masters da cut scorer ikke er sathvilkenvirkning cut score vil have på senere testtagere

7. The Benchmark Advisory Test (BAT) and Standard Setting
What is the BAT ?
Angoff’s footnote suggests that one might ”ask each judge to state the probability that the minimally acceptable person would answer each item correctly” (meaning of course a minimally competent examinee)
During first round of rating we answered the 2 questions: What level is this item and how many people at a threshold level would get this item right?

8. Our modified, modified Angoff (example from last round of rating)
Estimate the correct response rate for each item for examinees at 3 proficiency levels
The range entered may range from about 25% to 100 %
25%= the chance of random response being correct
100%= no chance of answering incorrectly, even due to a lapse in attention or any other reason
At one level below the target level
At the targeted level
At one level above the targeted level

9. The process was iterative. Three rounds of ratings, one via internet
Mean and Standard deviation of the ratings are calculated
(Each rater could see his/her own deviation from the mean)
Final passing score is calculated by averaging the rater means

10. Bibliografi
Reckase, Mark D.: Standard Setting Theory and Practise: Issues and Difficulties. In Linking the CEFR levels: Research Perspectives (ed N Figueras and José Noijons)(CITO; EALTA, Arnhem 2009)
Cizek, Gregory J. and Bunch , Michael B.: Standard Setting (sage Publications, 2007)