Standard Setting Zagreb, July 2009

The Problem: Setting cut scores for criterion referenced tests Not having many test takers Having no concurrent validity

What is a standard setting process? The step where the policy descriptions and elaborated descriptions are transformed into a different language, the numerical language of the test score There is an intention in the standard that is specified by the policy of the agency. (Reckase, 2009)

Purpose The purpose of the standard setting process must be made clear. We must be clear about what we want to achieve in making pass/fail decisions, Do we want to exclude or include, for example

Procedure Select a large and representative panel Choose a standard setting method Be sure to have good descriptors of performance categories Train participants to use decriptors and standard setting method Compile item ratings from participants Facilitate discussion among participants about first round of rating, Individual feedback and results of pretesting Make new round of item ratings Let participants review again – then arrive at final recommended cut score Assemble documentation af the standard setting process

About the procedure Three types of feedback to participants after first round Normative Reality Impact Hvordan deres egne resultater svarer til andres standard ceviation ets Hvordan items virkede i test population (kan selvfølgelig ikke have masters og non masters da cut scorer ikke er sathvilkenvirkning cut score vil have på senere testtagere

The Benchmark Advisory Test (BAT) and Standard Setting What is the BAT ? Angoff’s footnote suggests that one might ”ask each judge to state the probability that the minimally acceptable person would answer each item correctly” (meaning of course a minimally competent examinee) During first round of rating we answered the 2 questions: What level is this item and how many people at a threshold level would get this item right?

Our modified, modified Angoff (example from last round of rating) Estimate the correct response rate for each item for examinees at 3 proficiency levels The range entered may range from about 25% to 100 % 25%= the chance of random response being correct 100%= no chance of answering incorrectly, even due to a lapse in attention or any other reason At one level below the target level At the targeted level At one level above the targeted level

The process was iterative. Three rounds of ratings, one via internet Mean and Standard deviation of the ratings are calculated (Each rater could see his/her own deviation from the mean) Final passing score is calculated by averaging the rater means

Bibliografi Reckase, Mark D.: Standard Setting Theory and Practise: Issues and Difficulties. In Linking the CEFR levels: Research Perspectives (ed N Figueras and José Noijons)(CITO; EALTA, Arnhem 2009) Cizek, Gregory J. and Bunch , Michael B.: Standard Setting (sage Publications, 2007)