1. RelEx Introduction to the Standardization Phase Relating language examinations to the Common European Framework of Reference for Languages Gilles Breton & José Noijons Graz, 13-15 May 2009

2. RelEx Suggested Linking Procedures in the Manual Familiarisation with the CEFR Linking on the basis of specification of examination content Standardization and Benchmarking Standard setting Validation: checking that exam results relate to CEFR levels as intended

3. RelEx Claims through Specification and Standardization If a claim of a link to the CEFR is based on specification only, we do not know what score a candidate needs to claim that the candidate’s ability is at the CEFR-level the test claims to be at. Claims can be further substantiated through standardisation of judgements: item-difficulties are judged in relation to CEFR levels.

4. RelEx Five steps in the standardization phase 1.Assuring adequate familiarisation with the CEFR. 2.Training in rating productive skills using tables and scales in the CEFR or the Manual & scales or specific rating scales developed for the examination or test in question in relation to the relevant categories of the CEFR. 3.Training in rating receptive tasks using tables in the Manual & specifications developed for the examinations or tests in question in relation to the relevant categories of the CEFR. 4.Benchmarking performance samples. 5.Standard setting of receptive tasks.

5. RelEx Standardization Phase Both benchmarking and standard setting are procedures which require group decisions, which in turn have to be carefully prepared by appropriate training.

6. RelEx Benchmarking in direct tests  In holistically rated tests, the judgment on the CEFR level is direct, and therefore it is important to assist raters in giving valid judgments. The main tool used for this special type of standard setting is called benchmarking.  Benchmarking involves providing one (or more) typical sample(s) to illustrate performance at a given level both for standardisation training and to serve as a point of reference in making future decisions about performances of candidates.

7. RelEx CIEP Benchmarking Project

8. RelEx Standard Setting in indirect tests  For tests with a numerical score, performance standards have to be set. A performance standard is the boundary (cut-off score) between two levels on the scale reported by a test. A cut-off score of 30 says that a score of 30 or more on the tests grants a level of a particular level (e.g. B1) or higher, while a lower score points to a level lower than the level of the cut-off score (here: B1).  The process to arrive at a cut-off score is commonly referred to as standard setting. In the case of receptive skills (reading and listening) or underlying competences (grammar, vocabulary), cut-off scores need to be decided upon.

9. RelEx Standard setting The decision if a person has reached a given CEFR level is based on grading, not on scoring (marking). Therefore a score must be transferred to a grading scale. Transformation of scores to grades is based on a cut- off score on a test. A cut-off score is the border between the lowest acceptable score on a test to be assigned the relevant CEFR-level and the highest score on that test to fail that level.

10. RelEx Standard setting for Reading tasks Basket procedure: judges are to assign items (texts + tasks) to CEFR-levels. What minimum CEFR level does one need to master this item?

11. RelEx Standard Setting in Receptive Skills Differences between Texts and Tasks Texts cannot be assigned levels to Difficult texts can have simple questions and v.v. A text should be relevant to the task

12. RelEx Prerequisites of Basket Method –Clarity and discriminative power of the descriptors. –The degree to which panel members understand well the meaning of the descriptors. This implies thorough familiarisation with the CEFR itself and insight into standardization. –Items or tasks of the test or examination itself can be clearly and unambiguously described and understood in terms of specific level descriptors. Panellists have to understand clearly which “Can Do” statements do apply and which ones do not apply in each and every item or task.

13. RelEx Simplified Demonstration Round Read text 1 (do not “rate” the text!) Then decide on the minimum level for each task with that text Read the next text and repeat the above actions etc. Decide what CEFR standard needs to be set Calculate cut-score (WARNING!)

14. RelEx Group Work Basket Method Work in two language groups Set standards Reconvene Discuss experiences

15. RelEx Standard Setting for Writing: Body-of-Work Method Preconditions Collection of the work of a sample of candidates. Work has received a numerical score. Works must cover the range of possible scores. Scores must not be known to judges. Conversion of panel judgments to cut-scores needs to be done through logistic regression

16. RelEx Body-of-Work Method: Tasks for Judges Decide what/how many standards need to be set, e.g. –A2/B1 and/or B1/B2 –Check with level of task Give a CEFR-related judgment of works Categorize each work, e.g. –level B1: yes or no - OR –Level B2: yes or no - OR –not level B1, B1, B2 Have group discussions –Cf. Benchmarking –2 - 3 rounds

17. RelEx Simplified Demonstration Round Judge, discuss and decide on the level of the task Give reasoned score to each of the five/six works (with reference to CEFR grids) After discussion, works are placed in numerical order (based on scores) Decide what CEFR standard needs to be set (remember task level) Place each work on either of two piles: –At x-level –Below x-level Calculate cut-score (WARNING!)

18. RelEx Group Work Body-of-Work Method Work in two language groups Set standards Reconvene Discuss experiences