1. RelEx Introduction to the Standardization Phase Relating language examinations to the Common European Framework of Reference for Languages José Noijons APEOICVA/ECML Valencia, 27-28 March 2009

2. RelEx Suggested Linking Procedures in the Manual Familiarisation with the CEFR Linking on the basis of specification of examination content Standardization and Benchmarking Standard setting Validation: checking that exam results relate to CEFR levels as intended

3. RelEx Standardization Phase Both benchmarking and standard setting are procedures which require group decisions, which in turn have to be carefully prepared by appropriate training.

4. RelEx Benchmarking in direct tests  In holistically rated tests, the judgment on the CEFR level is direct, and therefore it is important to assist raters in giving valid judgments. The main tool used for this special type of standard setting is called benchmarking.  Benchmarking involves providing one (or more) typical sample(s) to illustrate performance at a given level both for standardisation training and to serve as a point of reference in making future decisions about performances of candidates.

5. RelEx Standard Setting in indirect tests  For tests with a numerical score, performance standards have to be set. A performance standard is the boundary (cut-off score) between two levels on the scale reported by a test. A cut-off score of 30 says that a score of 30 or more on the tests grants a level of a particular level (e.g. B1) or higher, while a lower score points to a level lower than the level of the cut-off score (here: B1).  The process to arrive at a cut-off score is commonly referred to as standard setting. In the case of receptive skills (reading and listening) or underlying competences (grammar, vocabulary), cut-off scores need to be decided upon.

6. RelEx Five steps in the standardization phase 1.Assuring adequate familiarisation with the CEFR. 2.Training in rating productive skills using tables and scales in the CEFR or the Manual & scales or specific rating scales developed for the examination or test in question in relation to the relevant categories of the CEFR. 3.Training in rating receptive tasks using tables in the Manual & specifications developed for the examinations or tests in question in relation to the relevant categories of the CEFR. 4.Benchmarking performance samples. 5.Standard setting of receptive tasks.

7. RelEx Objectives CIEP Benchmarking Project Produce a DVD which presents in a reliable and transparent way comparable samples of oral production at the six levels of the CEFR in German, English, Spanish, French and Italian, produced by young people from 13 to 18 years old.

8. RelEx Experts taking part in the CIEP Benchmarking Project 48 experts from 13 European countries who had already participated in benchmarking seminars (CIEP, Goethe, Perugia) and whose reception competence was at least B2 in 2 languages and B1 in a third one. An analysis of the language competences of the 48 experts showed that, as expected, the languages in common were English and French; 23 could evaluate German, 22 Spanish and 18 Italian.

9. RelEx ENIC-NARIC France Speaking Tasks in CIEP Project Format Autonomous – no interlocutor Production Phase (each candidate) Interaction Phase Content Topics adapted to the population (13 to 18) Common tasks in production phase A1/A2;B1/B2;C1/C2 and interaction A2/B1 and C1/C2 Specific tasks in interaction phase A1 and B2

10. RelEx ENIC-NARIC France Tasks: Interaction A2 and B1 Interaction levels A2 and B1 Length : 5 to 7 minutes Partners agree on one of the topics. Topic 1 Candidate A INTERACTION You are going to organize a party for the end of the school year with a pupil from your English course. You make suggestions and ask your partner if he/she agrees Topic 2 Candidate A INTERACTION You organize a trip with an English friend. You make suggestions and ask your partner if he/she agrees. Interaction levels A2 and B1 Length : 5 to 7 minutes Partners agree on one of the topics Topic 1 Candidate B INTERACTION You are going to organize a party for the end of the school year with a pupil from your English course. You make suggestions and ask your partner if he/she agrees. Topic 2 Candidat B INTERACTION You organize a trip with an English friend. You make suggestions and ask your partner if he/she agrees.

11. RelEx Rating Instruments CEFR Table 3 –Range, Accuracy, Fluency, Interaction –Rating Form from Manual (Eurocentres) Plus Levels –Supplementary grid Additional Scales –Spoken Interaction, Spoken Production –Pronunciation

12. RelEx Can link a series of shorter, discrete simple elements into a connected, linear sequence of points. Can initiate, maintain and close simple face-to-face conversa­tion on topics that are familiar or of personal interest. Can re­peat back part of what someone has said to con­firm mutual understand­ing. Can keep going compre­hensi­bly, even though pausing for grammatical and lexical plan­ning and repair is very evident, especially in longer stretches of free produc­tion. Uses reasonably accu­rately a repertoire of frequently used "routines" and patterns asso­ ciated with more predictable situations. Has enough language to get by, with sufficient vocabulary to express him/herself with some hesitation and circum­locu­tions on topics such as family, hobbies and interests, work, travel, and current events. B1 B1+ Can use a limited number of cohesive devices to link his/her utterances into clear, coherent discourse, though there may be some "jumpiness" in a long con­ tribution. Can initiate discourse, take his/her turn when appropriate and end conversation when he/she needs to, though he/she may not always do this elegantly. Can help the discussion along on familiar ground confirm­ing comprehen­sion, inviting others in, etc. Can produce stretches of language with a fairly even tempo; although he/she can be hesitant as he or she searches for patterns and expressions, there are few noticeably long pauses. Shows a relatively high de­gree of grammatical control. Does not make errors which cause misun­ derstanding, and can correct most of his/her mistakes. Has a sufficient range of language to be able to give clear descriptions, express viewpoints on most general topics, without much con­spicuous searching for words, using some complex sentence forms to do so. B2 B2+ Can produce clear, smoothly flowing, well-structured speech, showing controlled use of organisa­tional patterns, con­ nectors and cohesive devices. Can select a suitable phrase from a readily available range of discourse functions to preface his remarks in order to get or to keep the floor and to relate his/her own contri­butions skil­fully to those of other speakers. Can express him/herself fluently and spontaneously, almost effortlessly. Only a con­ceptually difficult subject can hinder a natural, smooth flow of language. Consistently maintains a high degree of grammati­cal accu­racy; errors are rare, difficult to spot and generally corrected when they do occur. Has a good command of a broad range of language allow­ing him/her to select a formulation to express him/ herself clearly in an appropriate style on a wide range of general, academic, professional or leisure topics without having to restrict what he/she wants to say. C1 Can create coherent and cohesive discourse making full and appropriate use of a variety of organisational patterns and a wide range of connectors and other cohe­sive devices. Can interact with ease and skill, picking up and using non-verbal and intonational cues apparently effortlessly. Can inter­weave his/her con­tribution into the joint discourse with fully natural turntaking, referencing, allu­sion making etc. Can express him/herself spontaneously at length with a natural colloquial flow, avoiding or backtracking around any difficulty so smoothly that the interlocutor is hardly aware of it. Maintains consistent gram­mati­cal control of complex language, even while atten­tion is otherwise engaged (e.g. in forward planning, in monitoring others' reactions). Shows great flexibility reformu­ lating ideas in differing linguistic forms to convey finer shades of meaning precisely, to give em­ phasis, to differentiate and to eliminate ambiguity. Also has a good command of idiomatic expressions and colloquialisms. C2 COHERENCEINTERACTIONFLUENCYACCURACYRANGE

13. RelEx Rating Form

14. RelEx Rating Procedure Individual vote –Show histogram Discussion –Small group, then plenary Vote after discussion –Show histogram Final vote –Hand in rating sheet

15. RelEx Conclusions Benchmarking CIEP Benchmarks can be used as benchmarks for holistic rating of oral performances in oral tests generally. CIEP Benchmarks can be used as exercise materials to set specific benchmarks in specific tests. Applying CIEP Benchmarks can help validate claims of links to CEFR. Applying other benchmark procedures can be valid if properly researched and documented.

16. RelEx Standard setting The decision if a person has reached a given CEFR level is based on grading, not on scoring (marking). Therefore a score must be transferred to a grading scale. Transformation of scores to grades is based on a cut- off score on a test. A cut-off score is the border between the lowest acceptable score on a test to be assigned the relevant CEFR-level and the highest score on that test to fail that level.

17. RelEx Claims through Specification and Standardization If a claim of a link to the CEFR is based on specification only, we do not know what score a candidate needs to claim that the candidate’s ability is at the CEFR-level the test claims to be at. Claims can be further substantiated through standardisation of judgements: item-difficulties are judged in relation to CEFR levels.

18. RelEx Standard setting for Reading tasks Basket procedure: judges are to assign items (texts + tasks) to CEFR-levels. What minimum CEFR level does one need to master this item?

19. RelEx Standard Setting in Receptive Skills Differences between Texts and Tasks Texts cannot be assigned levels to Difficult texts can have simple questions and v.v. A text should be relevant to the task

20. RelEx Prerequisites of Basket Method –Clarity and discriminative power of the descriptors. –The degree to which panel members understand well the meaning of the descriptors. This implies thorough familiarisation with the CEFR itself and insight into standardization. –Items or tasks of the test or examination itself can be clearly and unambiguously described and understood in terms of specific level descriptors. Panellists have to understand clearly which “Can Do” statements do apply and which ones do not apply in each and every item or task.

21. RelEx Standard Setting for Writing: Body-of-Work Method Preconditions Collection of the work of a sample of candidates. Work has received a numerical score. Works must cover the range of possible scores. Scores must not be known to judges. Conversion of panel judgments to cut-scores needs to be done through logistic regression

22. RelEx Body-of-Work Method: Tasks for Judges Decide what/how many standards need to be set, e.g. –A2/B1 and/or B1/B2 –Check with level of task Give a holistic, CEFR-related judgment of works Categorize each work, e.g. –level B1: yes or no - OR –Level B2: yes or no - OR –not level B1, B1, B2 Have group discussions –Cf. Benchmarking –2 - 3 rounds

23. RelEx Simplified Demonstration Round Give intuitive/reasoned score to each of the six works (no reference to CEFR) After discussion, works are placed in numerical order (based on scores) Decide what CEFR standard needs to be set (remember task level) Place each work on either of two piles: –x-level –below x-level Calculate cut-score (WARNING!)