Relating Examinations to the CEFR Empowering Language Professionals RelEx Project Relating Examinations to the CEFR Gilles Breton José Noijons ECML Conference 2011 Empowering Language Professionals Graz, 30 September 2011

RelEx: results and audience Expected results and output: Training materials Highlights from the Manual on relating examinations to the CEFR Target audience for the activities planned within the project:   Examiners > RelEx Teacher trainers > ClassRelEx Target audience for the project results:  Stakeholders in national language examinations Sector of education focussed on: lower and upper secondary school vocationally oriented schools

RelEx Team Members Jana Berešová: Trnava University, Slovak Republic Gilles Breton: CIEP, Centre international d'études pédagogiques, France José Noijons: (Coordinator), Cito, Institute for Educational Measurement,The Netherlands Gabor Szabo: European Consortium for the Certificate of Attainment in Modern Languages, Hungary

Specifics on (Class)RelEx workshops Technical, psychometric expertise is needed to actually carry out the linking process. Participants have been given information on linking to the CEFR, but there is no one (best) way of linking The CoE nor the ECML will validate or have validated any claims of links, there is no agency to do this. The RelEx project is to collect resources in one place.

Manual for relating language examinations to the CEFR Aim To help the providers of examinations to develop, apply and report transparent, practical procedures to relate their examinations to the CEFR Approach: To help describe the examination coverage, administration and analysis procedures To help relate results reported from the examination to the CEFR Common Reference Levels To help provide supporting evidence that reports the procedures followed to do so

Highlights from the Manual (1) An introduction to the Manual that highlights some of the most important aspects of the linking process. Highlights can play an important role in making the Manual better known to stakeholders. Highlights can make practising and future teachers aware that claims of links to the CEFR need to have been validated. When considering to use existing assessment instruments teachers can look for such validation. Teachers can apply some of the procedures as outlined in (Highlights from) the Manual when developing their own classroom-based tests.

Highlights from the Manual (2) Highlights from the Manual have been used in workshops organised by the ECML. Participants have commented on their conciseness and their accessibility to non-experts. Highlights have been taken from the Manual only: no texts have been changed or been added to, except in a few cases where a technical term has been explained.

Highlights from the Manual (3) The message of the (Class)RelEx project has been that linking foreign language examinations to the CEFR is an activity worth doing. It is also a complicated business that needs to be carried out by professionals.

Suggested Linking Procedures in the Manual *Familiarisation with the CEFR Linking on the basis of Specification of examination content Standardisation training and *Benchmarking Standard setting Empirical validation: checking that exam results relate to CEFR levels as intended 9

Familiarisation Training activities based on a detailed knowledge of the CEFR, its levels and illustrative descriptors.

Specification A self-audit of the coverage of the examination (content and task types) and improving the quality of the examination through the procedures having a certain awareness-raising function.

Standardisation Training and Benchmarking Producing a common understanding of the Common Reference Levels (spoken and written performances) Benchmarking – performance samples (from the tests) benchmarked to CEFR levels claimed in the specification phase

Standard Setting Reaching decisions on cut-off scores, borderline performances.

(External) Validation A continuous process of quality monitoring Not an ultimate verdict on the quality of the linking process.