Using the CEFR in Catalonia Neus Figueras

The EOI system ●State funded language schools (+16) ●Two levels defined (aimed at B1 and B2) ● Published curricula ●13 different languages ●40,000 students ●16,000 certificate exams every year ●Standardised certificates since 1995

Issues to be solved in ●Revise certificate examinations. ●Is examination difficulty equivalent across time? ●Is the lower certificate consistently easier than the higher certificate? ●Are the different certificates in the different languages comparable?

Issues to be solved in ●Develop level specifications related to the CEFR. ●Revise existing curricula in relation to the CEFR. ●Link certificates to CEFR levels.

● Project design ● Empirical scale development ● Item banking (English) ● Manual procedures for linkage, (Specification Standardisation, Empirical validation) ● Defining and exemplifying A2 ● Developing curriculum objectives for A2, B1 and B2 ● Developing test specifications for A2, B1 and B2 Project Overview

Scale Development Item Banking Linking tests Changing curricula CEFR

Challenges ●Where to start? ●Involve teachers (and item writers). ●Improve existing practice. ●Bring in the ELP onto the project. ●How? Need to count on experts. ●Limited resources.

Scale development (all languages)

CEFR Methodology Step 1: selecting level descriptors. Step 2: translation into Catalan. Step 3: mapping descriptors onto levels. Step 4: developing and validating new scales.

Descriptors Teachers* Languages ●Reading ●Listening ●Speaking ●Writing ●Grammar ●Vocabulary * Arabic, Basque, Catalan, Dutch, English (>40), French (>20), German (>15), Italian, Japanese, Portuguese, Russian, Spanish Teachers involved

Lessons learnt from scale development ●Continued training/familiarisation is necessary. ●Appearances need to be checked empirically. ●Exact correspondence may not be possible, but is it desirable?. ●Linkage does not mean equivalence.

Item Banking (English)

Methodology Step 1 ● Booklet development

Booklet design for anchoring items (2003) S1S2S3S4S5L1L2L3A1A2A3 V1XX X V2 XX X V3 XX X V4 XXX V5X X X V6 X XXX V7 XX XX V8 XX X X

Methodology Step 2 ● Data collection ( students per item) ● Analyses: CTT and IRT (Total surviving items :301) Step 3 ● Standard setting procedures: - test centered - examinee centered - annual average pass rate ● Setting (provisional) cut off scores at Elemental (B1) and Aptitud (B2)

TCC of the eight versions(2003)

Using the Manual for linking exams (English, French, German)

Challenges in Specification ●How to map the examination? Different versions of the test? Specifications? ●How to tackle differences of coverage (subskills)? ●Who does what? ●Who checks it?

Challenges in Standardisation ●Reference “r” materials not ready until ●Differences across skills, across languages. ●Assessing task vs. item levels. ● No “linguistic competence” reference materials.

The proposal from Dutch CEFR project ●Training. ●Describing texts and items according to set parameters (reading and listening). ●Estimating their CEFR levels. ●Pretesting the items thus labelled. ●Calibrating the items. ●Standard-setting on the scale coming from the calibration. ●Assigning a psychometric level to the items. ●Assigning a definitive level to the items.

Using the Council of Europe Item CD (German)

The German booklet ReadingListening Students B1 B Items A2 C1 40 Judges Native-non native 1210 InstitutionsDialang, Goethe, WBT, TestDAF, EOI Dialang, Goethe, WBT, EOI

Inter-judge Consistency in assessing items R15L22

Judges: Consistency with the Pre-Estimation

Consistency pre-estimation with Empirical Results

Developing CEFR based curricula

Developing CEFR-based curricula ●Focusing on what students can do. ●Drafting objectives. ●Defining content. ●Defining assessment criteria. + ●Methodological guidelines.

Conclusions so far

The CEFR IS a Bible ( but only in the widest sense of the word)

The Manual is no book of spells

There is no ONE holy Grail

●We learnt much more about our exams. ●It has been a competence building process. ●We have become less dogmatic. ●We know there is further work to do and room for improvement. ●Combining highly technical work with enthusiasm is crucial.

How do I know if my B1 is your B1? This is my B1. What’s your B1 like?