1. Using the CEFR in Catalonia Neus Figueras nfiguera@xtec.cat

2. The EOI system ●State funded language schools (+16) ●Two levels defined (aimed at B1 and B2) ● Published curricula ●13 different languages ●40,000 students ●16,000 certificate exams every year ●Standardised certificates since 1995

3. Issues to be solved in 2002-1 ●Revise certificate examinations. ●Is examination difficulty equivalent across time? ●Is the lower certificate consistently easier than the higher certificate? ●Are the different certificates in the different languages comparable?

4. Issues to be solved in 2002-2 ●Develop level specifications related to the CEFR. ●Revise existing curricula in relation to the CEFR. ●Link certificates to CEFR levels.

5. ● Project design ● Empirical scale development ● Item banking (English) ● Manual procedures for linkage, (Specification Standardisation, Empirical validation) ● Defining and exemplifying A2 ● Developing curriculum objectives for A2, B1 and B2 ● Developing test specifications for A2, B1 and B2 Project Overview 2003-2007

6. Scale Development Item Banking Linking tests Changing curricula CEFR

7. Challenges ●Where to start? ●Involve teachers (and item writers). ●Improve existing practice. ●Bring in the ELP onto the project. ●How? Need to count on experts. ●Limited resources.

8. Scale development (all languages)

9. CEFR Methodology Step 1: selecting level descriptors. Step 2: translation into Catalan. Step 3: mapping descriptors onto levels. Step 4: developing and validating new scales.

10. Descriptors Teachers* Languages ●Reading 40 10310 ●Listening46 9910 ●Speaking80 9212 ●Writing53 8912 ●Grammar34 8112 ●Vocabulary61 73 12 * Arabic, Basque, Catalan, Dutch, English (>40), French (>20), German (>15), Italian, Japanese, Portuguese, Russian, Spanish Teachers involved

11. Lessons learnt from scale development ●Continued training/familiarisation is necessary. ●Appearances need to be checked empirically. ●Exact correspondence may not be possible, but is it desirable?. ●Linkage does not mean equivalence.

12. Item Banking (English)

13. Methodology Step 1 ● Booklet development

14. Booklet design for anchoring items (2003) S1S2S3S4S5L1L2L3A1A2A3 V1XX X V2 XX X V3 XX X V4 XXX V5X X X V6 X XXX V7 XX XX V8 XX X X

15. Methodology Step 2 ● Data collection (260-784 students per item) ● Analyses: CTT and IRT (Total surviving items :301) Step 3 ● Standard setting procedures: - test centered - examinee centered - annual average pass rate ● Setting (provisional) cut off scores at Elemental (B1) and Aptitud (B2)

16. TCC of the eight versions(2003)

17. Using the Manual for linking exams (English, French, German)

18. Challenges in Specification ●How to map the examination? Different versions of the test? Specifications? ●How to tackle differences of coverage (subskills)? ●Who does what? ●Who checks it?

19. Challenges in Standardisation ●Reference “r” materials not ready until 2005. ●Differences across skills, across languages. ●Assessing task vs. item levels. ● No “linguistic competence” reference materials.

20. The proposal from Dutch CEFR project ●Training. ●Describing texts and items according to set parameters (reading and listening). ●Estimating their CEFR levels. ●Pretesting the items thus labelled. ●Calibrating the items. ●Standard-setting on the scale coming from the calibration. ●Assigning a psychometric level to the items. ●Assigning a definitive level to the items.

21. Using the Council of Europe Item CD (German)

22. The German booklet ReadingListening Students B1 B2 294307 Items A2 C1 40 Judges Native-non native 1210 InstitutionsDialang, Goethe, WBT, TestDAF, EOI Dialang, Goethe, WBT, EOI

23. Inter-judge Consistency in assessing items R15L22

24. Judges: Consistency with the Pre-Estimation

25. Consistency pre-estimation with Empirical Results

26. Developing CEFR based curricula

27. Developing CEFR-based curricula ●Focusing on what students can do. ●Drafting objectives. ●Defining content. ●Defining assessment criteria. + ●Methodological guidelines.

28. Conclusions so far

29. The CEFR IS a Bible ( but only in the widest sense of the word)

30. The Manual is no book of spells

31. There is no ONE holy Grail

32. ●We learnt much more about our exams. ●It has been a competence building process. ●We have become less dogmatic. ●We know there is further work to do and room for improvement. ●Combining highly technical work with enthusiasm is crucial.

33. How do I know if my B1 is your B1? This is my B1. What’s your B1 like?