1. © Stichting CITO Instituut voor Toetsontwikkeling 1 Mapping the Dutch Foreign Language State Examinations onto the Common European Framework of Reference Report of a Cito research project commissioned by the Dutch Ministry of Education, Culture and Science José Noijons & Henk Kuijper

2. © Stichting CITO Instituut voor Toetsontwikkeling 2 Project Goals To establish links between the existing examinations in French, German and English and the CEFR, following the steps as outlined in the Manual published by the Council of Europe. To study the possibilities of developing more comprehensive CEFR-related examinations in the foreign languages.

3. © Stichting CITO Instituut voor Toetsontwikkeling 3 Four Project Phases Phase 1: Familiarisation Phase 2: Specification Phase 3: Standardisation Phase 4: Empirical validation

4. © Stichting CITO Instituut voor Toetsontwikkeling 4 Phase 2: Specification Specification of texts Related to the reading scales for communicative language activities (CEFR chapter 4) Related to text type, source, topic, domain (Dutch Grid) Related to the scales for communicative language competence (CEFR chapter 5 & Dutch Grid)

5. © Stichting CITO Instituut voor Toetsontwikkeling 5 Phase 2: Specification Specification of items Question types Task dimensions 1 (recognising, inferences, evaluation) Task dimensions 2 (explicit vs. implicit) Task dimensions 3 (content of operations)

6. © Stichting CITO Instituut voor Toetsontwikkeling 6 Specification texts: reading scales communicative activities German

7. © Stichting CITO Instituut voor Toetsontwikkeling 7 Specification texts: text type, source, topic, domain Great variation in text source and communicative themes (topics) Increase of proportion of expository and argumentative texts from lower to higher educational levels Domain: mainly personal

8. © Stichting CITO Instituut voor Toetsontwikkeling 8 Specification texts: communicative language competences Level of abstraction Grammatical complexity Vocabulary Text length Dutch grid

9. © Stichting CITO Instituut voor Toetsontwikkeling 9 Specification texts: communicative language competences, level of abstraction Only concrete1 Mostly concrete2 Fairly extensive abstract3 Mainly abstract4

10. © Stichting CITO Instituut voor Toetsontwikkeling 10 Specification texts: communicative language competences, grammar Only simple structures1 Mainly simple structures2 Limited range of complex structures3 Wide range of complex structures4

11. © Stichting CITO Instituut voor Toetsontwikkeling 11 Specification items From lower to higher educational level: more inferences and evaluation, less recognising Relatively more implicit information Greater variation of operations

12. © Stichting CITO Instituut voor Toetsontwikkeling 12 Specification: general conclusions Increase of linguistic and cognitive complexity of texts Increase of variation of operations demanded in the items

13. © Stichting CITO Instituut voor Toetsontwikkeling 13 Claims through Specification and Standardisation If a claim of a link to the CEFR is based on specification only, we do not know what score a candidate needs to claim that the candidate’s ability is at the CEFR-level the test claims to be at. Claims can be further substantiated through standardisation of judgements: item- difficulties are judged in relation to CEFR levels.

14. © Stichting CITO Instituut voor Toetsontwikkeling 14 Standardisation Standard-setting Basket procedure: judges are to assign items (texts + tasks) to CEFR-levels. What minimum CEFR level does one need to master this item?

15. © Stichting CITO Instituut voor Toetsontwikkeling 15 Results for Exams in English Required minimum CEFR level for Exams in English

16. © Stichting CITO Instituut voor Toetsontwikkeling 16 Results for Exams in French Required minimum CEFR level for Exams in French

17. © Stichting CITO Instituut voor Toetsontwikkeling 17 Results for Exams in German Required minimum CEFR level for Exams in German

18. © Stichting CITO Instituut voor Toetsontwikkeling 18 Scores & Standards: English Distribution of scores & cut score vwo-exam English VWO English Score distribution

19. © Stichting CITO Instituut voor Toetsontwikkeling 19 Scores & Standards: French Distribution of scores & cut score vwo-exam French VWO French Score distribution VWO French

20. © Stichting CITO Instituut voor Toetsontwikkeling 20 Scores & Standards: German Distribution of scores & cut score vwo-exam German VWO English Score distribution VWO German

21. © Stichting CITO Instituut voor Toetsontwikkeling 21 Conclusions standard-setting In the Netherlands a candidate can “pass” an exam without attaining the relevant CEFR level for that exam. Only scores at, or higher than the CEFR cut score indicate that the candidate is at or above the CEFR level the exam has been estimated to be at. Empirical validation may help to show that a CEFR level (a score) attained on one exam is equivalent to a CEFR level on another exam.

22. © Stichting CITO Instituut voor Toetsontwikkeling 22 General Conclusions A test that is linked or validated through specification only, cannot provide sufficient information on how candidates need to perform on the test to claim they have reached relevant attainment targets. It is necessary through standard setting to compute minimum scores that are needed for candidates to claim they have reached relevant attainment targets. External validation is needed to verify claims of links to the CEFR.