1. Relating examinations to the CEFR – the Council of Europe Manual and supplementary materials Waldek Martyniuk ECML, Graz, Austria

2. The CEFR and assessment Potential Potential Challenges Challenges Solutions Solutions Recommendations Recommendations

3. CEFR and assessment: the potential „ … the Framework can be used: for the specification of the content of tests and examinations for the specification of the content of tests and examinations for stating the criteria to determine the attainment of a learning objective (…) for stating the criteria to determine the attainment of a learning objective (…) for describing the levels of proficiency in existing tests and examinations thus enabling comparisons to be made across different systems of qualifications” for describing the levels of proficiency in existing tests and examinations thus enabling comparisons to be made across different systems of qualifications” (CEFR (CEFR: 178)

4. CEFR and assessment : the potential 6 levels of proficiency defined largely in relation to empirically derived difficulty estimates based on stakeholder perceptions of what language functions expressed by Can-Do statements can be successfully performed at each level 6 levels of proficiency defined largely in relation to empirically derived difficulty estimates based on stakeholder perceptions of what language functions expressed by Can-Do statements can be successfully performed at each level „ least arbitrary sequence of scaled proficiency descriptors available to us at the moment ” „ least arbitrary sequence of scaled proficiency descriptors available to us at the moment ” (Weir 2004)

5. CEFR and assessment: challenges Test development and analysis Test development and analysis Relating examinations Relating examinations Validating claims Validating claims QMS QMS

6. Using CEFR for test development and analysis „the scales are premised on an incomplete and unevenly applied range of contextual variables/performance conditions (context validity); „the scales are premised on an incomplete and unevenly applied range of contextual variables/performance conditions (context validity); little account is taken of the nature of cognitive processing at different levels of ability (theory based validity); little account is taken of the nature of cognitive processing at different levels of ability (theory based validity); activities are seldom related to the quality of actual performance expected to complete them (scoring validity); activities are seldom related to the quality of actual performance expected to complete them (scoring validity); the wording for some of the descriptors is not consistent or transparent enough in places for the development of tests.” the wording for some of the descriptors is not consistent or transparent enough in places for the development of tests.” (Weir 2004)

7. CEFR and assessment: the limitations NOT a harmonisation tool: NOT a harmonisation tool: “We have NOT set out to tell practitioners what to do or how to do it. We are raising questions not answering them. It is not the function of the CEF to lay down the objectives that users should pursue or the methods they should employ” (CEFR “We have NOT set out to tell practitioners what to do or how to do it. We are raising questions not answering them. It is not the function of the CEF to lay down the objectives that users should pursue or the methods they should employ” (CEFR: xi) NOT a theory of language development: NOT a theory of language development: Scales describe learning outcomes, learner behaviours, not the invisible processes involved. NOT a test specification: NOT a test specification: Scales and lists can be consulted when drawing up a task specification (Ch. 4) or defining assessment criteria (Ch. 5) but need reference to detailed specifications for language & context. (North 2004)

8. CEFR and assessment: s olutions Reference Level Descriptions Reference Level Descriptions Grids for the analysis of test items and tasks Grids for the analysis of test items and tasks Manual for relating examinations Manual for relating examinations to the CEFR + supplement Benchmarking conferences Benchmarking conferences Benchmarked test items/tasks and Benchmarked test items/tasks andperformances Guidelines for examiners and test developers (revised) Guidelines for examiners and test developers (revised)

9. CEFR and assessment: recommendations Main types of assessment: Main types of assessment: Placement Placement Formative Formative Summative Summative Main modes of assessment: Main modes of assessment: Self-assessment Self-assessment Teacher assessment Teacher assessment Test Test ( North 2005)

10. Manual for relating language examinations to the CEFR Drafted by an authoring group in 2003 Drafted by an authoring group in 2003 Piloting the Draft: 2004 - 2007 Piloting the Draft: 2004 - 2007 Revising and publishing: 2009 Revising and publishing: 2009 Case study publication: 2009 Case study publication: 2009 Standard setting publication: 2009 Standard setting publication: 2009

11. Suggested Procedures Familiarisation with the CEFR Familiarisation with the CEFR Linking on the basis of specification of examination content Linking on the basis of specification of examination content Standardisation and Benchmarking Standardisation and Benchmarking Standard setting Standard setting Empirical validation: checking that exam results relate to CEFR levels as intended Empirical validation: checking that exam results relate to CEFR levels as intended

12. Reference Materials Reference Supplement Reference Supplement Videos and DVDs with calibrated sample spoken performances Videos and DVDs with calibrated sample spoken performances CD-ROMs with calibrated sample items and tasks for reading and listening CD-ROMs with calibrated sample items and tasks for reading and listening Calibrated sample written tasks and performances (web page based) Calibrated sample written tasks and performances (web page based) Grids for the analysis of test items, tasks and performances Grids for the analysis of test items, tasks and performances

13. The Manual 2009 Chapter 1: The CEFR and the manual Chapter 2: The Linking Process Chapter 3: Familiarisation Chapter 4: Specification Chapter 5: Standardisation Training and Benchmarking Chapter 6: Standard setting (panel based approach) Chapter 7: Validation Appendices:A: forms and scales for chapter 4 B: content analysis Grids (chap 4) C: forms and scales for chapter 5

14. The Manual 2009 Appendices: Appendices: A: forms and scales for chapter 4 A: forms and scales for chapter 4 B: content analysis Grids (chapter 4) B: content analysis Grids (chapter 4) C: forms and scales for chapter 5 C: forms and scales for chapter 5 Further material: Further material: Standard setting and standard maintaining (scalar perspectives) Standard setting and standard maintaining (scalar perspectives)

15. The Reference Supplement Section A: Summary of linking process Section A: Summary of linking process Section B: Standard Setting Section B: Standard Setting Section C: Classical Test Theory Section C: Classical Test Theory Section D: Qualitative Analysis Methods Section D: Qualitative Analysis Methods Section E: Generalisability Theory Section E: Generalisability Theory Section F: Factor Analysis Section F: Factor Analysis Section G: Item Response Theory Section G: Item Response Theory Section H: Test Equating Section H: Test Equating

16. All materials available for downloading from: www.coe.int/lang or www.coe.int/portfolio Thank you for your kind attention! www.coe.int/langwww.coe.int/portfolio www.coe.int/langwww.coe.int/portfolio