1. Centre for Applied Linguistics Dr Claudia Harsch Centre for Applied Linguistics University of Warwick From Norm- to Standards-based assessment What role can the CEFR play? CBSE-CAER International Conference 2014 Delhi-NCR, 28th and 29th April 2014

2. Centre for Applied Linguistics Overview Norm- and Criterion-Oriented Assessment The Common European Framework of Reference (CEFR) and its use in assessment – Classroom-based assessment – Educational Monitoring – International Certificates Transition from norm- to criterion-orientation: Chances and Challenges – Preparing students and teachers – Aligning exams to the criterion (CEFR) – The multi-lingual context

3. Centre for Applied Linguistics Norm-Orientation Criterion-Orientation

4. Centre for Applied Linguistics Comparison (e.g. Bachman & Palmer 2010, Douglas 2010) Norm-Orientation Quantitative score Ranking learners Comparing learners Placement decisions Grading decisions Criterion-Orientation Qualitative description of... - what skills / competencies learners have acquired - how well learners perform in relation to learning outcomes Transparent expectations, goals, outcomes expressed in criteria Comparability via common goal posts

5. Centre for Applied Linguistics 1. Classroom-based assessment 2. Educational Monitoring 3. International certification The Common European Framework of Reference (CEFR) in criterion-oriented assessment

6. Centre for Applied Linguistics The CEFR Instrument to enhance communication and mobility (language policy of the Council of Europe) At core: Proficiency model with levels of proficiency Description of learners’ abilities as positive ‘Can do’ statements Framework allows qualitative description of learning outcomes and assessment results (rather than only numerical scores) –> reference criteria

7. Centre for Applied Linguistics The CEFR levels A1 A2 B1 B2 C1 C2

8. Centre for Applied Linguistics The CEFR scale system Communicative Competence Production Communicative Language Competences Communicative Language Activities Communicative Strategies ReceptionInteraction Grammar Vocabulary Orthography Pronunciation

9. Centre for Applied Linguistics CEF Proficiency Levels Student with high proficiency Student with low proficiency C2: Can understand a wide range of long and complex texts, appreciating subtle distinctions of style and implicit as well as explicit meaning. A1: Can understand very short, simple texts a single phrase at a time, picking up familiar names, words and basic phrases and rereading as required.

10. Centre for Applied Linguistics CEFR in Classroom-based assessment Defining learning outcomes, develop curriculum using CEFR descriptors Self-assessment: via the self-assessment grid or the European Language Portfolio Peer-assessment via common criteria, to be derived and specified on basis of e.g. grid or CEFR descriptors Teacher assessment via the CEFR scales, need to specify for particular performances Advantage of common and transparent goals, criteria and levels, shared expectations Possibility of comparing different views on learners

11. Centre for Applied Linguistics CEFR in Educational Monitoring CEFR as external standard / criterion, e.g. National Educational Standards (NES) Germany: Defining learning outcomes /competencies expected at the end of secondary schooling in relation to CEFR levels and descriptors Learning outcomes operationalised in standards- based tests Tests formally aligned to the CEFR Test results reported on competency scale (like PISA), accompanied by qualitative descriptions (CEFR, NES) Advantage of linking external monitoring to internal teacher-/student-assessment

12. Centre for Applied Linguistics CEFR in International Certification CEFR as common proficiency framework International tests aligned to CEFR levels Test results/reports: scores and CEFR levels Comparability: CEFR as common reference point Challenge: “Is my B1 your B1?” High-stakes decisions, e.g. university admission Political agenda, e.g. visa and immigration decisions (ethical?)

13. Centre for Applied Linguistics 1. Preparing teachers and students 2. Aligning exams to the CEFR 3. The Multi-lingual Context Transition from norm- to criterion-based assessment

14. Centre for Applied Linguistics Preparing teachers and students Traditional focus on ranking, competition, errors, correct answers Now: shift towards criteria, outcomes, skills Focus on what learners can do with their language, whether they can perform certain communicative tasks Requires willingness and readiness for change Requires training resources

15. Centre for Applied Linguistics Preparing teachers and students Teacher Training: Familiarity with CEFR – Document available online: www.coe.int/lang-CEFRwww.coe.int/lang-CEFR – Online training: www.helsinki.fi/project/ceftrainwww.helsinki.fi/project/ceftrain Teacher Training: Familiarity with rating approaches, self- and peer-assessment, Portfolio http://www.coe.int/t/dg4/education/elp/Default_EN.asp http://www.coe.int/t/dg4/education/elp/Default_EN.asp Students: Introduction to self-assessment and Portfolio by their teachers, familiarity over time Classroom: Developing a culture of peer-assessment, supportive and honest but respectful feedback

16. Centre for Applied Linguistics Aligning Exams to the CEFR Manual to support such endeavours http://ww.coe.int/t/dg4/linguistic/manuel1_en.asp http://ww.coe.int/t/dg4/linguistic/manuel1_en.asp Several steps to follow – resources needed Range of standard setting methods available – expertise needed (e.g. Cizek et al 2007) Human judgement – uncertainties and biases Research projects ongoing to – report exam alignments in a transparent way – examine most feasible ways of alignment (e.g. Eckes 2012, Figueras et al. 2005, Harsch et al. 2010)

17. Centre for Applied Linguistics The Multi-Lingual Context Specific multi-lingual skills, learning outcomes, expectations Not covered in the CEFR Possibility to define such a set of multi-lingual skills for the Indian context – Relevance – Transparency – Added value: setting an example for other multi- lingual contexts

18. Centre for Applied Linguistics Claudia Harsch C.Harsch@warwick.ac.uk Thank You for Your Attention

19. Centre for Applied Linguistics Selected References Bachman, L. & Palmer, A. (2010). Language Assessment in Practice. OUP. Cizek, G.J., & Bunch, M.B. (2007). Standard-setting. A guide to establishing and evaluating performance standards on tests. California: Sage Publications Inc. Council of Europe (2001). The Common European Framework of Reference for Languages. Learning Teaching and Assessment. CUP. Online www.coe.int/lang-CEFRwww.coe.int/lang-CEFR Council of Europe (2009). Manual for Relating Language Examinations to the Common European Framework of Reference for Languages (CEFR). Retrieved from www.coe.int/t/dg4/linguistic/Manuel1_EN.aspwww.coe.int/t/dg4/linguistic/Manuel1_EN.asp Concil of Europe (2009). Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR). Further Material on Maintaining Standards across Languages, Contexts and Administrations by exploiting Teacher Judgment and IRT Scaling. Retrieved from www.coe.int/t/dg4/linguistic/Manuel1_EN.aspwww.coe.int/t/dg4/linguistic/Manuel1_EN.asp Douglas, D. (2010). Understanding Language Testing. London: Hodder. Eckes, T. (2012). Examinee-centered standard setting for large-scale assessments: The prototype group method. Psychological Test and Assessment Modeling, 54(3), 257-283. Figueras, N, North, B, Takala, S, Verhelst, N and Van Avermaet, P (2005). Relating Examinations to the Common European Framework: a Manual. Language Testing 22 (3), 1–19. Harsch, C., Pant, H.A. & Köller, O.(Eds.) (2010): Calibrating Standards-based Assessment Tasks for English as a First Foreign Language. Standard-setting Procedures in Germany. Münster: Waxmann.