1. Raili Hildén University of Helsinki Relating the Finnish School Scale to the CEFR

2. Starting point “The construction of a comprehensive, transparent and coherent framework for language learning and teaching does not imply the imposition of one single uniform system. On the contrary, the framework should be open and flexible, so that it can be applied, with such adaptations as prove necessary, to particular situations.” (CEFR, 2001, p. 7) “The construction of a comprehensive, transparent and coherent framework for language learning and teaching does not imply the imposition of one single uniform system. On the contrary, the framework should be open and flexible, so that it can be applied, with such adaptations as prove necessary, to particular situations.” (CEFR, 2001, p. 7)

3. How was the Finnish school scale (FSS) designed? Decision to adapt/adopt the CEFR levels was made by the language experts invited to be members of a national curriculum development working group; approved by the supervising board Sources consulted: CEFR – scales for multiple communicative activities Canadian Benchmarks Teacher judgement Working group members commented on the drafts

4. How was the school scale designed? Several versions were produced based on internal feedback First round of empirical check for inter- rater consistency (by the team members) Re-formulation of descriptors with low agreement/consistency Second round of empirical validation Agreeing on the current formulations

5. Proficiency levels (and their labels) in Finnish language curricula http://www.oph.fi A1 Basic communication in the most familiar situations A1.1 First stage of elementary proficiency A1.2 Developing elementary proficiency A1.3 Functional elementary proficiency A2 Communication in basic social situations and simple description A2.1 First stage of basic proficiency A2.2 Developing basic proficiency B1 Communication in everyday life B1.1 Functional basic proficiency B1.2 Fluent basic proficiency B2 Coping with regular relationships with native speakers B2.1 First stage of independent proficiency B2.2 Functional independent proficiency C1-C2 Proficient language use in demanding contexts C1.1 First stage of skilled proficiency

6. Categories included in the Finnish language curricula Listening comprehension Themes, text and tasks (1) Conditions and constraints (2) Speaking Themes, texts and tasks (monologue and interaction) Fluency Pronunciation Linguistic range Linguistic control Reading comprehension Themes, texts and tasks (2) Conditions and constraints (1) Writing Themes, texts and tasks (2) Linguistic range Linguistic control

7. Research Questions RQ1. What is the level of agreement between judges on the CEFR level of the FSS descriptors? RQ2. How are the Finnish syllabus descriptors related to CEFR levels?

8. Design FSS level descriptors were split up into 184 statements Listening 38, Speaking 66, Reading 31 Writing 49 statements The statements were coded and grouped in terms of communicative activities Criterion scales used in the rating of FSS descriptors were selected from among relevant CEF scales

9. Design A sample of 40 Finnish language teaching experts were contacted by an e-mail questionnaire 20 experts returned the questionnaire A randomised selection of statements referring to each of the four communicative activities was e-mailed to the raters (in Finnish translation by Huttunen & Jaakkola 2003)

10. Conducting the research Listening comprehension a (38 descriptors) Raters Sent to: Returned by: 1. Themes and texts b (13)4020 2. Tasks and activities (14)168 3. Conditions and constraints (11)1510 Notes: a/ CEFR scale used for the rating task- Overall listening comprehension b/ Anchor descriptors, rated by all raters

11. Speaking (66 descriptors) CEFR scaleRaters Sent to:Returned by: 1. Texts, themes and tasks (21) Overall spoken interaction 4020 2. Fluency (10)Spoken fluency146 3. Pronun- ciation (10) Phonological control 158 4.Linguistic range (14) Vocabulary range159 5. Linguistic control (11) Grammatical accuracy 1510

12. Conducting the research Reading comprehension a (31 descriptors) Raters Sent to:Returned by: 1. Themes and texts b (10)4020 2. Tasks (11) a 146 3. Conditions and constraints (10) 147 4. Tasks and activities c 127 Notes: a/ CEFR scale - Overall reading comprehension b/ Anchor descriptors, rated by all raters c/ CEFR scale Reading for information and argument

13. Writing (49 descriptors) CEF scaleRaters Sent to:Returned by: 1. Texts and themes (13) Creative writing4020 2. Tasks and activities (12) Overall written interaction 1510 3. Linguistic range (11) General linguistic range 157 4. Linguistic control (13) Grammatical accuracy 157 5. Texts and themes (12) Overall written production 156

14. FSS level codesCEFR level codes A11 - 1 A12 - 2 A13 - 3 A21 - 4 A22 - 5 B11 - 6 B12 - 7 B21 - 8 B22 - 9 C11 -10 A1 - 1 A2 - 2 A2+ - 3 B1 - 4 B1 + - 5 B2 - 6 B2+ - 7 C1 - 8 C2 - 9

15. RQ1.The range of raters’ agreement Range Note: Ranges of 7 and 8 were checked after the presentation and detected to be due to clerical errors. Thus the ”true” range is From 0 to 6.

16. variance=2,37 FINSS=B21 CEF=6 CEF reference B2 RQ 1. Agreement with the CEF levels Example of a descriptor with a low level of agreement between raters. “Can identify the writer’s bias and the purpose of the text and locate and integrate several specific pieces of information in a longer text. Can quickly identify the content and relevance of new items deciding whether closer study is worthwhile” Level

17. variance=0 FINSS=A11 CEF=1 CEF reference A1 RQ 1. Agreement with the CEF levels Example of a descriptor with a high level of agreement between raters. “Can write the alphabet of the language and all numbers and numerals. Can write down basic personal identification information and write a small number of familiar words and simple phrases.”

18. RQ2. Agreement between syllabus level descriptors (syllcode) and the original CEFR levels (levcode) Note: A quite good level of agreement was oberved (65%). There is, however, some tendency for an overestimation: 19 descriptors (10%) were assigned to a lower level while 46 (25%) were assigned to higher level.

19. RQ1. Range distribution per skill The average range is 2.66 for the whole pool. The range distribution per skill can be seen in the boxplot above, showing no clear differences between the four skills.

20. RQ2. Absolute agreement per skill between syllabus level and CEF level

21. RQ1. Factors affecting the ratings (= raters are quite ”homogeneous”)

22. RQ2. Agreement between individual rating and original (initial) levels (Syllabus - syll & CEF - level)

23. Plans for further exploration Calibrating the FSS descriptors Exploring the link to the Canadian Benchmarks in more detail Re-formulating or removing problematic descriptors Empirical validation and exemplication of the FSS scales through benchmarks for comprehension tasks and for oral and written performance samples

24. Summary The correspondence between the new Finnish school scale (FSS) and the CEFR scales were studied. Research question 1: What is the level of agreement between judges on the CEFR level of the FSS descriptors? Research question 2: How are the Finnish syllabus descriptors related to CEFR levels? 20 experienced raters judged FSS descriptors using relevant CEFR scales A good agreement was reached: 65% of the FSS descriptors were assigned to the original CEFR levels. For the rest of the cdescriptorts, some tendency of overestimation was observed. Inter-rater agreement was also quite good.

25. Acknowledgement I wish to thank –Sauli Takala, University of Jyväskylä –Feljanka Kaftandieva, University of Sofia for their help in carrying out this study.