1. UNIVERSITY OF JYVÄSKYLÄ CEFLING The linguistic basis of the Common European Framework levels: Combining second language acquisition and language testing research

2. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi FUNDING 2007 - 2009

3. UNIVERSITY OF JYVÄSKYLÄ VENUE http://www.jyu.fi/hum/laitokset/kielet/cefling/en http://www.jyu.fi/hum/laitokset/kielet/cefling/en Department of Languages Finnish and English Centre for Applied Language Studies Department of Teacher Education SLATE-network Belgium, France (Univ. Of Paris III), Germany, Italy, The Netherlands, Sweden, UK Summer school 8.-11.6.2009, Jyväskylä

4. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi CEFLINGS RESEARCHERS Riikka Alanen Hannele Dufva Ari Huhta Paula Kalaja Maisa Martin Katja Mäntylä Mirja Tarnanen PROJECT SECRETARIES Laura Hartikainen Riikka Purola DOCTORAL STUDENTS Taru Kynsijärvi Helena Miettinen Sanna Mustonen Nina Reiman Marja Seilonen Henna Tossavainen MA STUDENTS Anni Kettunen Sarianna Kivilahti Leena Nyyssönen Vilja Paavola Nina Pajunen Minna Roiha and others

5. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi RESEARCH TOPIC How does foreign or second language proficiency develop? How could that development be described as stages of achievement? Focus: writing

6. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Research question 1/4 What combinations of linguistic features characterise learners’ performance at the proficiency levels defined in the Common Framework and its Finnish adaptations?

7. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Research question 2/4 To what extent do adult and young learners who engage in the same communicative tasks, at a given level, perform in the same way linguistically? To what extent are the adult-oriented CEFR levels and their Finnish adaptations for young learners equivalent?

8. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Research question 3/4 To what extent are the pedagogical tasks found in the teaching materials in the Finnish comprehensive school comparable with the tasks defined in the CEFR and the new curriculum?

9. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Research question 4/4 What are the linguistic and communicative features that teachers (or National Certificates raters) pay attention to when assessing learners with the help of the Finnish adaptations of the CEFR scales? How do these features relate to the linguistic and communicative analysis of the same performances?

10. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi RATIONALE CEF very influential in Finland: - school curricula - adult education curricula - National Certificates of Proficiency - citizenship requirements

11. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi SUBJECTS AND LANGUAGES Written performances of adults taking the National Certificate of Proficiency examination (3 texts per subject) Similar texts on similar tasks from young learners (grades 7 – 9, ages 13-16) L2 Finnish – L1 varies L2 English – L1 Finnish or Swedish (possibly native speaker control groups)

12. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi RATING OF THE DATA three – four trained raters per writing sample inclusion for main data: complete interrater agreement or two in agreement, one + or – one level Problems with certain levels –(not enough A1 for adults and C1 and C2 for young writers)

13. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi CODING OF THE DATA CHILDES (.chat format, CLAN tools) Basic coding for all data, structural features as needed (several for Finnish, fewer for English at the moment) Automatic analyses for English only

14. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi

15. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi

16. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi RQ ¼ : What combinations of linguistic features characterise learners’ performance at each level Finnish: sentence types extension of local case use development of infinitive phrases development of negation and morphological marking Finnish vs. English: formulaic sequences cohesive features English: questions negative forms relative clauses agreement collocations derivative skills (word- formation)

17. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi RQ 2/4 To what extent do adult and young learners perform in the same way linguistically? /To what extent are the adult- oriented CEFR levels and their Finnish adaptations for young learners equivalent? Raters use both scales in separate rounds of rating. Results are statistically compared and raters interviewed. CEFR scale A1-C2 Finnish adaptation A1.1- C1.1 (more linguistically oriented)

18. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi RQ ¾ To what extent are the pedagogical tasks found in the teaching materials in the Finnish comprehensive school comparable with the tasks defined in the CEFR and the new curriculum? Textbooks and other teaching materials are analysed both for tasks and for the progression of structural skills.

19. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi RQ 4/4 What are the linguistic and communicative features that teachers/trained raters pay attention to when assessing learners/ How do these features relate to the linguistic and communicative analysis of the same performances? Teachers and trained raters are systematically interviewed re. their arguments for certain level, for decision making between levels etc.

20. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Plans for the future A longitudinal study to complement the cross-sectional data http://www.jyu.fi/hum/laitokset/kielet/kesakoulu2009 Language acquisition and assessment 8-11June, 2009 Plenary speakers Charles Alderson, Lancaster University, UK James Milton, Swansea University, UK. Gabriele Pallotti, University of Modena and Reggio Emilia, Italy Rob Schoonen, University of Amsterdam, The Netherlands Ineke Vedder, University of Amsterdam, The Netherlands

21. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi An example of what we’ve done Assessing lexical awareness: EFL learners and English word-formation (together with Ari Huhta, ari.huhta@campus.jyu.fi)ari.huhta@campus.jyu.fi How to test word formation skills? How do the three methods used in the study function? What is the relationship between the word-formation skills and overall written proficiency of Finnish school pupils ? (What kind of knowledge do they have on English word- formation?) (How do their word-formation skills develop?) (Is there any difference between Finnish and Swedish speaking participants?)

22. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Word-formation and SLA Word-formation and SLA in general Role in teaching English in a Finnish school? Derivation chosen because Productivity of the method The participants familiar with it at least implicitly

23. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Participants 7th graders 13-14 –year-olds, have studied English as a FL for at least 4 years 162 completed the word formation tests, 87 of whom also completed several writing tasks (next step: 8th and 9th graders) Different parts of Finland

24. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Word-formation test 1 Three written word-formation tests (revised after piloting) 1. Sentences / sentence pairs in English with a Finnish translation of the target word (productive gap-filling): –I am ________ (varma) that he will get the job in London. –He will _________ (varmasti) get the job in London. –  sure - surely

25. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Word-formation test 2 (final version) 2. Sentences with non-words with explanations in Finnish (gap-filling): Example in Finnish (with Finnish real words) –She could bourble animals very well because she was a good ____ bourble____. (henkilö, joka tekee lihavoidun sanan kuvaamaa toimintaa/työtä) –(a person who does the action described by the bolded word)

26. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Word-formation test 3 –3. A list of prefixes from which the participants were to choose suitable ones to fill in the gaps in sentences –He did not follow the instructions. He had – ___ understood them. anti- de- dis- in- im- il- ir- inter- intra- mega- mini- mis- mono- neo- non- poly- post- pre- pro- re- trans- un-

27. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Writing tasks  Email to a friend  Email to one’s teacher  Email to a store  Opinion piece  Narrative piece

28. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Marking word-formation tests Double marking Scoring: –Productive gap-fill test : 0-1-2-3-4 Non-words based test and List-choice based test : 0-1-2 The respondents were very creative: minigabl Spelling errors more or less ignored in scoring: unbelievubl,unbelievevabl, unbelievobl; understant, anderstand (cf. shore  sure, deffreno  different)

29. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Analysis of the word formation tests TiaPlus programme (CITO, the Netherlands) – for classical item analyses – for norm-referenced tests

30. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Characteristics of the 3 tests ItemsMean score (percent) Standard Deviat- ion Std. Error of Mean Cron- bach’s Alpha Alpha for 40-item test Average item /total correlation Productive gap-fill test 1970%21.01.6.87.93.55 Non-words based test 831.5% (27%) 23.6 (24.4) 2.0 (1.9).70 (.74).92 (.93).58 (.61) List-choice based test 1236%20.61.6.74.90.51 All 3 tests together 3951%.90.91(.47)

31. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Correlation between word-formation tests and the rating of writing skill WRITING SKILL (on CEFR scale) Mean rating across 4 raters and all tasks completed by student Productive gap-fill test.691 - items tapping the base form of word (9).578 (.589, if item A17 removed) - items tapping the inflected form of word (10).713 Non-words based test.575 (n = 76) List-based test.675 n = 87 p =.000

32. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Mean word-formation test scores (%) across CEFR levels (based on writing) A1 (n= 27) A2 (n= 42) B1 (n= 15) B2 (n= 2) Productive gap-fill test 467587100 - items tapping the base form of word (9) 577884100 - items tapping the inflected form of word (10) 377390100 Non-words based test 13283881 List-based test 15324771

33. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Correlations between word-formation tests N = 140 / 162A (total) Productive gap-fill test A1. items tapping the base form A2. items tapping the inflected form B. Non- words based test C. List- based test A. Productive gap-fill test 1.00 (.902)(.966).596.615 A1. Items tapping the base form 1.00.765.497.554 A2. Items tapping the inflected form 1.00.555.596 B. Non-words based test 1.00.628 C. List-based test 1.00

34. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Comparison of the three word- formation test methods ProsCons Productive gap-fill testvery familiar test typememorising words? marking difficult (if very fine-tuned) the relationship between items Non-words based testfocus only on word- formation relative difficulty unfamiliar test type List-based testquick and easy to take and mark focus on word-formation (relatively) memorising words? unfamiliar test type? difficult to write easy items?

35. UNIVERSITY OF JYVÄSKYLÄ 13 February, 2009kmantyla@campus.jyu.fi Overall conclusions about the word- formation tests Productive gap-fill: possibly useful for testing word-formation but more work needed to develop / select suitable items Non-words test: promising (e.g. as reliable as the others; construct relevant) but there is a ’threshold’ to overcome for the test-taker, i.e. to understand what it is about List-based gap-fill: promising (fairly construct relevant), but the difficulty of the words need to suit students’ level better Which level of affixes/words? Relationship with the word frequency?