1. UNIVERSITY OF JYVÄSKYLÄ L2 learners performance across L2 writing tasks: comparing tasks and language proficiency through CEFR scales Riikka Alanen riikka.a.alanen@jyu.fi Ari Huhta ari.huhta@campus.jyu.fi Centre for Applied Language Studies University of Jyväskylä, Finland

2. UNIVERSITY OF JYVÄSKYLÄ 2 Background CEFLING (http://www.jyu.fi/cefling ), funded by the Academy of Finland (2007-2009)http://www.jyu.fi/cefling Part of the SLATE network Integrating SLA and language testing research perspectives CEFLING: Language testing and assessment: Ari Huhta, Mirja Tarnanen + others from CALS; SLA: Maisa Martin (project leader) L2 Finnish, Riikka Alanen, Florencia Franceschina, Paula Kalaja, Katja Mäntylä L2 English. Aim of the project is to describe the linguistic features of L2 Finnish and L2 English at various levels of language proficiency as described by the scales of the Common European Framework (CEFR).

3. UNIVERSITY OF JYVÄSKYLÄ 3 Task in CEFLING Operational definition adopted by CEFLING: –an activity which requires learners to use language, with emphasis on meaning, to attain an objective (Bygate, Skehan & Swain, 2001, p.11). A unit of analysis whereby we can analyze and compare data on a number of social and cognitive aspects involved in language learning In short, our talk is about analysing our unit of analysis What kind of L2 performance – in terms of CEFR proficiency scales -- did the tasks used in CEFLING measure?

4. UNIVERSITY OF JYVÄSKYLÄ 4 Tasks in CEFLING A set of communicative tasks were used to collect written L2 data from 7 th to 9 th graders (aged 12-16) studying English and Finnish as L2 at school –Tasks carefully designed (see Alanen, Huhta, Tarnanen in preparation) (1) Communicative L2 writing tasks: functions, text types and register taken into account: –From informal and formal email messages (a complaint to an Internet company) to argumentative and narrative texts. (2) Proficiency level: Aimed at A1 to B2 level Tasks piloted In the end, pupils asked to do five of the following tasks: Tasks 1 and 2 were alternatives, so that pupils either did Task 1, 3 - 6 or Task 2, 3 - 6.

5. UNIVERSITY OF JYVÄSKYLÄ 5 Tasks in CEFLING Task 1: Informal email message to a friend Task 2: Informal email message to the teacher Task 3: Formal message to an Internet store Task 4: Opinion Task 5: Story (Task 6: Translation of a letter from Finnish into English for the grandmother) –Excluded from the study

6. UNIVERSITY OF JYVÄSKYLÄ 6 What kind of L2 performance did the tasks used in CEFLING measure? Q1. How do CEFLING L2 tasks compare with L2 teaching and testing materials used in Finland? Q2. What was the raters perception of the CEFR level of the tasks before rating? Q3. How did L2 students perform on CEFR proficiency scales across various task types? Is there any systematic variation in students performance that is task-related?

7. UNIVERSITY OF JYVÄSKYLÄ 7 Q1. How do CEFLING L2 tasks compare with L2 teaching and testing materials used in Finland? Both textbooks and the National Certificate examination system used task types that involved messages (cf. Tasks 1 -3) –Stylistic variation: informal – formal –Functional variation: greeting – inquiry -- complaint – reclamation etc. –in the NC, the informal and functionally less demanding types, were mostly found at the Basic level (i.e., intended for A1 and A2 learners) Both textbooks and the National Certificate examination system used task types that involved expressing opinion (cf. Task 4) –In the NC, opinion pieces were usually -- although not exclusively -- intended for Intermediate and Advanced learners (B1 – C2 learners) Only textbooks used task types that involved telling a story (narrative) (cf. Task 5)

8. UNIVERSITY OF JYVÄSKYLÄ 8 Rating Tasks and Scripts At the same time, the assessment criteria and procedures necessary for the reliable and valid rating were prepared by Ari Huhta and Mirja Tarnanen A team of raters for L2 Finnish and L2 English was selected and trained for the project –Assessment criteria: CEFR scales for written interaction, correspondence and notes, messages and forms, creative writing and thematic development, and overall written production. –Benchmarks: From the pilot performances; Benchmarks published by the Council of Europe for L2 English (no such benchmarks exist for Finnish) Before rating the scripts, the raters were asked for their perceptions of the CEFR level of the tasks In this talk, this is referred to as Perceived Task Difficulty for short

9. UNIVERSITY OF JYVÄSKYLÄ 9 Collecting and Analysing the Scripts 3427 performances were collected from 7 th to 9 th graders (aged 12-16) from schools; 1789 scripts were selected for assessment by a team of trained raters (N=9 for English, N = 11 for Finnish). –Each script was rated by four (L2 English) or three (L2 Finnish) raters chosen from the team Only those scripts were selected for the analysis of linguistic features on which the raters were in sufficient agreement (three out of four raters for L2 English, two out of three for L2 Finnish). Final corpus for L2 English contained 562 scripts and for L2 Finnish 825 scripts. CHAT & CLAN (CHILDES) http://childes.psy.cmu.edu/ adapted for transcription and preliminary codinghttp://childes.psy.cmu.edu/

10. UNIVERSITY OF JYVÄSKYLÄ 10 Task by Task Analysis Q2. What was the raters perception of the CEFR level of the tasks before rating? –Raters were asked which CEFR level the task seemed best suited for; they were also asked to indicate the second best level as well as the lowest and highest potential level only the raters perceptions about the most suitable level will be reported here Q3. How did L2 students perform on CEFR proficiency scales across various task types, i.e., how did the raters rate the scripts? Is there any systematic variation in students performance that is task-related?

11. UNIVERSITY OF JYVÄSKYLÄ 11 Good to Know Before Task by Task Analysis National Core Curriculum Objectives for L2 English Writing By the end of 6th grade: A1.3 Functional elementary language proficiency for good performance By the end of 9th grade: A2.2 Developing basic language proficiency for good performance National Core Curriculum Objectives for L2 Finnish By the end of 6th grade: Not set By the end of 9th grade: B1.1 to B1.2 Functional or fluent basic proficiency (Global proficiency -- not set separately for L2 writing) A2 Figure 1. Median of (median) Ratings for L2 English Table 1a. National Core Curriculum Objectives for L2 English Writing Table 1b. National Core Curriculum Objectives for L2 Finnish

12. UNIVERSITY OF JYVÄSKYLÄ 12 Rubrics L2 English and L2 Finnish tasks very similar although not identical Language of instructions: L1 or L2 Finnish Slight differences in wording e.g in – Task 2: No mention of English teacher to L2 Finnish learners (instead only teacher) – Task 3: syntax not as complex in L2 Finnish task – Task 4: the topics of the opinion pieces slightly different and in Finnish for L2 Finnish learners Both L2 Finnish and English learners did two tasks in 2 45 minute lessons (plus Task 6, some other tasks for L2 English), specified for English 10-15 minutes per communicative task

13. UNIVERSITY OF JYVÄSKYLÄ 13 Task 1 Note to a friend Youve set up a meeting with your English-speaking friend at a café. However, something has come up and you have other things to do. Send an email message to your friend. Explain why you cant come. Suggest a new time and place. Remember to begin and end the message in appropriately.

14. UNIVERSITY OF JYVÄSKYLÄ 14 Perceived Difficulty Level and L2 English Learner Performance for Task 1

15. UNIVERSITY OF JYVÄSKYLÄ 15 Perceived Difficulty Level and L2 Finnish Learner Performance for Task 1

16. UNIVERSITY OF JYVÄSKYLÄ 16 Task 2 Message to your teacher Youve been away from school for a week. Soon youll have an English exam. Your teacher, Mary Brown, speaks only English. Send an email message to the teacher. Tell her why youve been away. Ask two things about the exam. Ask two things about the English lessons that were held during the week. Remember to begin and end the message appropriately.

17. UNIVERSITY OF JYVÄSKYLÄ 17 Perceived Difficulty Level and L2 English Learner Performance for Task 2

18. UNIVERSITY OF JYVÄSKYLÄ 18 Perceived Difficulty Level and L2 Finnish Learner Performance for Task 2

19. UNIVERSITY OF JYVÄSKYLÄ 19 Task 3 Message to an internet store Your parents have ordered a PC game for you from a British internet store. When you get the game you notice that it doesnt work properly. You get upset and decide to write an email message to the internet store. In the message, say who you are what your parents ordered why youre unhappy (mention at least two defects/problems) how you would like them to take care of the matter give your contact information Remember to begin and end the message appropriately.

20. UNIVERSITY OF JYVÄSKYLÄ 20 Perceived Difficulty Level and L2 English Learner Performance for Task 3

21. UNIVERSITY OF JYVÄSKYLÄ 21 Perceived Difficulty Level and L2 Finnish Learner Performance for Task 3

22. UNIVERSITY OF JYVÄSKYLÄ 22 Task 4 Opinion Choose one of the topics and write about what you think about the matter. Give reasons for your opinion. 1. Boys and girls should go to different classes at school. 2. No mobile phones at school!

23. UNIVERSITY OF JYVÄSKYLÄ 23 Perceived Difficulty Level and L2 English Learner Performance for Task 4

24. UNIVERSITY OF JYVÄSKYLÄ 24 Perceived Difficulty Level and L2 Finnish Learner Performance for Task 4

25. UNIVERSITY OF JYVÄSKYLÄ 25 Task 5 Narrative Tell about the scariest / funniest / greatest experience in your life. Choose one. Tell what happened (what, where, when, and so on). Tell why the experience was scary / funny / great. Write in English in clear characters in the space below (continues on the reverse side).

26. UNIVERSITY OF JYVÄSKYLÄ 26 Perceived Difficulty Level and L2 English Learner Performance for Task 5

27. UNIVERSITY OF JYVÄSKYLÄ 27 Perceived Difficulty Level and L2 Finnish Learner Performance for Task 5

28. UNIVERSITY OF JYVÄSKYLÄ 28 Results of Rasch Analyses FACETS used to run Rasch analyses: –Ordinal data transformed into interval data –Takes into account L2 learners proficiency and the relative toughness of their raters Makes possible to analyse task difficulty

29. UNIVERSITY OF JYVÄSKYLÄ 29 Rasch Analyses Zero stands for the center of the range of item difficulties (default origin) Order of task difficulty in L2 English and L2 Finnish, from the easiest down For L2 English, there was.65 logit points difference between the tasks, with Task 3 being the easiest For L2 Finnish, there was.75 logit points difference between the tasks, with Task 2 being the easiest MeasureL2 Finnish Task.29.25.20 -.29 -.46 Task 2 Task 1 Task 3 Task 5 Task 4 MeasureL2 English Task.38.00.02 -.10 -.27 Task 3 Task 5 Task1 Task2 Task4 Table 2a. Order of Task Difficulty for L2 English Table 2b. Order of Task Difficulty for L2 Finnish

30. UNIVERSITY OF JYVÄSKYLÄ 30 Summary L2 English –Tasks fall within half a level of each other, both sides of A2 and B1 on CEFR scale Task 4 was the most difficult, Task 3 easiest L2 Finnish –Tasks fall within half a level of each other, within A2 just under B1 on CEFR scale Task 4 was again the most difficult, Task 2 easiest Task 4 was quite difficult as expected for both languages

31. UNIVERSITY OF JYVÄSKYLÄ 31 Discussion Task 3 was easier than expected –Perceived as a B1 task in both languages –Learner performance was a solid A2 Slightly skewed towards A1 for L2 English Slightly skewed towards B1 for L2 Finnish Task 2 slightly more difficult for L2 English –Perceived as a B1 task in both languages, but: –In L2 English, 34.6% of the scripts were A1 (cf. Task 4: 28.5% of the scripts were A1) greater number of A1 performances may explain the result? –In L2 Finnish, 44% of the scripts were A2 (31.7% were B1)

32. UNIVERSITY OF JYVÄSKYLÄ 32 Conclusion Tasks were designed to elicit data from A1 to B2 learners –Tasks appeared suitable for the target group which ranged mostly from A1 to B1 –Tasks were capable of eliciting performances from CEFR levels ranging from A0-C1 (depending on the task) FACETS used to assign difficulty levels to tasks some variation among the tasks but not much

33. UNIVERSITY OF JYVÄSKYLÄ 33 Further Study Needed Task 2 : why did L2 English learners perform lower than expected on this task? Unlike the other tasks, in Task 2, pupils were asked to make at least four questions Task complexity? Question formation in L2 English linked to the mastery of certain grammatical structures (like fronting and inversion), while in L1 Finnish, Stage 5 English questions follow SVO pattern (with wh-fronting) : Mistä hän tulee? [lit.] Where+from he comes?

34. UNIVERSITY OF JYVÄSKYLÄ 34 Further Study Needed In a separate study (Alanen & Kalaja 2009), we analysed L2 English question formation across proficiency levels and task types in CEFLING L2 English data –Task 2 elicited a much greater number of questions than other tasks (hardly surprising) Overall, tasks elicited questions from all stages across proficiency levels A1 to B1, but B1 scripts contained hardly any non target-like, incorrect, Stage 3 questions like *where we go now?, unlike A1 and A2 scripts B1 scripts characterized by the absence of non target-like questions from a particular stage of development (i.e., lack of errors)? –Other features must have played a role as well in task assessment closer analysis needed

35. UNIVERSITY OF JYVÄSKYLÄ 35 Further Study Needed Do tasks which require L2 learners to ask questions place different demands on them depending on their L2 and L1? – e.g. is asking questions in L2 Finnish less complex than in L2 English? We need more research on the relationship between task difficulty (task features) and test takers/learners characteristics –For example, interview studies, think aloud protocol studies on how A1 learners experience a B2 task or vice versa

36. UNIVERSITY OF JYVÄSKYLÄ References Alanen, R. & Kalaja, P. (2009). Where the teacher get the phones?– The Emergence of L2 English Questions in the Communicative Tasks Written by L1 Finnish Learners. Paper presented at BAAL 2009, Newcastle, 3-5 September 2009. Alanen, R., Huhta, A. & Tarnanen, M. (MS in preparation) Developing tasks and CEFR-based assessment procedures for SLA research purposes. Bygate, M., Skehan, P. & Swain, M. (2001). Introduction. In M. Bygate, P. Skehan & M. Swain (Eds.), Researching pedagogic tasks. Second langauge learning, teaching and testing (pp. 1-20). Harlow, England: Longman/Pearson Education.