L2 learners’ performance across L2 writing tasks: comparing tasks and language proficiency through CEFR scales Riikka Alanen riikka.a.alanen@jyu.fi Ari Huhta ari.huhta@campus.jyu.fi Centre for Applied Language Studies University of Jyväskylä, Finland

Background CEFLING (http://www.jyu.fi/cefling ), funded by the Academy of Finland (2007-2009) Part of the SLATE network  Integrating SLA and language testing research perspectives CEFLING: Language testing and assessment: Ari Huhta, Mirja Tarnanen + others from CALS; SLA: Maisa Martin (project leader) L2 Finnish, Riikka Alanen, Florencia Franceschina, Paula Kalaja, Katja Mäntylä L2 English. Aim of the project is to describe the linguistic features of L2 Finnish and L2 English at various levels of language proficiency as described by the scales of the Common European Framework (CEFR).

Task in CEFLING Operational definition adopted by CEFLING: “an activity which requires learners to use language, with emphasis on meaning, to attain an objective” (Bygate, Skehan & Swain, 2001, p.11). A unit of analysis whereby we can analyze and compare data on a number of social and cognitive aspects involved in language learning  In short, our talk is about analysing our unit of analysis  What kind of L2 performance – in terms of CEFR proficiency scales -- did the tasks used in CEFLING measure?

Tasks in CEFLING A set of communicative tasks were used to collect written L2 data from 7th to 9th graders (aged 12-16) studying English and Finnish as L2 at school Tasks carefully designed (see Alanen, Huhta, Tarnanen in preparation) (1) Communicative L2 writing tasks: functions, text types and register taken into account: From informal and formal email messages (a complaint to an Internet company) to argumentative and narrative texts. (2) Proficiency level: Aimed at A1 to B2 level Tasks piloted In the end, pupils asked to do five of the following tasks: Tasks 1 and 2 were alternatives, so that pupils either did Task 1, 3 - 6 or Task 2, 3 - 6.

Tasks in CEFLING Task 1: Informal email message to a friend Task 2: Informal email message to the teacher Task 3: Formal message to an Internet store Task 4: Opinion Task 5: Story (Task 6: Translation of a letter from Finnish into English for the grandmother) Excluded from the study

What kind of L2 performance did the tasks used in CEFLING measure? Q1. How do CEFLING L2 tasks compare with L2 teaching and testing materials used in Finland? Q2. What was the raters’ perception of the CEFR level of the tasks before rating? Q3. How did L2 students perform on CEFR proficiency scales across various task types? Is there any systematic variation in students’ performance that is task-related?

Q1. How do CEFLING L2 tasks compare with L2 teaching and testing materials used in Finland? Both textbooks and the National Certificate examination system used task types that involved messages (cf. Tasks 1 -3) Stylistic variation: informal – formal Functional variation: greeting – inquiry -- complaint – reclamation etc. in the NC, the informal and functionally less demanding types, were mostly found at the Basic level (i.e., intended for A1 and A2 learners) Both textbooks and the National Certificate examination system used task types that involved expressing opinion (cf. Task 4) In the NC, opinion pieces were usually -- although not exclusively -- intended for Intermediate and Advanced learners (B1 – C2 learners) Only textbooks used task types that involved telling a story (narrative) (cf. Task 5)

Rating Tasks and Scripts At the same time, the assessment criteria and procedures necessary for the reliable and valid rating were prepared by Ari Huhta and Mirja Tarnanen A team of raters for L2 Finnish and L2 English was selected and trained for the project Assessment criteria: CEFR scales for written interaction, correspondence and notes, messages and forms, creative writing and thematic development, and overall written production. Benchmarks: From the pilot performances; Benchmarks published by the Council of Europe for L2 English (no such benchmarks exist for Finnish) Before rating the scripts, the raters were asked for their perceptions of the CEFR level of the tasks  In this talk, this is referred to as Perceived Task Difficulty for short

Collecting and Analysing the Scripts 3427 performances were collected from 7th to 9th graders (aged 12-16) from schools; 1789 scripts were selected for assessment by a team of trained raters (N=9 for English, N = 11 for Finnish). Each script was rated by four (L2 English) or three (L2 Finnish) raters chosen from the team Only those scripts were selected for the analysis of linguistic features on which the raters were in sufficient agreement (three out of four raters for L2 English, two out of three for L2 Finnish).  Final corpus for L2 English contained 562 scripts and for L2 Finnish 825 scripts. CHAT & CLAN (CHILDES) http://childes.psy.cmu.edu/ adapted for transcription and preliminary coding

Task by Task Analysis Q2. What was the raters’ perception of the CEFR level of the tasks before rating? Raters were asked which CEFR level the task seemed best suited for; they were also asked to indicate the second best level as well as the lowest and highest potential level  only the raters’ perceptions about the most suitable level will be reported here Q3. How did L2 students perform on CEFR proficiency scales across various task types, i.e., how did the raters rate the scripts? Is there any systematic variation in students’ performance that is task-related?

Good to Know Before Task by Task Analysis National Core Curriculum Objectives for L2 English Writing By the end of 6th grade: A1.3 Functional elementary language proficiency for ”good performance” By the end of 9th grade: A2.2 Developing basic language proficiency for ”good performance” A2 Table 1a. National Core Curriculum Objectives for L2 English Writing National Core Curriculum Objectives for L2 Finnish By the end of 6th grade: Not set By the end of 9th grade: B1.1 to B1.2 Functional or fluent basic proficiency (Global proficiency -- not set separately for L2 writing) Figure 1. Median of (median) Ratings for L2 English Table 1b. National Core Curriculum Objectives for L2 Finnish

Rubrics L2 English and L2 Finnish tasks very similar although not identical Language of instructions: L1 or L2 Finnish Slight differences in wording e.g in Task 2: No mention of English teacher to L2 Finnish learners (instead only ”teacher”) Task 3: syntax not as complex in L2 Finnish task Task 4: the topics of the opinion pieces slightly different and in Finnish for L2 Finnish learners Both L2 Finnish and English learners did two tasks in 2 45 minute lessons (plus Task 6, some other tasks for L2 English), specified for English 10-15 minutes per communicative task

Task 1 Note to a friend You’ve set up a meeting with your English-speaking friend at a café. However, something has come up and you have other things to do. Send an email message to your friend. Explain why you can’t come. Suggest a new time and place. Remember to begin and end the message in appropriately.

Perceived Difficulty Level and L2 English Learner Performance for Task 1

Perceived Difficulty Level and L2 Finnish Learner Performance for Task 1

Task 2 Message to your teacher You’ve been away from school for a week. Soon you’ll have an English exam. Your teacher, Mary Brown, speaks only English. Send an email message to the teacher. Tell her why you’ve been away. Ask two things about the exam. Ask two things about the English lessons that were held during the week. Remember to begin and end the message appropriately.

Perceived Difficulty Level and L2 English Learner Performance for Task 2

Perceived Difficulty Level and L2 Finnish Learner Performance for Task 2

Task 3 Message to an internet store Your parents have ordered a PC game for you from a British internet store. When you get the game you notice that it doesn’t work properly. You get upset and decide to write an email message to the internet store. In the message, say who you are what your parents ordered why you’re unhappy (mention at least two defects/problems) how you would like them to take care of the matter give your contact information Remember to begin and end the message appropriately.

Perceived Difficulty Level and L2 English Learner Performance for Task 3

Perceived Difficulty Level and L2 Finnish Learner Performance for Task 3

Task 4 Opinion Choose one of the topics and write about what you think about the matter. Give reasons for your opinion. 1. Boys and girls should go to different classes at school. 2. No mobile phones at school!

Perceived Difficulty Level and L2 English Learner Performance for Task 4

Perceived Difficulty Level and L2 Finnish Learner Performance for Task 4

Task 5 Narrative Tell about the scariest / funniest / greatest experience in your life. Choose one. Tell what happened (what, where, when, and so on). Tell why the experience was scary / funny / great. Write in English in clear characters in the space below (continues on the reverse side).

Perceived Difficulty Level and L2 English Learner Performance for Task 5

Perceived Difficulty Level and L2 Finnish Learner Performance for Task 5

Results of Rasch Analyses FACETS used to run Rasch analyses: Ordinal data transformed into interval data Takes into account L2 learners’ proficiency and the relative toughness of their raters  Makes possible to analyse task difficulty

Rasch Analyses Measure L2 English Task .38 .00 .02 -.10 -.27 Task 3 Table 2a. Order of Task Difficulty for L2 English Table 2b. Order of Task Difficulty for L2 Finnish Measure L2 English Task .38 .00 .02 -.10 -.27 Task 3 Task 5 Task1 Task2 Task4 Measure L2 Finnish Task .29 .25 .20 -.29 -.46 Task 2 Task 1 Task 3 Task 5 Task 4 Zero stands for the center of the range of item difficulties (default origin) Order of task difficulty in L2 English and L2 Finnish, from the easiest down For L2 English, there was .65 logit points difference between the tasks, with Task 3 being the easiest For L2 Finnish, there was .75 logit points difference between the tasks, with Task 2 being the easiest

Summary L2 English Task 4 was the most difficult, Task 3 easiest Tasks fall within half a level of each other, both sides of A2 and B1 on CEFR scale Task 4 was the most difficult, Task 3 easiest L2 Finnish Tasks fall within half a level of each other, within A2 just under B1 on CEFR scale Task 4 was again the most difficult, Task 2 easiest  Task 4 was quite difficult as expected for both languages

Discussion Task 3 was easier than expected Perceived as a B1 task in both languages Learner performance was a solid A2 Slightly skewed towards A1 for L2 English Slightly skewed towards B1 for L2 Finnish Task 2 slightly more difficult for L2 English Perceived as a B1 task in both languages, but: In L2 English, 34.6% of the scripts were A1 (cf. Task 4: 28.5% of the scripts were A1)  greater number of A1 performances may explain the result? In L2 Finnish, 44% of the scripts were A2 (31.7% were B1)

Conclusion Tasks were designed to elicit data from A1 to B2 learners Tasks appeared suitable for the target group which ranged mostly from A1 to B1 Tasks were capable of eliciting performances from CEFR levels ranging from A0-C1 (depending on the task) FACETS used to assign difficulty levels to tasks  some variation among the tasks but not much

Further Study Needed Task 2 : why did L2 English learners perform lower than expected on this task? Unlike the other tasks, in Task 2, pupils were asked to make at least four questions  Task complexity? Question formation in L2 English linked to the mastery of certain grammatical structures (like fronting and inversion), while in L1 Finnish, Stage 5 English questions follow SVO pattern (with wh-fronting) : Mistä hän tulee? [lit.] ’Where+from he comes?’

Further Study Needed In a separate study (Alanen & Kalaja 2009), we analysed L2 English question formation across proficiency levels and task types in CEFLING L2 English data Task 2 elicited a much greater number of questions than other tasks (hardly surprising) Overall, tasks elicited questions from all stages across proficiency levels A1 to B1, but B1 scripts contained hardly any non target-like, ”incorrect”, Stage 3 questions like *where we go now?, unlike A1 and A2 scripts  B1 scripts characterized by the absence of non target-like questions from a particular stage of development (i.e., lack of errors)? Other features must have played a role as well in task assessment → closer analysis needed

Further Study Needed  Do tasks which require L2 learners to ask questions place different demands on them depending on their L2 and L1? – e.g. is asking questions in L2 Finnish less complex than in L2 English?  We need more research on the relationship between task difficulty (task features) and test takers/learners’ characteristics For example, interview studies, think aloud protocol studies on how A1 learners experience a B2 task or vice versa

References Alanen, R. & Kalaja, P. (2009). ”Where the teacher get the phones?”– The Emergence of L2 English Questions in the Communicative Tasks Written by L1 Finnish Learners. Paper presented at BAAL 2009, Newcastle, 3-5 September 2009. Alanen, R., Huhta, A. & Tarnanen, M. (MS in preparation) Developing tasks and CEFR-based assessment procedures for SLA research purposes. Bygate, M., Skehan, P. & Swain, M. (2001). Introduction. In M. Bygate, P. Skehan & M. Swain (Eds.), Researching pedagogic tasks. Second langauge learning, teaching and testing (pp. 1-20). Harlow, England: Longman/Pearson Education.