L2 communicative levels meet linguistic profiling Plenary paper presented at the Estonian Association of Applied Linguistics Conference, Tallinn, Estonia, April 2008 Florencia Franceschina

1. Introduction

Question 1 What stages do L2 learners go through as they develop their L2 knowledge and skills? Question 2 What are the linguistic indicators that a learner is at a particular stage on his/her way to becoming a proficient L2 speaker?

Increasing L2 proficiency can be thought of in terms of: a)Increasing ability to do things with language (eg: write notes, ask the time, talk about your family, understand announcements at a railway station, etc.) b)Increasing linguistic knowledge (eg: larger vocabulary, greater range of known syntactic structures, higher level of grammatical accuracy, etc.) c)Increasing ability to use linguistic knowledge in real time (eg: fluent spontaneous speech, good listening or reading comprehension, etc.)

L2 proficiency scales L2 proficiency scales used in language teaching and testing tend to focus on a mixture of these levels. The original CEFR scales have little information about linguistic knowledge typical of each level (among other reasons, because they have been designed to be language- neutral).

CEFR global scale C2 Can understand with ease virtually everything heard or read. Can summarise information from different spoken and written sources, reconstructing arguments and accounts in a coherent presentation. Can express him/herself spontaneously, very fluently and precisely, differentiating finer shades of meaning even in more complex situations. C1… B2… B1… A2… A1 Can understand and use familiar everyday expressions and very basic phrases aimed at the satisfaction of needs of a concrete type. Can introduce him/herself and others. Can ask and answer questions about personal details such as where he/she lives, people he/she knows and things he/she has. Can interact in a simple way provided the other person talks slowly and clearly and is prepared to help.

CEFR general linguistic range illustrative scale C2… C1… B2… B1… A2 Can produce brief everyday expressions in order to satisfy simple needs of a concrete type: personal details, daily routines, wants and needs, requests for information. Can use basic sentence patterns and communicate with memorised phrases, groups of a few words and formulae about themselves and other people, what they do, places, possessions etc. Has a limited repertoire of short memorised phrases covering predictable survival situations; frequent breakdowns and misunderstandings occur in non-routine situations. A1

CEFR grammatical accuracy illustrative scale C2… C1… B2… B1 Uses reasonably accurately a repertoire of frequently used ‘routines’ and patterns associated with more predictable situations. A2 Uses some simple structures correctly, but still systematically makes basic mistakes – for example tends to mix up tenses and forget to mark agreement; nevertheless, it is usually clear what he/she is trying to say. A1

Finnish National Curriculum scales for L2s/FLs C1.1 … B2.2 … B2.1 … B1.2 … B1.1 … A2.2 … A2.1 Can manage in the most routine everyday situations in writing. Can write brief, simple messages (personal letters, notes), which are related to everyday needs, and simple, enumerated descriptions of very familiar topics (real or imaginary people, events, personal or family plans). Can use concrete vocabulary related to basic needs, basic tenses and co ordinate sentences joined by simple connectors (and, but). Can write the most simple words and structures with reasonable accuracy, but makes frequent basic errors (tenses, inflection) and uses many awkward expressions in free writing. A1.3 … A1.2 … A1.1 …

Our challenge To find reliable linguistic indicators that a learner is at a particular stage on his/her way to becoming a proficient L2 speaker when proficiency level has been established using communicative scales such as the CEFR scales.

This presentation Some answers to Q1 and Q2 we can glean from the existing literature How CEFLING are going about trying to find full answers to these questions –Selection and management of relevant datasets (learner corpora) –Methodology for tracking linguistic indicators in the data (DEMfad) –General methodological suggestions

2. What we already know

SLA generalizations about L2 development 1. Factors such as type and amount of L2 instruction, type and amount of naturalistic exposure to the L2, language aptitude, age, L1 background, among others, are likely to have an effect on the rate of L2 development but do not seem to affect the route of L2 development in a dramatic way.

Dulay and Burt (1974)

*Dulay and Burt (1974) ** Bailey et al. (1974)

Dulay and Burt (1973)

Allwood (1993)

Banerjee et al. (2007)

2. There is a set of features F 1 that is used productively by learners in their early stages of development (e.g., case contrasts in English pronouns) and there is another set F 2 comprising features that appear later in development (e.g., passivization). There is some relevant evidence for this, but the information is complex and no clear detailed picture has emerged yet. This is part of what we are trying to sort out.

3. What linguistic indicators of L2 proficiency levels have been investigated so far? And how?

Linguistic indicators from the SLA literature Examples of existing surveys or collections of papers on L2 development, often with a predominance of L2 English data Dulay and Burt (1982) Felix (1980) Harley et al. (1990) Hatch (1974) Hatch (1978) Hawkins (2001) Klein (1986) Perdue (1993) White (2003)

Examples of surveys exclusively on the L2 development of languages other than English: L2 French: Carlo et al. (2006) L2 Italian: Giacalone Ramat (2003) L2 Spanish: Montrul (2004)

Type of indicators discussed in the existing SLA literature Semantic categories (Perdue, 1993) Morphosyntactic categories (Dulay and Burt, 1982) Functional grammar categories (Giacalone Ramat, 2003; Carlo et al., 2006) Generative grammar categories (Hawkins, 2001)

Examples of indicators of L2 proficiency level from the SLA literature Perdue (1993) Word formation –N-N word formation –Derivation and word formation –Connective elements in N-N compositions –Kingship reference –Reference to possession Temporality Reference to space

Examples of indicators of L2 proficiency level from the SLA literature Giacalone Ramat (2003) Noun morphology –Gender –Number –Agreement Definite article Indefinite article NP-internal adjectives Predicative adjectives and past participles 3PS pronouns Clitics The verb –Verb morphology and verb placement –Tense –Mood –Aspect –Voice Simple phrases –Negation –Adverbs Subordination and coordination –Adverbials –Completives –Relatives Text-level phenomena –Anaphoric reference –Connectives Marked word order

Examples of indicators of L2 proficiency level from the SLA literature Hawkins (2001) Grammatical morphemes Negation –Sentential negation –Negation and verb movement Word order –Location of verbs –Basic word order Questions Relative clauses Unaccusative verb constructions Null subjects and objects Articles Number Wh-movement Anaphors

Examples of indicators of L2 proficiency level from the SLA literature Carlo et al (2006) NP –Pronouns –N agreement –Clitics Negation –Anaphoric negation –Negation in verbless constituents –Pre-verbal negation –Post-verbal negation Mood and modality VP –Finiteness –SV agreement –Temporal marking From simple to complex constructions –Early morphological inflection –Subordination

Examples of indicators of L2 proficiency level from the SLA literature Dulay and Burt (1982) Pronoun case Article the, a Progressive –ing Contractible copula ‘s Regular short plural -s Contractible auxiliary ‘s Regular past –ed Irregular past Long regular plural –es Possessive ‘s 3PS -s

Type of indicators mentioned by policy makers Suggested contents for the reference level descriptions (RLDs) by the Language Policy Division, DG IV, Council of Europe (November 2005), as minimum common features of RLDs for specific languages: inventories of the linguistic realisation of: –general notions (e.g., existence, quantity, space, time, qualities of things and people, logical relations) –acts of discourse or functions (e.g., imparting and seeking information, expressing and finding out attitudes, deciding on and managing course of action, socialising, structuring discourse, assuring and repairing communication) –specific notions* –lexical elements* –morphosyntactic elements* indication as to whether the proposed forms should be known for reception only or also for production *No specific examples of these categories have been provided in the ‘Guide for production of RLDs’ (Nov 2005).

Other indicators are being explored by: SLA-informed linguistic profiling systems. Eg: –Rapid Profile (Pienemann, Kessler, Lenzing and Hoffmann) –Direkt Profil (Grandfeldt, Kostanidov, L. Persson, E. Persson and Nugues) LT-informed NLP systems. Eg: –E-rater (ETS) Corpus linguistics error tagging systems. Eg: –Cambridge Learner Corpus’ (Nichols, 2003) –Chinese Learner English Corpus’ (Gui and Yang, 2003) –UC Louvain’s (Dagneaux et al., 1998) –FALKO’s (Lüdeling et al., 2005)

Summary We can make a start at answering Q1 and Q2 based on the existing literature. However, the generalizations we can make based on this body of knowledge are still too vague for our purposes, as they don’t allow us to establish precise links between communicative competence levels and linguistic indicators of those levels.

4. Tracing DEMfad

Method for the empirical description of linguistic indicators of development We have adopted a method of processing the empirical data from the literature and from our own studies that consists of 3 main elements: 4.1. Principled selection of D 4.2. DEMfad 4.3. Operators , >> and >

4.1. Principled section of linguistic domains (D) The selection of D is inevitably theory- specific. The Ds selected will depend on one’s chosen: Theory of language Theory of learning There are also likely to be practical considerations that influence our choice of D that are not related to these theoretical assumptions.

4.2. DEMfad D EM f a d D=domain E=emergence M=mastery f=frequency a=accuracy d=distribution

4.3. Operators The following operators describe the possible relationships among relevant phenomena in language development: X >> Y = X takes place before Y X  Y = if Y then X (i.e., Y does not occur without X also occurring) X > Y = the value of X is larger than the value of Y (on measure m)

Accuracy (SOC) orders in Dulay & Burt (1974)

Emergence (E) Caveat: emergence criteria and their operationalisation vary across studies. For example: Min. no. of occurrences in productive use (Rapid Profile, Pienemann, )

Pallotti (2006) has discussed the advantages and also some practical difficulties of using emergence criteria for investigating L2 development. Also see Meisel et al (1981) for an early discussion of these issues. Unfortunately, some studies (typically early case studies) do not even define their emergence criterion and simply report on the first recorded occurrence of specific forms. This is of very limited use for our purposes.

Mastery (M) Caveat: ‘Acquisition criteria’ and their operationalisation vary across studies. Example from FLA research: Brown (1973) used 90% criterion 90% suppliance in three successive samples each sample contains 5 or more obligatory contexts

Examples from SLA research: Andersen (1978) used 80% criterion Ellis (1988) used 75% criterion Vainikka and Young-Scholten (1994) used 60% criterion

It has also been found that certain forms become fully accurate at different times in different functions/contexts. For example, Wode’s (1978) longitudinal study of his four L1 German-speaking children during their 6 month-stay in the US found that while the children were fully target-like from the start regarding the distribution of the regular plural allophones, the same forms were often used incorrectly in possessives. Mxa reg plural -s >> Mxa possessive ‘s

Caveat: Mastery needs to be considered in relation to all of fad, as mastery may be reached in one area but not another. For example, learners may be fully accurate in their production of X, but the frequency of use and/or the distribution of use of X may not be target-like. See, for example, the literature on avoidance strategies, for some concrete examples. Also see the development of demonstratives in Banerjee et al. (2006).

Frequency of use (f) Example: Ringbom (1998) D = verbs f = occurrences of the most frequent verbs per 10,000 words f be > f have > f do > f can

Accuracy (a) Example 1: Ellis (1982) D= verb TO BE M= 75% accuracy a= SOC Mxa cop ‘be’ >> Mxa aux ‘be’

Example 2: Andersen (1978) D= grammatical morphemes M= 80% accuracy a= SOC scores Mxa cop ‘be’ >> Mxa aux ‘be’ >> Mxa irreg past >> Mxa ’have’ aux

Distribution (d) Example 1: Ellis (1988) D= copula BE a= SOC d= pronoun subject contexts vs NP subject contexts axd pron S+cop BE > axd NP S+cop BE

Example 2: Cancino et al (1978) D=negation E= d=VP contexts in which negation is used This study of the acquisition of negation found that the first verbal forms to be negated post-verbally were cop ‘is’ and ‘can’, followed by other auxiliaries. This took place at their proposed stage 3, which comes before the stage at which negative devices are productively combined with auxiliaries in a range of tense and agreement forms. Exd pres cop+neg, can+neg >> Exd other aux+neg

Advantages of this method a.DEMfad can be used with different theories of language. b.The combined use of the three operators ( , >> and >) in the description of the relationship between different phenomena over time and when compared to other phenomena allows the researcher to capture both the stage-like and continuous characteristics of the L2 developmental route.

c.The combined use of the three operators makes the the comparison of cross-sectional and longitudinal data easier that other approaches. d.DEMfad allows us to account for more of the varation/systematicity in the data than approaches looking at E, M, f, a or d independently. e.The three operators allow us to produce implicational hierarchies. These are very cost- effective tools for diagnosis.

5. CEFLING

CEFLING Linguistic Basis of the Common European Framework for L2 English and L2 Finnish Project funded by the Academy of Finland Based at the University of Jyväskylä, but with collaborators at a number of universities in Europe (SLATE network) Homepage:

Research questions 1.What combinations of linguistic features characterise learners’ performance at the proficiency levels defined in the Common Framework and its Finnish adaptations? 2.To what extent do adult and young learners who engage in the same communicative tasks, at a given level, perform in the same way linguistically? To what extent are the adult-oriented CEFR levels and their Finnish adaptations for young learners equivalent?

3.To what extent are the pedagogical tasks found in the teaching materials in the Finnish comprehensive school comparable with the tasks defined in the CEFR and the new curriculum? 4.What are the linguistic and communicative features that teachers (or National Certificates raters) pay attention to when assessing learners with the help of the Finnish adaptations of the CEFR scales? How do these features relate to the linguistic and communicative analysis of the same performances?

Tasks From all learners: – to a friend or colleague – to your teacher – to a store –Opinion piece –Narrative piece From subsets of learners: –Translation (L1 Finnish --> L2 English) –Word formation tasks

Structure of the dataset

Transcription and coding We are adapting CHILDES tools for use with writing performances The coding system is being developed. Some of it can be done semi- automatically (e.g., MOR analysis for English) but other codes have to be entered manually.

Planned analyses D n EMfad t6 C2 D n EMfad t5 C1 D n EMfad t4 B2 D n EMfad t3 B1 D n EMfad t2 A2 D n EMfad t1 A1

Ongoing and planned analyses A number of Ds are currently being explored. See the list of current subprojects at ng/en/Subprojects ng/en/Subprojects

Examples of initial findings L2 Finnish: –It’s not just emergence, mastery or frequency of relative clause use that are particularly informative (RCs appear at all levels), but the frequency x distribution of RCs (some types of RCs are typical at specific levels). L2 English: –Number marking on N is not a discriminating feature in terms of emergence (E is clear even at A1) or mastery (accuracy is not 100% even at C2), but relative accuracy in plural contexts allows us to conclude that if pl TLU<90%, then the performance is below B1 level.

6. Final thoughts

Some methodological suggestions 1.Collaboration is better than competition Get in touch with other teams doing this type of work (e.g., the SLATE network, your country’s RLD team, etc.) Aim to open up your work to a range of experts and stake holders (e.g., SLA researchers, language testers, teachers, policy makers, etc.)

2.Datasets The following variables should ideally be controlled for: task, topic, conditions under which data are produced by learners (e.g., timed/untimed, high-/low-stakes, etc.) The NS comparison may be helpful (e.g., to help us understand M better) We may need to explore some data from levels below A1 to understand E better Format of transcription and coding should ideally be done following some generally accepted standard in the field (e.g., CHILDES)

3.Rating of performances Assigning performances to communicative levels should be done with great care (or else you risk the linguistic analyses that follow not being as reliable or informative as they could be).

4. Linguistic analyses There is no such thing as an a-theoretical linguistic analysis of data. It’s potentially problematic to work with unprincipled selections of Ds. Overlap between levels is common (esp. between adjacent levels). This suggests that communicative levels tend to yield linguistic indices that are probabilistic rather than categorical. A single feature (e.g., number) is unlikely to be a completely reliable index of level, but groups of features may be. Given the typically high levels of individual variation (within-level and within-learner) in L2 development, group scores should be used and interpreted with caution. Combined, multivariate, factorial analyses (e.g., DEMfad) are likely to explain more than analyses of independent aspects of development.

Much of the work reported here was possible thanks to the financial support provided by the following institutions: