Assessing lexical awareness: EFL learners and English word- formation Katja Mäntylä and Ari Huhta

CEFLING Linguistic Basis of the Common European Framework for L2 English and L2 Finnish  Project funded by the Academy of Finland  Based at the University of Jyväskylä; part of the European SLATE network (Second Language Acquisition and Testing in Europe)  Homepage:

Questions  How to test word formation skills? How do the three methods used in the study function?  What is the relationship between the word-formation skills and overall written proficiency of Finnish school pupils ?  (What kind of knowledge do they have on English word- formation?)  (How do their word-formation skills develop?)  (Is there any difference between Finnish and Swedish speaking participants?)

Word-formation and SLA  Word-formation and SLA in general  Role in teaching English in a Finnish school?  Derivation chosen because Productivity of the method The participants familiar with it at least implicitly

Participants  7th graders –year-olds, have studied English as a FL for at least 4 years 162 completed the word formation tests, 87 of whom also completed several writing tasks (next step: 8th and 9th graders)  Different parts of Finland

Word-formation test 1  Three written word-formation tests (revised after piloting) 1. Sentences / sentence pairs in English with a Finnish translation of the target word (productive gap-filling): I am ________ (varma) that he will get the job in London. He will _________ (varmasti) get the job in London.  sure - surely

Word-formation test 2 (pre-pilot version) 2. Sentences with non-words with explanations in Finnish (gap-filling) Some of the non-words taken from the DIALANG placement test (English) designed by Paul Meara Example in Finnish (with Finnish non-words) She could bourble animals very well because she was a good _________. (henkilö, joka tekee lihavoidun sanan kuvaamaa toimintaa/työtä) (a person who does the action described by the bolded word)

Word-formation test 2 (final version) 2. Sentences with non-words with explanations in Finnish (gap-filling): Example in Finnish (with Finnish real words) She could bourble animals very well because she was a good ____ bourble____. (henkilö, joka tekee lihavoidun sanan kuvaamaa toimintaa/työtä) (a person who does the action described by the bolded word)

Word-formation test 3 3. A list of prefixes from which the participants were to choose suitable ones to fill in the gaps in sentences He did not follow the instructions. He had ___ understood them. anti- de- dis- in- im- il- ir- inter- intra- mega- mini- mis- mono- neo- non- poly- post- pre- pro- re- trans- un-

Writing tasks  to a friend  to one’s teacher  to a store  Opinion piece  Narrative piece Each student wrote 3-4 texts Each text was assessed by four raters the rating scale was a combination of several writing scales from the CEFR that best suited the writing tasks

Marking word-formation tests  Double marking  Scoring: Productive gap-fill test : Non-words based test and List-choice based test :  The respondents were very creative: minigabl  Spelling errors more or less ignored in scoring: unbelievubl,unbelievevabl, unbelievobl; understant, anderstand  (cf. shore  sure, deffreno  different)

Analysis of the word formation tests  TiaPlus programme (CITO, the Netherlands) – for classical item analyses – for norm-referenced tests

Characteristics of the 3 tests ItemsMean score (percent) Standard Deviat- ion Std. Error of Mean Cron- bach’s Alpha Alpha for 40-item test Average item /total correlation Productive gap-fill test 1970% Non-words based test 831.5% (27%) 23.6 (24.4) 2.0 (1.9).70 (.74).92 (.93).58 (.61) List-choice based test 1236% All 3 tests together 3951%.90.91(.47)

Test 1 (Productive gap-fill test)

Test 2 (Non-words based test)

Test 3 (List-choice based test)

Non-words based test (22 non-respondents removed) ITEMMEAN SCORE (PERCENT) ST. DEVIATION (on 0-2 scale) ITEM / TOTAL CORRELATION (Hennyson’s correction) ITEM / REST CORRELATION

Examples / non-words based test (item 1) Score groups: 1 = lowest scoring group in non- words test 4 = highest scoring group Rit = item-total correlation P = mean score (%)  She could bourble animals very well because she was a good ______bourble________. (= a person who does the action described by the bolded word [in Finnish]) Item B1 Rit = 0.34 P = 0.75 Percent correct Score Groups

Examples / non-words based test (item 6) 6. Before we can finally honch this car you need to ________honch__________ it. (= do first / in advance of the action described in the bolded word [in Finnish]) Item B6 Rit = 0.63 P = 0.10 Percent correct Score Groups

Examples / non-words based test (item 8) 8. I did not monadate the story that your friend told me yesterday but what you tell me now is much more _________monadate____________. (= has / cointains the thing described in the bolded word [in Finnish])) Item B8 Rit = 0.10 P = 0.19 Percent correct Score Groups

Correlation between the word formation tests and the rating of writing skill WRITING SKILL (on CEFR scale) (Mean rating across 4 raters and all tasks completed by the student) Productive gap-fill test.691 Non-words based test.575 (n=75) List-based test.675 n = 87 p =.000

Correlation between word-formation tests and the rating of writing skill (2) WRITING SKILL (on CEFR scale) Mean rating across 4 raters and all tasks completed by student Productive gap-fill test items tapping the base form of word (9).578 (.589, if item A17 removed) - items tapping the inflected form of word (10).713 Non-words based test.575 (n = 76) List-based test.675 n = 87 p =.000

Mean word-formation test scores (%) across CEFR levels (based on writing) A1 (n= 27) A2 (n= 42) B1 (n= 15) B2 (n= 2) Productive gap-fill test items tapping the base form of word (9) items tapping the inflected form of word (10) Non-words based test List-based test

Correlations between word-formation tests N = 140 / 162A (total) Productive gap-fill test A1. items tapping the base form A2. items tapping the inflected form B. Non- words based test C. List- based test A. Productive gap-fill test 1.00 (.902)(.966) A1. Items tapping the base form A2. Items tapping the inflected form B. Non-words based test C. List-based test 1.00

Structure of the non-words based test Initial factor analyses suggest that  Item 8 differs most from all the other (non-words or other items used)  If Item 8 is deleted, there appears to be 2 factors underlying the word- formation test: 1)Items 3 & 7, plus item 1 2)Item 6, plus items 5 & 4 to some extent - Item 2 is evenly split between the two factors  Item 1 differs from the other non-word items and is closely related to several productive gap-fill items  Item 4 may also differ from the other non-word items  Some non-word items (2, 3, 6, 7) quite strongly related to List-based items (items 8, 11, 12)

Comparison of the three word-formation test methods ProsCons Productive gap-fill testvery familiar test typememorising words? marking difficult (if very fine-tuned) the relationship between items Non-words based testfocus only on word- formation relative difficulty unfamiliar test type List-based testquick and easy to take and mark focus on word-formation (relatively) memorising words? unfamiliar test type? difficult to write easy items?

Overall conclusions about the word- formation tests  Productive gap-fill: possibly useful for testing word-formation but more work needed to develop / select suitable items  Non-words test: promising (e.g. as reliable as the others; construct relevant) but there is a ’threshold’ to overcome for the test-taker, i.e. to understand what it is about  List-based gap-fill: promising (fairly construct relevant), but the difficulty of the words need to suit students’ level better  Which level of affixes/words? Relationship with the word frequency?