1. Tracking L2 Lexical and Syntactic Development Xiaofei Lu CALPER 2010 Summer Workshop July 14, 2010

2. 2 Outline Lexical & syntactic complexity: The what and why Syntactic complexity in EFL writing Lexical complexity in EFL speaking

3. Lexical and Syntactic Complexity: The What and Why

4. 4 What is lexical and syntactic complexity Lexical complexity  A multidimensional feature of language use encompassing lexical density, sophistication and variation (Wolfe-Quintero et al. 1998; Read 2000)  Does not focus on errors, a dimension in Read’s (2000) conceptualization of lexical richness Syntactic complexity  The range of forms that surface in language production and the degree of sophistication of such forms (Ortega 2003)

5. 5 Why measure linguistic complexity? First language acquisition & psycholinguistics  Studies of L1 developmental sequence  Objective measures of L1 developmental level  Ordering experimental stimuli by complexity  Relationship of complexity in childhood to symptoms of Alzheimer’s disease (Kemper et al. 2001)

6. 6 Why measure linguistic complexity? Second language acquisition  Objective L2 developmental indices  Assessing cross-proficiency differences  Assessing effect of pedagogical intervention  Tracking L2 learners’ linguistic development over time  Relationship between lexical/syntactic complexity and proficiency claimed in many test rating scales

7. Syntactic Complexity in EFL Writing Lu, X. (forthcoming 2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15. Lu, X. (forthcoming 2010). A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development. TESOL Quarterly, 44(4).

8. 8 Outline Measures of L2 syntactic complexity L2 syntactic complexity analyzer Syntactic complexity & EFL writing development Summary

9. 9 Measures of L2 syntactic complexity Measures reviewed in two research syntheses  Wolfe-Quintero et al. (1998)  Ortega (2003) Selection criterion  At least one previous study showed at least weak correlation with or effect for proficiency Issues among previous studies  Variation in measure selection and definition  Variation in experiment design  Inconsistent results reported on the same measures

10. 10 Measures of L2 syntactic complexity Length of production 1.Mean length of clause (MLC) 2.Mean length of sentence (MLS) 3.Mean length of T-unit (MLT) Sentence complexity 4.Mean number of clauses per sentence (C/S)

11. 11 Measures of L2 syntactic complexity Subordination 5.Mean number of clauses per T-unit (C/T) 6.Mean number of complex T-units per T-unit (CT/T) 7.Mean number of dependent clauses per clause (DC/C) 8.Mean number of dependent clauses per T-unit (DC/T)

12. 12 Measures of L2 syntactic complexity Coordination 9.Mean number of coordinate phrases per clause (CP/C) 10.Mean number of coordinate phrases per T-unit (CP/T) 11.Mean number of T-units per sentence (T/S) Particular grammatical structures 12.Mean number of complex nominals per clause (CN/C) 13.Mean number of complex nominals per T-unit (CN/T) 14.Mean number of verb phrases per T-unit (VP/T)

13. 13 L2 syntactic complexity analyzer Input: plain English text Step 1: Parsing using Stanford parser Step 2: Retrieving & counting occurrences of  Words, sentences, clauses, dependent clauses  T-units, complex T-units  Coordinate phrases, complex nominals, verb phrases Step 3: Computing ratios for the 14 measures Output: 14 syntactic complexity indices

14. 14 How counting is done Word: all non-punctuation tokens Other units: Tregex (Levy & Andrew, 2006)  Define the units linguistically  Formulate Tregex patterns matching the unit definitions  Query the parse trees with the Tregex patterns  Retrieve/count (sub)trees matching each pattern

15. 15 Definition and pattern examples Clause: subject + finite verb (Polio 1997) ‘S|SINV|SQ < (VP <# MD|VBP|VBZ|VBD)’ Dependent clause: adverbial, adjectival or nominal clause ‘SBAR < (S|SINV|SQ < (VP <# MD|VBP|VBZ|VBD))’

16. 16 Evaluation Experiment setup  40 essays from the Written English Corpus of Chinese Learners (Wen et al. 2005), average 315 words  Written by English majors in four-year colleges in China  20 used for training, 20 for testing  Two annotators counted unit occurrences in the essays Inter-annotator agreement  Evaluated on 10 essays  F-score for unit identification:.907 (CN) - 1.000 (S)  Correlations of complexity ratios:.912 (CT/T) - 1.000 (MLS)

17. 17 Unit identification results on test data CountsSystem-annotator agreement StructureSystemManualIdenticalPrecisionRecallF-score S357 1.000 C545558530.972.950.961 DC170178161.947.904.925 T376380369.981.971.976 CT129136126.977.926.951 CP138135125.906.926.916 CN660572511.774.893.830 VP750758698.931.921.926

18. 18 Correlations of complexity ratios MeasureDevelopmentTestMeasureDevelopmentTest MLC.941.932DC/T.950.941 MLS1.000 CP/C.845.834 MLT.989.987CP/T.876.871 C/S.939.928T/S.931.919 C/T.978.961CN/C.883.867 CT/T.903.892CN/T.904.896

19. 19 Error analysis Attachment and conjunction scope errors  e.g., benefit a lot from [the Internet in academic study]  More reliable in identifying higher-level units: S, C, T, CT Learner errors not a major cause for problems  Advanced EFL learners  Idiomaticity vs. grammatical completeness  Some errors do not lead to structural misanalysis

20. 20 Syntactic complexity & EFL writing development Research questions The WECCL corpus Results Summary

21. 21 Research questions 1)Effect of sampling condition 2)Measures discriminating proficiency levels 3)Magnitudes for differences to be significant 4)Relationships between measures 5)Patterns of development for the measures

22. 22 The WECCL corpus Essay length: range=[89, 892], mean=315, sd=87 School Level ArgumentationNarrationExpositionAll Time d UntimedTimedUntimedTimedUntimed 16953958903001209 244139824602801113 35044599103001084 460088000148 All1700125251408803554

23. 23 Effect of sampling condition Institution: sig. inter-institution dif. for  All metrics using all data  12 metrics using Y1-3 timed arg essays Genre: sig. dif. between arg vs. nar for  All metrics using arg & nar essays  All metrics using timed arg & nar essays  13 metrics using timed arg & nar essays from ND

24. 24 Sampling condition effect (cont) Timing: sig. dif. between un/timed arg for  13 measures using all arg essays  11 metrics using arg essays from ND Data for other research questions  422 timed argumentative essays from ND

25. 25 Measures discriminating levels 3 showed sig. dif between first 3 levels  MLC, CN/C, and CN/T 4 showed sig. dif between first 2 levels  MLS, MLT, CP/C, and CP/T 5 showed sig. dif. between non-adjacent levels  C/S, C/T, CT/T, DC/C, and DC/T 2 showed no sig. between-level dif.  T/S and VP/T

26. 26 Significant magnitudes MetricMagnitudeLevelsMeasureMagnitudeLevels MLC.5732-3DC/C-.0331-4 MLS1.6581-2DC/T-.0711-4 MLT1.6511-2CP/C.0401-2 C/S-.1122-4CP/T.0611-2 C/T-.0782-4CN/C.1332-3 CT/T-.0432-4CN/T.1782-3

27. 27 Relationships between measures Strong relationship between measures of the same type or involving the same structure MLS and MLT show weak-moderate correlations with subordination measures MLC shows low-weak negative correlations with subordination measures Length measures show moderate-high correlations with CN measures and weak-moderate correlations with CP measures CN and CP measures weakly correlated with each other

28. 28 Developmental patterns Measures with sig. positive changes  Linear increase Y1-4: MLC, CN/C  Increase Y1-3 (Y4=Y3): CP/C  Increase Y1-3 (Y4<Y3, insig.): MLS, MLT, CP/T, CN/T Measures with sig. negative changes  Linear decrease Y1-4: C/S  Nonlinear Y1 Y3>Y4: DC/C, DC/T

29. 29 Summary of findings Important to control for the effects of relevant learner-, task- and context-related factors Seven measures recommended for future use  CN/C, MLC: discriminates 2+ adjacent levels, linear increases  CN/T, MLS, MLT: 2 adjacent levels; positive sig changes  CP/C, CP/T: nonadjacent levels, positive sig changes Developmental prediction: complexification at the phrasal level vs. the clausal level

30. 30 Summary of findings (cont.) Smaller magnitudes than reported previously Clause as a potentially more informative unit of analysis than T-unit

31. 31 Limitations and future research Incorporating more measures and flexible definitions of structures into the analyzer Other conceptualizations of proficiency level Effect of L1 on syntactic development Relationship between developmental measures of fluency, accuracy and complexity at different linguistic levels

32. Lexical Complexity in EFL Speaking Lu, X. (under review). The relationship of lexical richness to the quality of ESL speakers’ oral narratives.

33. 33 Outline Research goals and motivation Measures of lexical complexity Methodology Results Conclusion

34. 34 Research goals and motivation Research goals  Automate lexical complexity analysis using 25 measures  Evaluate the relationship of these measures plus the D measure to the quality of EFL speakers’ oral narratives Motivation  Lexical complexity an important construct in L2 teaching and research  Relationship between lexical complexity and proficiency claimed in many test rating scales

35. 35 Measures of lexical complexity Lexical complexity measures proposed in language acquisition studies and reviewed in  Wolfe-Quintero et al. (1998)  Read (2000)  Malvern et al. (2004) Measures of the following three dimensions  Lexical density  Lexical sophistication  Lexical variation

36. 36 Lexical density Proportion of lexical words (N lw / N) (Ure 1971) Previous findings  Lower in spoken than written texts (Halliday 1985)  Affected by various sources (O’Loughlin 1995)  Relation to L2 writing non-significant (Engber 1995) Inconsistent definition of lexical words  All nouns and adjectives  Adverbs with adjective base  Full verbs (excluding modal/auxiliary verbs)

37. 37 Lexical sophistication Five measures examined  LS1: N slw / N lw (Linnarud 1986; Hyltenstam 1988)  LS2: T s / T (Laufer 1984)  VS1: T sv / N v (Harley & King 1989)  CVS1: T sv / sqrt(2N v ) (Wolfe-Quintero et al. 1998)  VS2: T sv 2 / N v (Chaudron & Parker 1990)

38. 38 Lexical sophistication (cont.) Previous findings  LS1: NS-NNS dif sig (Linnarud 1986) ; non-sig (Hyltenstam 1988)  LS2: sig pre-and post-essay dif (Laufer 1984)  VS1: sig NS-NNS dif (Harley & King 1989) Varying definitions of sophistication  2000-word BNC frequency list (Leech et al. 2001)

39. 39 Lexical variation 20 measures examined 4 based on NDW  NDW: Number of different words  NDW-50: NDW in first 50 words of sample  NDW-ER50: mean NDW of 10 random 50-word subsamples  NDW-ES50: mean NDW of 10 random 50-word sequences

40. 40 Lexical variation (cont.) 7 based on TTR for total vocabulary  Type token ratio (TTR)  Mean TTR of all 50-word segments (MSTTR)  LogTTR, Corrected TTR, Root TTR, Uber  The D measure (McKee et al. 2000) 9 based on TTR for word classes  T {LW, V, N, Adj, Adv, Mod} / N lw  T v / N v, T v 2 / N v, T v / sqrt(2N v )

41. 41 Lexical variation (cont.) Previous findings  NDW and TTR useful, but affected by sample size  Transformations of NDW & TTR not equally useful  D claimed superior; results mixed (Jarvis 2002; Yu 2010)  Mixed results for word class TTR measures  No consensus on a single best measure

42. 42 Research questions How does LD relate to the quality of EFL speakers’ oral narratives? How do the LS measures compare with and relate to each other as indices of the quality of EFL speakers’ oral narratives? How do the LV measures compare with and relate to each other as indices of the quality of EFL speakers’ oral narratives? How do LD, LS and LV compare with and relate to each other as indices of the quality of EFL speakers’ oral narratives?

43. 43 Data Spoken English Corpus of Chinese Learners (Wen et al. 2005)  Transcripts of TEM-4 Spoken Test data in 1996-2002  Task 2 data used: 3-minute oral narratives  Students ranked within groups of 32-35  12 groups of data used (1999-2002; N=32-35 each)  Only rankings available, but not actual scores Example topic (2001)  Describe a teacher of yours whom you found unusual

44. 44 Computing the measures Preprocessing  Part-of-speech tagging (Stanford tagger)  Lemmatization (Morpha) Measure computation  D measure: vocd utility in CLAN  Type counting: w, sw, lw, slw, v, sv, n, adj, adv  Token counting: w, lw, slw, v  Computation of the other 25 ratios

45. 45 Analysis Spearman’s rho computed for each group  X: test takes’ rankings within the group  Y: Values of each of the 26 measures  Meta-analysis to combine results from the 12 groups Students divided into 4 levels based on rankings  Levels A, B, C and D  ANOVA’s run to determine inter-level differences

46. 46 Analysis (cont.) Alpha level =.05 / 28 =.0018 Identification of discriminative measures  Significant combined rho (p <.0018)  Significant between-level differences with linear decreases from Level A to Level D

47. 47 Lexical density and sophistication MeasureCombined rhop-valueMeasureCombined rhop-value Words.437.000LS2.050.336 W/Min.437.000VS1.133.010 LD.011.836CVS1.166.001 LS1.048.355 VS2.165.001

48. 48 Lexical density and sophistication (cont.) MeasureABCDFSig. Words336.16295.95297.76256.3428.335.000 W/Min112.05298.65099.25285.44628.335.000 LD.417.415.409.414.896.443 LS1.227.235.221.225.681.564 LS2.261.272.256.2602.736.043 VS1.072.086.067.0732.629.050 CVS1.343.383.299.2973.722.042 VS2.314.401.274.2622.760.042

49. 49 Lexical density and sophistication (cont.) LS1LS2VS1CVS1VS2 LS11.000 LS2.637**1.000 VS1.456**.391**1.000 CVS1.414**.382**.966**1.000 VS2.381**.350**.909**.935**1.000

50. 50 Relationships among the dimensions Low to weak correlations among measures in different dimensions Lexical variation demonstrated strongest relationships to raters’ judgments of the quality of EFL speakers’ oral narratives

51. 51 Summary of findings The three dimensions posited in language acquisition literature appear different constructs No/small effect for lexical density/sophistication found Lexical variation correlated strongly with quality 9 LV measures recommended NDW correlates strongly with length, but worth considering in the case of timed oral narratives Transformed TTR measures perform better than the original TTR measures

52. 52 Limitations and future research A factor analysis will show patterns of relationships No scores available, so not possible to run regression models Division of students into 4 levels could be problematic Replication using EFL writing data and other conceptualizations of proficiency Effects of task-related variables Relations among factors determining quality