Compiling and Analyzing Your Own Learner Corpus Xiaofei Lu CALPER 2012 Summer Workshop July 16, 2012

2 Workshop outline Opening discussion and corpora overview Graphic Online Language Diagnostic (GOLD) overview Sample GOLD (and related) projects GOLD (or related tool) project lab GOLD (or related tool) project discussions Concluding discussion

3 Opening discussion Brief introduction of your professional/language background and teaching/research interests Prior experience with corpus linguistics Primary challenges you are dealing with Primary purposes and goals for taking this workshop and for learning about corpus linguistics in general Any other relevant information

4 Corpora overview What is a corpus Types of corpora Corpus design and compilation Corpus annotation Corpus querying and analysis Learner corpora and L2 development Resources

5 What is a corpus? Leech (1992):  an unexciting phenomenon, a helluva lot of text, stored on a computer Sinclair (1991):  a collection of naturally-occurring language text, chosen to characterize a state or a variety of language Sinclair (2004):  a collection of pieces of language text in electronic form, selected according to external criteria to represent, as far as possible, a language or language variety as a source of data for linguistic research

6 Types of corpora General-purpose vs. specialized corpora  British National Corpus & Russian National Corpus British National CorpusRussian National Corpus  Michigan Corpus of Academic Spoken English Michigan Corpus of Academic Spoken English Native vs. learner corpora  International Corpus of Learner English International Corpus of Learner English  Spanish Learner Language Oral Corpora Spanish Learner Language Oral Corpora Monolingual vs. parallel & comparable corpora  The JRC-Acquis Multilingual Parallel Corpus The JRC-Acquis Multilingual Parallel Corpus  The English-Chinese Parallel Concordancer The English-Chinese Parallel Concordancer

7 Types of corpora (cont.) Corpora representing one or diverse varieties  International Corpus of English International Corpus of English Synchronic vs. diachronic corpora  The Corpus of Historical American English The Corpus of Historical American English Spoken vs. written corpora  Michigan Corpus of Upper-Level Student Papers Michigan Corpus of Upper-Level Student Papers

8 Corpus design Purpose and type of corpus  Spoken/written; cross-sectional/longitudinal External criteria for content selection  Communicative function of a text  Mode, medium, interaction, domain, topic Representativeness, balance, size, sampling Design of the BNC

9 Corpus design (cont.) Encoding meaningful metadata information  Learner: L1, gender, program level, discipline …  Sample: date, mode, task, genre, rating …  Facilitates contrastive and longitudinal studies MICASE speaker and transcript attributes Corpus markup: The ICE exampleThe ICE example

10 Corpus annotation Why annotate Levels of corpus annotation Difficulties for corpus annotation Standards and encoding

11 Why annotate Raw text vs. annotated text: How do you…  Count the number of words in a Chinese text?  Calculate the lexical density of an English text?  Count the frequency of can as a modal verb?  Know how many T-units in a text are complex?  Extract all imperative sentences from a text?  Know whether a syntactic structure is used in a text?

12 Levels of corpus annotation Sentence and word segmentation Part-of-speech (POS) tagging and lemmatization Syntactic parsing Semantic, pragmatic, and discourse annotation Learner corpora: error annotation Project-specific annotation

Sentence and word segmentation Why is this non-trivial? I went to the shops in Jones St. Saturday afternoon with Mr. Smith. I can’t remember whether it’s a second- or third-grade book.  克林顿在讲话中指出 Clinton pointed out in his speech (that…) 克林顿在 讲话中指出 Clintonat speechmiddlepoint-out  克林顿 在 讲话 中指 出 Clinton at speech middle-finger out

POS tagging The what and why What are the difficulties?  Ambiguity: 48% tokens in the Brown Corpus  Unknown words: neologism Tagsets: overspecificatin vs. underspecification  Penn Treebank Tagset vs. CLAWS7 Tagset Penn Treebank TagsetCLAWS7 Tagset

Lemmatization Counting linguistic items  Types – number of different words  Tokens – number of words What constitutes a different word type?  go, went, gone, goes, going?  differ, difference, different, differently?  can as a noun, verb, and modal verb?

16 Demos and tools: Part 1 Xerox morphological analyzer (demo only) Xerox ICTCLAS for Chinese segmentation and POS tagging ICTCLAS Querying POS-tagged corpora and Querying POS-tagged corpora Stanford POS tagger for English Stanford POS tagger Tree Tagger for multiple languages Tree Tagger

Chunking and parsing Partial/full structural analysis of each sentence My dog likes eating sausage. (ROOT (S (NP (PRP$ My) (NN dog)) (VP (VBZ likes) (S (VP (VBG eating) (NP (NN sausage))))) (..)))

Chunking and parsing (cont’d) What is it useful for?  Retrieving examples of grammatical patterns  Grammar checking, syntactic complexity analysis  NLP applications that require syntactic analysis Difficulties  Ungrammatical sentences  Ambiguities, e.g., PP attachment  Errors from preprocessing steps

19 Semantic and discourse analysis Semantic and discourse features Word sense disambiguation Propositional idea density Coherence and cohesion

20 Annotation standards and encoding Useful standards  Separable, linguistically consensual  Documentation, compatibility with existing standards Encoding  Simple encoding: present_JJ  XML-style: present  Format varies, depending on level of annotation Manual, computer-aided, and automatic annotation  Efficiency, scale, reliability  UAM CorpusTool UAM CorpusTool

21 Demos and tools: Part 2 Stanford parser for Arabic, Chinese and English Stanford parser Word sense disambiguation demo Computerized Propositional Idea Density Rater Coh-Metrix for text coherence analysis Coh-Metrix CHILDES and CLAN Computerized Profiling WMatrix

22 Corpus querying and analysis Manual analysis? Corpus-specific online interfaces  Raw: MICASE and MICUSPMICUSP  POS-tagged: BYU  Grammatically and semantically tagged: RNCRNC General-purpose online interfaces: GOLD Windows-based querying/concordancing tools  WordSmith Tools & AntConc WordSmith ToolsAntConc

23 Corpus querying and analysis Natural language processing tools  Good for processing annotated corpora  Extracting occurrences of grammatical patterns  Examples: Stanford parser and TregexStanford parser and Tregex

24 Resources Books and journals  Hunston (2002): Corpora in Applied Linguistics  McEnery (2006): Corpus-Based Language Studies  International Journal of Corpus Linguistics  Corpus Linguistics and Linguistic Theory  Corpora Websites and mailing lists  Bookmarks for corpus-based linguists Bookmarks for corpus-based linguists  Linguistic data consortium Linguistic data consortium  The corpora list; corpus in delicious The corpora listcorpus in delicious  Stanford Natural Language Processing Group Stanford Natural Language Processing Group

25 Discussion What kind of corpus do you intend to compile and/or use? For what purpose? What are the design issues? How do you intend to format, organize and store your files? Do you intend to annotate your corpus in some way? How? How do you intend to search/query your corpus?

26 Learner corpora and L2 development Samples from same students at different times  Did (targeted) language development take place?  Was a particular pedagogical intervention effective? Samples from different students  What areas do students show different levels of development?  What factors affect students’ language development?

27 Graphic Online Language Diagnostic A free online tool for teachers to assess their students’ language development  Developed at CALPER, Penn State, funded by DOE  Project co-directors: Xiaofei Lu and Michael McCarthy Teachers can use GOLD to  Compile, upload, and manage their own corpora  Share corpora with each other  Search and analyze corpora Demonstration

28 Corpus compilation A user can compile a corpus by  Directly compiling and uploading an XML file  Using the easy-to-use guided XML creation interface An uploaded corpus can be easily managed  Documents can be added or deleted  The whole corpus can be deleted  Content and metadata of individual documents can be easily accessed

29 Corpus sharing GOLD facilitates easy data sharing A corpus may be set to be  Private, shared, or public Corpus owner may give other users right to  View, add, edit, or delete corpora Demonstration

30 Basic corpus information Word count  Alphabetic or numeric order  Can be downloaded as a text file Corpus and document statistics  Mean sentence length  Mean word length  Type-token ratio Demonstration

31 Corpus search Select one or more corpora to search Specify key words or phrases  May use the wildcard character, e.g. book* Specify contexts  Size of context window  Context words and their positions Specify metadata conditions

32 Corpus search results Display of search results  Sortable KWIC display of search results  Sortable graphic display of search results Demonstration

33 Lexical bundle/collocation search Procedure  Select one or more corpora to search  Specify search word  Specify contexts  Specify metadata conditions Search results  Sortable list of n-grams found in selected corpora Demonstration

34 Summary of features Difference from other online tools  Can create, share, and search multiple corpora  Can easily search subsets of data  Can work with any language Summary of corpus analysis functions  Word list  Corpus and document statistics: mean sentence length, mean word length, type-token ratio  Corpus search and collocation search

35 Sample questions to ask With data from an individual student, one can either describe or track development in  Patterns of usages of words and phrases – frequency, underuse, overuse, etc.  Lexical and syntactic complexity  Appropriate usage of words and phrases in context  Patterns of usages of lexical bundles

36 Sample questions to ask (cont.) With data from different (groups of) students, one can compare similarities or differences among different (groups of) students in terms of  Patterns of usages of words and phrases – frequency, underuse, overuse, etc.  Lexical and syntactic complexity  Appropriate usage of words and phrases in context  Patterns of usages of lexical bundles

37 Future enhancements Corpora for benchmarking Multilingual natural language processing Suggestions on desirable functions welcome