1 Evaluating word sketches Adam Kilgarriff Lexical Computing Ltd Lexicography MasterClass Ltd Universities of Leeds and Sussex

Geneva, April 2010 Adam Kilgarriff 2 Overview Research programme Examples: Case study Word sketching Evaluating word sketches

Geneva, April 2010 Adam Kilgarriff 3 What is language?

Geneva, April 2010 Adam Kilgarriff 4 What is language? In our heads

Geneva, April 2010 Adam Kilgarriff 5 What is language? In our heads In texts and sound signals

Geneva, April 2010 Adam Kilgarriff 6 What is language? In our heads In texts and sound signals Both

Geneva, April 2010 Adam Kilgarriff 7 Methodology Study language in our heads Competence Chomsky “rationalist” (Descartes, Leibniz)‏

Geneva, April 2010 Adam Kilgarriff 8 Methodology Study language in our heads Competence Chomsky “rationalist” (Descartes, Leibniz)‏ Odd method for objective science Practical problems: coverage, arbitrariness

Geneva, April 2010 Adam Kilgarriff 9 Methodology Study text “empiricist” (Locke, Hume)‏ Physics: forces, matter Chemistry: chemicals, bonds Language: text, speech signals

Geneva, April 2010 Adam Kilgarriff 10 It goes against the grain What is important about a sentence? its meaning Corpus methodology: Throw away individual sentence meaning Find patterns

Geneva, April 2010 Adam Kilgarriff 11 Twenty years of rapid ascent Computer power Corpora bigger and bigger data sets Language technology tools lemmatizers, POS-taggers, parsers Machine learning, pattern-finding

Geneva, April 2010 Adam Kilgarriff 12 A virtuous circle Pattern finding Linguistic processing Corpus Lexicon Part-of-speech tagging Parsing Lemmatizing More data → gets richer each time round

Geneva, April 2010 Adam Kilgarriff 13 Case study: corpus lexicography - four ages

Geneva, April 2010 Adam Kilgarriff 14 Age 1: Pre-computer Oxford English Dictionary: 20 million index cards

Geneva, April 2010 Adam Kilgarriff 15 Age 2: KWIC Concordances From 1980 Computerised

Geneva, April 2010 Adam Kilgarriff 16 Age 2: KWIC Concordance

Geneva, April 2010 Adam Kilgarriff 17 Age 2: KWIC Concordances From 1980 Computerised COBUILD project was innovator the coloured-pens method

Geneva, April 2010 Adam Kilgarriff 18 The coloured pens method

Geneva, April 2010 Adam Kilgarriff 19 Age 2: limitations as corpora get bigger: too much data 50 lines for a word: read all 500 lines: could read all, takes a long time 5000 lines: no

Geneva, April 2010 Adam Kilgarriff 20 Age 3: Collocation statistics Problem: too much data - how to summarise? Solution: list of words occurring in neighbourhood of headword, with frequencies Sorted by salience

Geneva, April 2010 Adam Kilgarriff 21 Collocation listing For collocates of save (>5 hits), window 1-5 words to right of nodeword word yourmoney estimatedjobs faceannually thousandsenormous costslives dollars$1.2 lifeforests

Geneva, April 2010 Adam Kilgarriff 22 Age 4: The word sketch A corpus-derived one-page summary of a word’s grammatical and collocational behaviour

Geneva, April 2010 Adam Kilgarriff 23 Age 4: The word sketch Large corpus Parse to find subjects, objects, heads, modifiers etc One list for each grammatical relation Statistics to sort each list, as before

Geneva, April 2010 Adam Kilgarriff 24 Macmillan English Dictionary For Advanced Learners Ed: Rundell, 2002

Geneva, April 2010 Adam Kilgarriff 25 Euralex 2002

Geneva, April 2010 Adam Kilgarriff 26 Euralex 2002 Can I have them for my language please

Geneva, April 2010 Adam Kilgarriff 27 The Sketch Engine Input: any corpus, any language Lemmatised, part-of-speech tagged specification of grammatical relations Word sketches integrated with corpus query system Developer: Pavel Rychly, Brno

Geneva, April 2010 Adam Kilgarriff 28 Users: Dictionary publishers Oxford UP, Collins, Chambers, Macmillan, Cambridge UP Universities Teaching, research Framenet Language teaching Self-registration for free trial account

Geneva, April 2010 Adam Kilgarriff 29 Lexical Computing Ltd Since 2003 Directors Adam Kilgarriff (UK), Pavel Rychly (Cz), Diana McCarthy (UK, since Oct 2009)‏ Main activities Sketch engine service Corpus development Research-led

Geneva, April 2010 Adam Kilgarriff 30 (demo)‏

Geneva, April 2010 Adam Kilgarriff 31 Evaluating word sketches 10 years Feedback Good but anecdotal Formal evaluation

Geneva, April 2010 Adam Kilgarriff 32 Goal Collocations dictionary Model: Oxford Collocations Dictionary Publication-quality Ask a lexicographer For 42 headwords For 20 best collocates per headwords “should we include this collocation in a published dictionary?”

Geneva, April 2010 Adam Kilgarriff 33 Sample of headwords Nouns verbs adjectives, random High (Top 3000)‏ N space solution opinion mass corporation leader V serve incorporate mix desire Adj high detailed open academic Mid ( )‏ N cattle repayment fundraising elder biologist sanitation V grieve classify ascertain implant Adj adjacent eldest prolific ill Low (10, ,000)‏ N predicament adulterer bake bombshell candy shellfish V slap outgrow plow traipse Adj neoclassical votive adulterous expandable

Geneva, April 2010 Adam Kilgarriff 34 Precision and recall We test precision Recall is harder How do we find all the collocations that the system should have found? Current work 200 collocates per headword Selected from All the corpora we have Various parameter settings Plus just-in-time evaluation for 'new' collocates

Geneva, April 2010 Adam Kilgarriff 35 Four languages, three families Dutch ANW, 102m-word lexicographic corpus English UKWaC, 1.5b web corpus Japanese JpWaC, 400m web corpus Slovene FidaPlus, 620m lexicographic corpus

Geneva, April 2010 Adam Kilgarriff 36 User evaluation Evaluate whole system Will it help with my task Eg preparing a collocations dictionary Contrast: developer evaluation Can I make the system better? Evaluate each module separately Current work

Geneva, April 2010 Adam Kilgarriff 37 Components Corpus NLP tools Segmenter, lemmatiser, POS-tagger Sketch grammar Statistics

Geneva, April 2010 Adam Kilgarriff 38 Practicalities Interface Good, Good-but Merge to good Maybe, Maybe-specialised, Bad Merge to bad For each language Two/three linguists/lexicographers If they disagree Don't use for computing performance

Geneva, April 2010 Adam Kilgarriff 39 Results Dutch 66% English71% Japanese87% Slovene71%

Geneva, April 2010 Adam Kilgarriff 40 Corpus evaluation Collocation-finding Typical corpus task Recall Hold all else constant Statistic, NLP tools, grammar Best results: best corpus (for collocation-finding)‏ Pomikalek: de-duplication

Geneva, April 2010 Adam Kilgarriff 41 Other topics Dante a new lexical database for English Corpus building (mostly from the web)‏ Instant corpora with WebBootCaT Bigger and better (English)‏ BiWeC and New Model Corpus Corpus Factory (many languages)‏ Corpus comparison, similarity, evaluation Statistics: collocations, keyword lists Word frequency lists Word senses and lexicography SADD: semi-automatic dictionary drafting

Geneva, April 2010 Adam Kilgarriff 42 Thank you

Geneva, April 2010 Adam Kilgarriff 43 Words and word senses automatic thesauruses words

Geneva, April 2010 Adam Kilgarriff 44 Words and word senses automatic thesauruses words manual thesauruses simple hierarchy is appealing homonyms

Geneva, April 2010 Adam Kilgarriff 45 Words and word senses automatic thesauruses words manual thesauruses simple hierarchy is appealing homonyms “aha! objects must be word senses”

Geneva, April 2010 Adam Kilgarriff 46 Problems Theoretical Practical

Geneva, April 2010 Adam Kilgarriff 47 Theoretical

Geneva, April 2010 Adam Kilgarriff 48

Geneva, April 2010 Adam Kilgarriff 49

Geneva, April 2010 Adam Kilgarriff 50 Wittgenstein Don’t ask for the meaning, ask for the use

Geneva, April 2010 Adam Kilgarriff 51 Practical

Geneva, April 2010 Adam Kilgarriff 52 Problems Practical a thesaurus is a tool if the tool organises words senses you must do WSD before you can use it WSD: state of the art, optimal conditions: 80%

Geneva, April 2010 Adam Kilgarriff 53 Problems “To use this tool, first replace one fifth of your input with junk”

Geneva, April 2010 Adam Kilgarriff 54 Avoid word senses

Geneva, April 2010 Adam Kilgarriff 55 Avoid word senses This word has three meanings/senses

Geneva, April 2010 Adam Kilgarriff 56 Avoid word senses This word has three meanings/senses This word has three kinds of use well founded empirical we can build on it