Unsupervised Morpheme Analysis – Overview of Morpho Challenge 2007 in CLEF Mikko Kurimo, Mathias Creutz, Matti Varjokallio, Ville Turunen Helsinki University of Technology, Finland

My job at Helsinki: Multimodal Adaptive Informatics (Research Centre of Academy of Finland)

Continuous Speech Recognition Adaptive Natural Language Modelling Content Based Image and Video Retrieval Multimodal Interfaces: Proactive audio-visual information navigation, Effective multilingual interaction, Intermodal cross-over of semantics Research topics of MMI group

Motivation of Morpho Challenge To design statistical machine learning algorithms that discover which morphemes words consist of Follow-up to Morpho Challenge 2005 (segmentation of words into morphs) Morphemes are useful as vocabulary units for statistical language modeling in: Speech recognition, Machine translation, Information retrieval

The vocabulary problem Many applications require a large vocabulary: e.g. speech recognition, information retrieval, machine translation. Agglutinative and highly-inflected languages suffer from a severe vocabulary explosion We need more efficient representation units Unique words per corpus size Unique words (millions) Corpus size (million words)

Scientific objectives To learn of the phenomena underlying word construction in natural languages To discover approaches suitable for a wide range of languages and tasks To advance machine learning methodology

Morpho Challenge 2007 Part of the EU Network of Excellence PASCAL’s Challenge Program Organized in collaboration with CLEF Participation is open to all and free of charge Word sets are provided for: Finnish, English, German and Turkish Implement an unsupervised algorithm that discovers morpheme analysis of words in each language !

Thanks Thanks to all who made Morpho Challenge 2007 possible: PASCAL network, CLEF, Leipzig corpora collection Morpho Challenge organizing committee Morpho Challenge program committee Morpho Challenge participants Morpho Challenge evaluation team CLEF 2007 organizers!

Rules Morpheme analysis are submitted to the organizers and two different evaluations are made Competition 1: Comparison to a linguistic morpheme "gold standard“ Competition 2: Information retrieval experiments, where the indexing is based on morphemes instead of entire words.

Training data Word lists downloadable at our home page Each word in the list is preceded by its frequency Finnish: 3M sentences, 2.2M word types Turkish: 1M sentences, 620K word types German: 3M sentences, 1.3M word types English: 3M sentences, 380K word types Small gold standard sample available in each language

Examples of gold standard analyses English: baby-sitters baby_N sit_V er_s +PL Finnish: linuxiin linux_N +ILL German: zurueckzubehalten zurueck_B zu be halt_V +INF Turkish: kontrole kontrol +DAT

1. A new linguistic evaluation method Problem: The unsupervised morphemes may have arbitrary names, not the same as the ”real” linguistic morphemes, nor just subword strings Solution: Compare to the linguistic gold standard analysis by matching the morpheme- sharing word pairs Compute matches from a large random sample of word pairs where both words in the pair have a common morpheme

Evaluation measures F-measure = 1/(1/Precision + 1/Recall) Precision is the proportion of suggested word pairs that also have a morpheme in common according to the gold standard Recall is the proportion of word pairs sampled from the gold standard that also have a morpheme in common according to the suggested algorithm

Participants Delphine Bernhard, TIMC-IMAG, F (now moved to Darmstadt, D) Stefan Bordag, Univ. Leipzig, D Paul McNamee and James Mayfield, JHU, USA Daniel Zeman, Karlova Univ., CZ Christian Monson et al., CMU, USA Emily Pitler and Samarth Keshava, Univ. Yale, USA Morfessor MAP, Helsinki Univ. Tech, FI (Michael Tepper, Univ. Washington, USA)

Results: Finnish, 2.2M word types

Results: Turkish, 620K word types

Results: German, 1.3M word types

Results: English, 380K word types

2. Practical evaluation Real world application for morpheme analysis: Information Retrieval Analysis is needed to handle morphology (inflection, compounding) CLEF collections for Finnish, German and English

Data sets Finnish (CLEF 2004) 55K documents from articles in Aamulehti test queries and 23K binary relevance assessments English (CLEF 2005) 107K documents from articles in Los Angeles Times 94 and Glasgow Herald test queries and 20K binary relevance assessments German (CLEF 2003) 300K documents from short articles in Frankfurter Rundschau 94, Der Spiegel and SDA test queries and 23K binary relevance assessments

Reference methods Morfessor Baseline: our public code since 2002 Morfessor Categories-MAP: improved, public since 2006 dummy: no segmentation grammatical: gold standard segmentations –all: all alternatives included –first: only first alternative Porter: LEMUR's default stemmer Tepper: hybrid method based on Morfessor MAP

Evaluation 1/2 Words in the documents and queries were replaced by the submitted segmentations New words: –the CLEF collections contained words that were not in the original word list –additional segmentations were requested –if segmentation was not provided, words were indexed as such

Evaluation 2/2 LEMUR-toolkit ( ) Okapi BM25 retrieval, default parameter settings Okapi seems to handle common morphemes poorly => stoplist for most common ones (above a fixed frequency threshold) Also an alternative set of non-stoplisted results with TFIDF

Results: Finnish

Results: German

Results: English

Conclusions Analysis of new words important for Finnish, less so for German and English Porter stemming unbeaten for English (so far) Unsupervised morpheme analysis works very well for IR!

Future directions? Finnish, Turkish, English, German,...? Language modeling, Speech recognition, Information Retrieval,...? Venice, Budapest,...? PASCAL, CLEF,...?

Summary different unsupervised algorithms 8 participating research groups Evaluations for 4 languages (3 for IR) Good results in all languages and IR Full report and papers in the CLEF proceedings Details, presentations, links, info at website:

Acknowledgments Data from Leipzig and CLEF Gold standard providers in all languages! Workshop organization by CLEF Funding from PASCAL and Academy of Finland Competition participants!