Mining the Web to Create Minority Language Corpora Rayid Ghani Accenture Technology Labs - Research Rosie Jones Carnegie Mellon University Dunja Mladenic J. Stefan Institute, Slovenia

Who Needs a Language Specific Corpus? Language Technology Applications Language Modeling Speech Recognition Machine Translation Linguistic and Socio-Linguistic Studies Multilingual Retrieval

What Corpora are Available? Explicit, marked up corpora: Linguistic Data Consortium languages [Liebermann and Cieri 1998] Search Engines -- implicit language-specific corpora, European languages, Chinese and Japanese Excite - 12 languages Google - 25 languages AltaVista - 25 languages Lycos - 25 languages

BUT what about Slovenian? Or Tagalog? Or Tatar? You’re just out of luck!

The Human Solution Start from Yahoo->Slovenia… Crawl Search on the web, look at documents, modify query, analyze documents, modify query,… Repetitive, time-consuming, requires reasonable familiarity with the language

Task Given: 1 Document in Target Language 1 Other Document (negative example) Access to a Web Search Engine Create a Corpus of the Target Language quickly with no human effort

Algorithm Query GeneratorWWW Seed Docs Language Filter

Web Word Statistics Initial Docs Build Query Filter Relevant Non-Relevant Learning

Query Generation Examine current relevant and non-relevant documents to generate a query likely to find documents that ARE similar to the relevant ones and NOT similar to non-relevant ones A Query consists of m inclusion terms and n exclusion terms e.g +intelligence +web –military

Query Term Selection Methods Uniform (UN) – select k words randomly from the current vocabulary Term-Frequency (TF) – select top k words ranked according to their frequency Probabilistic TF (PTF) – k words with probability proportional to their frequency

Query Term Selection Methods RTFIDF – top k words according to their rtfidf scores Odds-Ratio (OR) – top k words according to their odds-ratio scores Probabilistic OR (POR) – select k words with probability proportional to their Odds- Ratio scores

3. Language Filter TextCat (van Noord’s implementation) trained on a handful of documents Manually evaluated through sampling 100 Slovenian documents and found to be 99% accurate Contains models for 60 languages

Evaluation Goal: Collect as many relevant documents as possible while minimizing the cost Cost Number of total documents retrieved from the Web Number of distinct Queries issued to the Search Engine Evaluation Measures Percentage of retrieved documents that are relevant Number of relevant documents retrieved per unique query

Experimental Setup Language: Slovenian Initial documents: 1 web page in Slovenian, 1 in English Search engine: Altavista

Results

Results – Precision at 3000 Percentage of Target Docs after 3000 Docs Retrieved

Results – Docs Per Query

Results - Summary In terms of documents: For lengths 1-3, Odds-Ratio works best In terms of queries: Odds-Ratio is consistently better than others Long queries are usually very precise but do not result in a lot of documents (low recall)

Results - Num of Docs Retrieved

Results – Num of Queries

Further Experiments Comparison to Altavista’s “More Like This” Better performance than Altavista’s feature Keywords Similar results when initializing with keywords instead of documents Other Languages Similar results with Croatian, Czech and Tagalog

Comparison with Altavista’s “More Like This” Feature Our Query- Generation mechanism with 5 inclusion and exclusion terms each using Odds-Ratio scoring outperforms Altavista’s “Find Similar Pages” Feature

Effect of Initial Documents No Visible differences until over a 1000 documents have been retrieved TypeLengthVocab Size 1Formal News Informal9074

Initializing with Keywords Obtaining entire documents in a language may not always be possible Used six different sets of 10 keywords obtained from Slovenian speakers as seed documents with Odds-Ratio 3 Each keyword set resulted in performance comparable to using entire documents

Other Languages Tried the same approach with Croatian Tagalog Czech Odds-Ratio outperformed other term- selection methods for all these languages

Other Languages MethodLanguageTarget Docs at 1000 total Docs TF-3Slovenian178 PTF-3Slovenian646 OR-3Slovenian835 TF-3Croatian39 PTF-3Croatian410 OR-3Croatian677 TF-3Czech385 PTF-3Czech451 OR-3Czech743 TF-3Tagalog440 PTF-3Tagalog359 OR-3Tagalog664

Other Languages – Sample Queries Typical queries that most commonly found positive documents in our experiments

Conclusions Successfully able to build corpora for minority languages (Slovenian, Croatian, Czech, Tagalog) using Web search engines Not sensitive to initial “seed” documents System and Corpora are/will be available at

Conclusions Automatic Query Generation is a cheap way to collect Language Corpora Odds-Ratio term selection works well Mostly independent of “seed” documents Can be “seeded” with a handful of keywords in a language

Ideas for Future Work Explore other Term-Selection methods From Language specific corpus to Topic Specific corpus as an alternative to focused spidering Finding documents matching a user profile – Personal Agent

Fixed Query Parameters Fix Query Lengths and Vary Term-Selection Methods Fix Term-Selection Methods and Vary Query Lengths Results (Ghani et al., SIGIR 2001): Odds-Ratio works well overall Long Queries are precise but with low recall

Algorithm 1. Initialization 2. Generate query terms from relevant and non- relevant documents 3. Retrieve document using the Query from Use the language filter to add the new document to the relevant or non-relevant set of documents. 5. Update frequencies and scores 6. Return to Step 2.

1. Initialize Given documents in the target and non-target languages Calculate various statistics over the words in each set