1. Language Model in Turkish IR Melih Kandemir F. Melih Özbekoğlu Can Şardan Ömer S. Uğurlu

2. Outline Indexing problem and proposed solution Previous Work System Architecture Language Modeling Concept Evaluation of the System Conclusion

3. Indexing Problem “A Language Modeling Approach to Information Retrieval” Jay M. Ponte and. W. Bruce Croft, 1998 Indexing model is important at probabilistic retrieval model Current models do not lead to improved retrieval results

4. Indexing Problem Failure because of unwarranted assumptions: 2-Poisson model –“elite” documents N-Poisson model –Mixture of more than 2 Poission distributions

5. Proposed Solution Retrieval based on probabilistic language modeling Language model refers to probabilistic distribution that captures statistical regularities of the generation of language A language model is inferred for each document

6. Proposed Solution Estimate probability of generating the query Documents are ranked according to these probabilities Users have a reasonable idea of terms tf, idf are integral parts of language model

7. Previous Work Robertson–Sparck Jones model and Croft–Harper model –They focus on relevance Fuhr integrated indexing and retrieval models. –Used statistics as heuristics Wong and Yao used utility theory and information theory

8. Previous Work Kalt’s approach is the most similar –Maximum likelihood estimator is used –Collection statistics are integral parts of the model –Documents are members of language classes

9. System Overview Application Server Index DB (PostgreSQL) LM-Search JDBC UI Document Repository Query Evaluator Indexer USER Different Resultsets

10. System Architecture Document Repository Stemming & Term Selection No Stemming First 5 Lemmatiser Inverted Index Generation tf.idf Language Model Index DB

11. tf.idf vs. Language model Different Resultsets GUI for seeing differences between results tf.idf LM tf.idf

12. Vocabulary Extraction No stemmer – Turkish is aggluntinative – Expectation: low precision First 5 characters – As effective as more complex solutions Lemmatiser: – Expectation: high precision. – Zemberek2 (MPL license) Open Source Software Java Interface, easy to use Find stems of the words First valid stem will be used, Word sense disambiguation (using wordnet or POS) may be added in the future Stemming & Term Selection No Stemming First 5 Lemmatiser

13. Database Index contains fields for both methods Table can be evaluated in any form Hybrid Index Structure Stemmed First 5 Lemmatiser 3 Indices tf.idf LanguageModel Inverted Index PostgreSQL database server Implementation Index DB

14. Language Modeling : Inverted Index Implementation An example inverted index for m terms : t1  cf t d1P(t1|Md1)tf 1 d 1 dnP(t1|Mdn)tf 1 d n t2  cf t d3P(t2|Md3)tf 2 d 2 dpP(t2|Mdp)tf 2 d p tm  cf t d4P(t4|Md4)tf m d 4 dkP(tm|Mdk)tf m d k … … … … If a document does not contain term then probability can be calculated using cf t f t = mean term frequency = mean probability of t in documents containing it cf t =frequency of t in all documents

15. The Baseline Approach : tf.idf We will use the traditional tf.idf term weighting approach as the baseline model Robertson’s tf score Standard idf score

16. Language Modeling : Definition An alternative approach to indexing and retrieval Definition of Language Model: A probability distribution that captures the statistical regularities of the generation of language Intuition Behind : Users have a reasonable idea of terms that are likely to occur in documents of interest and will choose query terms that distinguish these documents from others in the collection

17. Language Modeling : The Approach The following assumptions are not made : Term distributions in the documents are parametric Documents are members of pre-defined classes “Query generation probability” rather than “Probability of relevance”

18. Language Modeling : The Approach P(t | M d ) : Probability that the term t is generated by the language model of document M d

19. Language Modeling : Theory Maximum likelihood estimate of the probability of term t under the term distribution for document d: tf (t,d) : raw term frequency in document d dl d : total number of terms in the document

20. Language Modeling : Theory An additional more robust estimate from a larger amount of data : p avg : Mean probability of term t in documents containing it df t : Number of documents that contain term t

21. Language Modeling : Theory The risk function : : Mean term frequency of term t in documents which contains it.

22. Language Modeling : The Ranking Formula Let the probability of term t being produced by document d given the document model Md : The probability of producing Q for a given document model M d is :

23. Language Modeling : Inverted Index Implementation An example inverted index for m terms : t1  cf t d1P(t1|Md1)dnP(t1|Mdn) t2  cf t d3P(t2|Md3)dpP(t2|Mdp) tm  cf t d4P(t4|Md4)dkP(tm|Mdk) … … … …

24. Evaluation Perform recall/precision experiments – Recall/precision results – Non-interpolated average precision – Precision figures for the top N documents For several values of N R-Precision

25. Other Metrics Compare the baseline (tf.idf) results to our language model. – Percent Change between two result sets – I / D I : count of queries performance improved D : count of queries performance changed

26. Document Repository Milliyet (2001-2005) XML file ( 1.1 GB ) 408304 news Ready for indexing XML Schema......(FIXME) Document Source

27. Summary Indexing and stemming – Zemberek2 lemmatiser – Java environment Data – News archive from 2001 to 2005, from Milliyet Evaluation – Methods will be compared according to performance over recall/precision values

28. Conclusion First language modelling approach to Turkish IR The LM approach – Non-parametric – Less assumptions – Relaxed Expected a better performance than baseline tf.idf method

29. Thanks for listening … Any Questions?