1 Alexander Gelbukh Moscow, Russia

2 Mexico

3 Computing Research Center (CIC), Mexico

4 Chung-Ang University, Korea Electronic Commerce and Internet Application Lab

5 Special Topics in Computer Science The Art of Information Retrieval Alexander Gelbukh

6 Information Retrieval In a huge amount of poorly structured information find the information that you need when you dont know exactly what you need or cant explain it The Web User information need Ranking

7

8

9 Information Retrieval In a huge amount of poorly structured information find the information that you need when you dont know exactly what you need or cant explain it The Web User information need Ranking

10 Importance Knowledge: the main treasure of man Web: Repository? Cemetery of information! Natural language and multimedia information oPoorly structured, badly written Corporate and organizational document bases oSenate speeches: Mexico oMedical data collections oCorporate memory. Microsoft knowledge base Future: data explosion increasing importance

11 Perspectives Corporations: corporate databases Organizations: document bases Government oEuropean Union multilingual problem oThe same in Asia Academy oLots of open research topics oWeb topics oComputational Linguistics topics oIntelligent technologies, AI

12 Textbook

13 Contents 1.Introduction 2.Modeling 3.Retrieval Evaluation 4.Query Languages 5.Query Operations 6.Text and Multimedia Languages and Properties 7.Text Operations 8.Indexing and Searching 9.Parallel and Distributed IR 10.User Interfaces and Visualization 11.Multimedia IR: Models and Languages 12.Multimedia IR: Indexing and Searching 13.Searching the Web 14.Libraries and Bibliographical Systems 15.Digital Libraries

14 Calendar 1.September 18Chapter 1 Introduction 2.September 25Chapter 2 Modeling 3.October 2Chapter 3 Retrieval Evaluation 4.October 9Chapter 4 Query Languages 5.October 16Chapter 5 Query Operations October 23 – midterm exam 6.October 30Chapter 6 Text and Multimedia Languages... 7.November 6Chapter 7 Text Operations 8.November 13Chapter 8 Indexing and Searching 9.November 20Chapter 10 User Interfaces and Visualization 10.November 27Chapter 13 Searching the Web 11.December 4Chapter 14 Libraries and Bibliographical Systems 12.December 11Chapter 15 Digital Libraries December – final exam

15 Class structure Main course: Information Retrieval Discussion of previous chapter. Questions I briefly present a new chapter Research seminar: Natural Language Processing Discussion of previous paper. Questions. oIdentification of possible research topics Presentation of a new paper or current work Discussion and questions Goal: publications!

16 Natural Language Processing Research Seminar

17 What CL is about Computers to process natural language text Understand Generate Search Organize Translate … Useful in IR

18 Methods No: text as a stream of letters oBrute force statistics oSimplified heuristics (ex.: Porter) Yes: attention to language rules oLinguistically motivated approaches oKnowledge-based approaches oCorpus-based approaches

19 What IR is about Classical IR: find words? Concepts! Question answering Summarization Clustering … Take language seriously

20 Text representations for IR Represent the retrieval features oStrings stems (lexemes), synsets, phrases. oWomen woman, lady, female oOld men and women old woman Structured representation of text oNetwork of related events and entities oEnables logical inference

21 CL tasks useful in IR Morphology (stemming) POS / Word dense disambiguation Word relatedness Anaphora resolution Parsing and semantics (phrase search) Synonymic rephrasing Translation etc… Each one a whole science in itself

22 Morphology Q: pig T: piggish Simple: stemming opiggish pig- Lexeme: set of word forms osame stem can give different words opigment not pig; piny pine, not pin Dictionary/corpus-based methods oLearning; dictionary management

23 Part of Speech Disambiguation Q: oil well T: He did very well Q: what is an are? T: They are nice Important for English, Chinese. Less important for other types Perhaps not so helpful directly, but is necessary for most other tasks Usually statistical / heuristic methods

24 Word Sense Disambiguation Q: bank account T: on the beautiful banks of Han river... bill: document, banknote, law, ax, peak, Gates... Very frequent, almost any word in text Statistical & dictionary methods International competitions

25 Word relatedness Q: female T: woman (women) oSynonyms. Subtypes/super-types oDictionaries. WordNet. Similarity. Lesk. Q: Korea T: Seoul oOther linguistic relationships (e.g., part) oReal-world relationships (facts) Q: Clinton T: Lewinsky oStatistical co-occurrence (MI)

26 Anaphora resolution Q: Awards of Prof. Han T: Prof. Han said... He did... IBM awarded him... oFrequency oPhrases, co-occurrence, summarization, inference, translation Heuristic (Mitkov) and knowledge- based methods Other types of co-reference

27 Parsing, semantics Q: Awards of Prof. Han T1: Prof. Han among many other prizes has several IBM awards T2: Mr. Kang has an award Prof. Han does not know of Understanding of text oRich structured representation Better phrase search; question answering, summarization,...

28 Synonymic rephrasing, reasoning Q: experienced computer scientists T: Prof. Han has been programming for many years and awarded an IBM award Requires good syntactic and semantic analysis Knowledge-based methods

29 Multilingual access Q: T: We sell excellent yoghurt. Продаем йогурт. Se vende rico yogur. oSearch multilingual collections Europe: dozens of official languages of EU oIf you dont know how to say it in English Dictionaries, bilingual corpora,...

30 Tasks are entangled Many of CL tasks require other tasks oMorphology syntax semantics Many CL tasks form circles oparsing WSD parsing oI see a wild cat with a telescope (tripod?) Can be done quick-and-dirty (?) oFighting for last %s oZipf law: 20% of men drink 80% of beer

31 Tools and infrastructure Analysis tools oTasks, methods Dictionaries and grammars oTypes, structure oAutomatic acquisition Corpora oCorpora analysis tools and methods

32 Possible tasks WSD to help IR Clustering + summarization in IR results Anaphora and coreference resolution to help IR Multilingual IR Applications to Korean... a lot of others

33 Reading Textbooks oManning & Schütze, Allen, Jurafsky, Hausser,... CICLing proceedings Computational Linguistics Google, ResearchIndex

34 Questions Who expects to publish? Who will make a presentation at the next seminar?

35 Thank you! Till September 18