Challenges in Information Retrieval and Language Modeling Michael Shepherd Dalhousie University Halifax, NS Canada

Report of a Workshop James Allan, et al., “Challenges in Information Retrieval and Language Modeling”. Report of a Workshop held in the Center for Intelligent Information Retrieval, University of Massachusetts Amherst, September The following presentation is based on:

Long-Term Challenges LT Challenge 1 – Global Information Access –Satisfy human information needs through natural, efficient interaction with an automated system that leverages world-wide structured and unstructured data in any language Need –Massively distributed, multi-lingual retrieval systems –Techniques from distributed retrieval, data fusion, cross-lingual IR

Long-Term Challenges LT Challenge 2 – Contextual Retrieval –Combine search technologies and knowledge about query and user context into a single framework in order to provide the most “appropriate” answer for a user’s information needs Need –Context and query features to infer characteristics of the info need such as query type, answer type, answer level, task etc.

User Information Need Query User Profile Task Activity

Topics 1.Retrieval Models 2.Cross-Lingual information Retrieval 3.Web Search 4.User Modeling 5.Filtering, Topic Detection & Tracking, and classification 6.Summarization 7.Question Answering 8.Metasearch and distributed retrieval 9.Multimedia retrieval 10.Information extraction 11.Testbeds

Topics 1.Retrieval Models 2.Cross-Lingual information Retrieval 3.Web Search 4.User Modeling 5.Filtering, Topic Detection & Tracking, and classification 6.Summarization 7.Question Answering 8.Metasearch and distributed retrieval 9.Multimedia retrieval 10.Information extraction 11.Testbeds

User Modeling Much research over the past number of years has abstracted the user out of the retrieval problem But, in recent years, the rate of improvement of IR systems has slowed One reason may be that generic IR systems are “good-enough” for everyone but “never great” for anyone It is suggested that greater focus on the user will enable major advances in IR

How Do We Get Info About the User?

a priori –Ask the user a posteriori –Explicit Show user a document and ask them if it was relevant –Implicit Track what the user does –Web logs –Time spent reading a page

How Do We Model the User?

IR Technique –A vector of terms or features supplied by the user or drawn from documents deemed relevant to the user –May be static or adaptive Machine Learning Technique –An adaptive technique such as a neural net that “learns” the preferences of the user –Feature set selection is important

User Model as Filter Query representation Document representation Matching algorithm results Information need User Model as Filter

User Model as Query Document representation Matching algorithm results Information need User Model as Query

Integrating the User Model and the Query Query User Profile Modified Query Moving the Query within the Document Space

Integrating the User Model and the Query Document Space p q q'q'

Integrating the User Profile and the Query Document Space pq

Integrating the User Profile and the Query Document Space p q

Short-term/Long-term Interests Users’ interests change over time May have short-term interests but we do not want these to skew our models away from our long-term interests Particular focus is electronic news

Single task/Multiple tasks Most user models are built for a specific task, such as filtering news items looking for certain types of news Most people multi-task so we currently run multiple user models for different tasks for the same user Really would like to have a single model for multiple tasks

Filtering, Topic Detection & Tracking and Classification Some of these technologies have been adopted widely These topics are grouped together because they are similar technologies used in similar applications

Routing of and phone messages for Customer Relationship Management Message Message Routing System Service Department New Accounts Customer Complaints

Categorization of Trouble Tickets Trouble Ticket Ticket Routing System Trouble Category 1 Trouble Category 2 Trouble Category 3

Topic Detection News Item News Item Routing System Topic 1 Topic 2 Topic 3 New Topic

Topic Tracking Topic Sub-Topic

Topic Tracking WMD in Iraq Invasion of Iraq to locate WMD Cannot find WMD Bush and Kerry debate reasons for invading Iraq Election Day in USA Nov ‘02Mar ‘03 Jan ‘04Sept ‘04 Nov ‘04

Summarization Text summarization is an active field of research in both IR and Natural Language Processing (NLP) NLP is required for high-quality summarization IR summarization can provide access to large repositories of data in an efficient way IR summarization shares some basic techniques with indexing as both are concerned with identifying what a document is “about”

Summarization A summary can consist of: –A set of keywords or noun phrases –A set of sentences with “important” terms A summary can be about: –A single document (but not generally) –A set of documents –A web site

Summarization Each document is represented as a vector and tf.idf is used to determine the best terms Cluster the documents, create the centroids, and determine the best terms Sentences are given weights based on occurrence of terms and the associated tf.idf weights

Metasearch and Distributed Retrieval Retrieving and combining information from multiple sources: –Data fusion the combination of information from multiple sources that index an effectively common data set –Collection fusion or distributed retrieval the combination of info from multiple sources that index effectively disjoint data sets

Issues for Metasearch and DR Resource description Resource ranking Resource selection Searching Merging of results

Major Issue Resource description Resource ranking Resource selection Searching Merging of results Semantic Interoperability

Summary IR is no longer the domain of the “specialist” – everyone gets to play Drowning in information Next Generation IR tools must be dramatically better than what we have IR field must rethink its basic assumptions and evaluation methodologies because the ones that brought us to the level of success we have today will not be sufficient to reach the next level

Long-Term Challenges Global Information Access Contextual Retrieval