CMU Y2 Rosetta GnG Distillation Jonathan Elsas Jaime Carbonell

Rosetta GnG System Evolution Y1 System Y2 System Rank Learning Y3 Eval+ Y1: CMU initiated bulk of the work, essentially an indri-backed passage retrieval system with simple duplicate detection. IBM handled post-filtering, snippet composition, redundancy, etc. Y2: IBM took over main development of system, still using Indri primarily for document retrieval. We addressed specific challenges identified in the Y1 system -- how to utilize previously identified relevant docs & passages to tune the importance of different aspects of the templated query. Y2 Eval Y1 Eval

Distillation Challenges Multiple aspects to information need: Query arguments, Locations, Related Words Static expansion terms/phrases Bigrams, trigrams, term windows Named-Entity wildcards & constraints Occurrence of each of these in a document* is a “feature” indicating relevance of the document* to the information need. Question: How to best choose the weights for each feature? Challenges in retrieval identified in the first year: How to weight different aspects of the information need to optimize ranked retrieval performance * Or sentences, paragraphs, “nuggets”, etc.

Query Feature Construction DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING… Location : Middle East Equivalent terms: Mahmoud Abbas Abu Mazen President of the Palestinian National Authority Query Features: Unigram Features

Query Feature Construction DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING… Location : Middle East Equivalent terms: Mahmoud Abbas Abu Mazen President of the Palestinian National Authority Query Features: Bigram & Term Window Features

Query Feature Construction DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING… Location : Middle East Equivalent terms: Mahmoud Abbas Abu Mazen President of the Palestinian National Authority Query Features: Entity-Type Constrained Features

Query Feature Construction DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING… Location : Middle East Equivalent terms: Mahmoud Abbas Abu Mazen President of the Palestinian National Authority Co-ref features: Aliases, Nominal references (roles, descriptions), Pronominal references Query Features: Entity Co-reference Features

Query Feature Construction DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING… Location : Middle East Equivalent terms: Mahmoud Abbas Abu Mazen President of the Palestinian National Authority Query Features: Static Template-based expansion (unigram, bigram, term windows)

Query Feature Construction DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING… Location : Middle East Equivalent terms: Mahmoud Abbas Abu Mazen President of the Palestinian National Authority Just scratches the surface. Other features include (1) dynamic query expansion within corpus or using external corpora (2) document-structure based features (headline, body, slug) (3) SRL-based features *** EMPHASIZE THIS (4) predictive annotation features (5) features derived from translation/ASR artifacts Impractical to do an exhaustive search of the hypothesis space with as few as 4 or 5 features. Query Features: + potentially many more: structural features, PRF, & SRL annotations

Learning Approach to Setting Feature Weights Goal: Utilize existing relevance judgments to learn optimal weight setting Recently has become a hot research area in IR. “Learning to Rank”

Pair-wise Preference Learning Learning a document scoring function Treated as a classification problem on pairs of documents: Resulting scoring function is used as the learned document ranker. Correct Not just Documents --- passages, nuggets, documents, etc. Why pair-wise preference instead of list-wise or classifying rel/nonrel? (1) allows application of existing classification techniques (2) from a operational perspective, it may be easier/more intuitive to collect preference data rather than forcing users to put documents into some graded relevance scale (3) it works better than classifying rel/nonrel Incorrect

Committee Perceptron Algorithm Online algorithm (instance-at-a-time) Fast training, low memory requirements Ensemble method Selectively chooses N best hypotheses encountered during training “N heads are better than 1” approach Significant advantages over previous perceptron variants Many ways to combine output of hypotheses Voting, score averaging, hybrid approaches This is the focus of current research Our approach shows performance improvements over existing rank learning algorithms with a Significant reduction in training time -- 45 TIMES faster

Committee Perceptron Training Training Data Committee q, dR, dN Current Hypothesis R N

Committee Perceptron Training Training Data Committee q, dR, dN Current Hypothesis R N

Committee Perceptron Training Training Data Committee q, dR, dN Current Hypothesis R N If current hypothesis better than worst: Replace worst hypothesis in committee Otherwise: discard current hypothesis Update current hypothesis to better classify this training example

Committee Perceptron Training Training Data Committee q, dR, dN Current Hypothesis R N If current hypothesis better than worst: Replace worst hypothesis in committee Otherwise: discard current hypothesis Update current hypothesis to better classify this training example

Committee Perceptron Performance Comparable or better performance than two state-of-the-art batch leanring algorithms Added Bonus: more than 45 times faster training time than RankSVM

Committee Perceptron Learning Curves Committee/Ensemble approach a better solution faster than existing perceptron variants

Next Steps (in progress) Integrate current work with GALE GnG system Document ranking is the obvious first step Passage ranking poses additional challenges Both will be addressed this year Implement feature-based query generation framework for Rosetta GnG System Extend & improve performance of our rank learning algorithm

Future Work Investigate application of preference learning in Utility system, adapting to real-time user preference feedback.