1. CMU Y2 Rosetta GnG Distillation
Jonathan Elsas
Jaime Carbonell

2. Rosetta GnG System Evolution
Y1 System
Y2 System
Rank
Learning
Y3 Eval+
Y1: CMU initiated bulk of the work, essentially an indri-backed passage retrieval system with simple duplicate detection. IBM handled post-filtering, snippet composition, redundancy, etc.
Y2: IBM took over main development of system, still using Indri primarily for document retrieval. We addressed specific challenges identified in the Y1 system -- how to utilize previously identified relevant docs & passages to tune the importance of different aspects of the templated query.
Y2 Eval
Y1 Eval

3. Distillation Challenges
Multiple aspects to information need:
Query arguments, Locations, Related Words
Static expansion terms/phrases
Bigrams, trigrams, term windows
Named-Entity wildcards & constraints
Occurrence of each of these in a document* is a “feature” indicating relevance of the document* to the information need.
Question: How to best choose the weights for each feature?
Challenges in retrieval identified in the first year: How to weight different aspects of the information need to optimize ranked retrieval performance
* Or sentences, paragraphs, “nuggets”, etc.

4. Query Feature Construction
DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING…
Location : Middle East
Equivalent terms:
Mahmoud Abbas
Abu Mazen
President of the Palestinian National Authority
Query Features:
Unigram Features

5. Query Feature Construction
DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING…
Location : Middle East
Equivalent terms:
Mahmoud Abbas
Abu Mazen
President of the Palestinian National Authority
Query Features:
Bigram & Term Window Features

6. Query Feature Construction
DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING…
Location : Middle East
Equivalent terms:
Mahmoud Abbas
Abu Mazen
President of the Palestinian National Authority
Query Features:
Entity-Type Constrained Features

7. Query Feature Construction
DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING…
Location : Middle East
Equivalent terms:
Mahmoud Abbas
Abu Mazen
President of the Palestinian National Authority
Co-ref features: Aliases, Nominal references (roles, descriptions), Pronominal references
Query Features:
Entity Co-reference Features

8. Query Feature Construction
DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING…
Location : Middle East
Equivalent terms:
Mahmoud Abbas
Abu Mazen
President of the Palestinian National Authority
Query Features:
Static Template-based expansion
(unigram, bigram, term windows)

9. Query Feature Construction
DESCRIBE THE ACTIONS OF [Mahmoud Abbas] DURING…
Location : Middle East
Equivalent terms:
Mahmoud Abbas
Abu Mazen
President of the Palestinian National Authority
Just scratches the surface. Other features include (1) dynamic query expansion within corpus or using external corpora
(2) document-structure based features (headline, body, slug)
(3) SRL-based features *** EMPHASIZE THIS
(4) predictive annotation features
(5) features derived from translation/ASR artifacts
Impractical to do an exhaustive search of the hypothesis space with as few as 4 or 5 features.
Query Features:
+ potentially many more: structural
features, PRF, & SRL annotations

10. Learning Approach to Setting Feature Weights
Goal: Utilize existing relevance judgments to learn optimal weight setting
Recently has become a hot research area in IR. “Learning to Rank”

11. Pair-wise Preference Learning
Learning a document scoring function
Treated as a classification problem on pairs of documents:
Resulting scoring function is used as the learned document ranker.
Correct
Not just Documents --- passages, nuggets, documents, etc.
Why pair-wise preference instead of list-wise or classifying rel/nonrel?
(1) allows application of existing classification techniques
(2) from a operational perspective, it may be easier/more intuitive to collect preference data rather than
forcing users to put documents into some graded relevance scale
(3) it works better than classifying rel/nonrel
Incorrect

12. Committee Perceptron Algorithm
Online algorithm (instance-at-a-time)
Fast training, low memory requirements
Ensemble method
Selectively chooses N best hypotheses encountered during training
“N heads are better than 1” approach
Significant advantages over previous perceptron variants
Many ways to combine output of hypotheses
Voting, score averaging, hybrid approaches
This is the focus of current research
Our approach shows performance improvements over existing rank learning algorithms with a Significant reduction in training time
-- 45 TIMES faster

13. Committee Perceptron Training
Training Data
Committee
q, dR, dN
Current Hypothesis R N

14. Committee Perceptron Training
Training Data
Committee
q, dR, dN
Current Hypothesis R N

15. Committee Perceptron Training
Training Data
Committee
q, dR, dN
Current Hypothesis R N
If current hypothesis better than worst:
Replace worst hypothesis in committee
Otherwise: discard current hypothesis
Update current hypothesis to better classify this training example

16. Committee Perceptron Training
Training Data
Committee
q, dR, dN
Current Hypothesis R N
If current hypothesis better than worst:
Replace worst hypothesis in committee
Otherwise: discard current hypothesis
Update current hypothesis to better classify this training example

17. Committee Perceptron Performance
Comparable or better performance than two state-of-the-art batch leanring algorithms
Added Bonus: more than 45 times faster training time than RankSVM

18. Committee Perceptron Learning Curves
Committee/Ensemble approach a better solution faster than existing perceptron variants

19. Next Steps (in progress) Integrate current work with GALE GnG system
Document ranking is the obvious first step
Passage ranking poses additional challenges
Both will be addressed this year
Implement feature-based query generation framework for Rosetta GnG System
Extend & improve performance of our rank learning algorithm

20. Future Work
Investigate application of preference learning in Utility system, adapting to real-time user preference feedback.