1. A Support Vector Method for Optimizing Average Precision
SIGIR 2007
Yisong Yue
Cornell University
In Collaboration With:
Thomas Finley, Filip Radlinski, Thorsten Joachims
(Cornell University)

2. Motivation Learn to Rank Documents
Optimize for IR performance measures
Mean Average Precision
Leverage Structural SVMs
[Tsochantaridis et al. 2005]

3. MAP vs Accuracy
Average precision is the average of the precision scores at the rank locations of each relevant document.
Ex: has average precision
Mean Average Precision (MAP) is the mean of the Average Precision scores for a group of queries.
A machine learning algorithm optimizing for Accuracy might learn a very different model than optimizing for MAP.
Ex: has average precision of about 0.64, but has a max accuracy of 0.8 vs 0.6 in above ranking.

4. Recent Related Work Greedy & Local Search
[Metzler & Croft 2005] – optimized for MAP, used gradient descent, expensive for large number of features.
[Caruana et al. 2004] – iteratively built an ensemble to greedily improve arbitrary performance measures.
Surrogate Performance Measures
[Burges et al. 2005] – used neural nets optimizing for cross entropy.
[Cao et al. 2006] – used SVMs optimizing for modified ROC-Area.
Relaxations
[Xu & Li 2007] – used Boosting with exponential loss relaxation

5. Conventional SVMs Input examples denoted by x (high dimensional point)
Output targets denoted by y (either +1 or -1)
SVMs learns a hyperplane w, predictions are sign(wTx)
Training involves finding w which minimizes
subject to
The sum of slacks upper bounds the accuracy loss

6. Conventional SVMs Input examples denoted by x (high dimensional point)
Output targets denoted by y (either +1 or -1)
SVMs learns a hyperplane w, predictions are sign(wTx)
Training involves finding w which minimizes
subject to
The sum of slacks upper bounds the accuracy loss

7. Adapting to Average Precision
Let x denote the set of documents/query examples for a query
Let y denote a (weak) ranking (each yij 2 {-1,0,+1})
Same objective function:
Constraints are defined for each incorrect labeling y’ over the set of documents x.
Joint discriminant score for the correct labeling at least as large as incorrect labeling plus the performance loss.

8. Adapting to Average Precision
Maximize
subject to
where
and
Sum of slacks upper bound MAP loss.
After learning w, a prediction is made by sorting on wTxi

9. Adapting to Average Precision
Maximize
subject to
where
and
Sum of slacks upper bound MAP loss.
After learning w, a prediction is made by sorting on wTxi

10. Too Many Constraints!
For Average Precision, the true labeling is a ranking where the relevant documents are all ranked in the front, e.g.,
An incorrect labeling would be any other ranking, e.g.,
This ranking has Average Precision of about 0.8 with (y,y’) ¼ 0.2
Exponential number of rankings, thus an exponential number of constraints!

11. Structural SVM Training
STEP 1: Solve the SVM objective function using only the current working set of constraints.
STEP 2: Using the model learned in STEP 1, find the most violated constraint from the exponential set of constraints.
STEP 3: If the constraint returned in STEP 2 is more violated than the most violated constraint the working set by some small constant, add that constraint to the working set.
Repeat STEP 1-3 until no additional constraints are added. Return the most recent model that was trained in STEP 1.
STEP 1-3 is guaranteed to loop for at most a polynomial number of iterations. [Tsochantaridis et al. 2005]

12. Illustrative Example Original SVM Problem Structural SVM Approach
Exponential constraints
Most are dominated by a small set of “important” constraints
Structural SVM Approach
Repeatedly finds the next most violated constraint…
…until set of constraints is a good approximation.

13. Illustrative Example Original SVM Problem Structural SVM Approach
Exponential constraints
Most are dominated by a small set of “important” constraints
Structural SVM Approach
Repeatedly finds the next most violated constraint…
…until set of constraints is a good approximation.

14. Illustrative Example Original SVM Problem Structural SVM Approach
Exponential constraints
Most are dominated by a small set of “important” constraints
Structural SVM Approach
Repeatedly finds the next most violated constraint…
…until set of constraints is a good approximation.

15. Illustrative Example Original SVM Problem Structural SVM Approach
Exponential constraints
Most are dominated by a small set of “important” constraints
Structural SVM Approach
Repeatedly finds the next most violated constraint…
…until set of constraints is a good approximation.

16. Finding Most Violated Constraint
Structural SVM is an oracle framework.
Requires subroutine to find the most violated constraint.
Dependent on formulation of loss function and joint feature representation.
Exponential number of constraints!
Efficient algorithm in the case of optimizing MAP.

17. Finding Most Violated Constraint
Observation
MAP is invariant on the order of documents within a relevance class
Swapping two relevant or non-relevant documents does not change MAP.
Joint SVM score is optimized by sorting by document score, wTx
Reduces to finding an interleaving
between two sorted lists of documents

18. Finding Most Violated Constraint
Start with perfect ranking
Consider swapping adjacent relevant/non-relevant documents ►

19. Finding Most Violated Constraint
Start with perfect ranking
Consider swapping adjacent relevant/non-relevant documents
Find the best feasible ranking of the non-relevant document ►

20. Finding Most Violated Constraint
Start with perfect ranking
Consider swapping adjacent relevant/non-relevant documents
Find the best feasible ranking of the non-relevant document
Repeat for next non-relevant document ►

21. Finding Most Violated Constraint
Start with perfect ranking
Consider swapping adjacent relevant/non-relevant documents
Find the best feasible ranking of the non-relevant document
Repeat for next non-relevant document
Never want to swap past previous non-relevant document ►

22. Finding Most Violated Constraint
Start with perfect ranking
Consider swapping adjacent relevant/non-relevant documents
Find the best feasible ranking of the non-relevant document
Repeat for next non-relevant document
Never want to swap past previous non-relevant document
Repeat until all non-relevant documents have been considered ►

23. Quick Recap SVM Formulation
SVMs optimize a tradeoff between model complexity and MAP loss
Exponential number of constraints (one for each incorrect ranking)
Structural SVMs finds a small subset of important constraints
Requires sub-procedure to find most violated constraint
Find Most Violated Constraint
Loss function invariant to re-ordering of relevant documents
SVM score imposes an ordering of the relevant documents
Finding interleaving of two sorted lists
Loss function has certain monotonic properties
Efficient algorithm

24. Experiments Used TREC 9 & 10 Web Track corpus.
Features of document/query pairs computed from outputs of existing retrieval functions.
(Indri Retrieval Functions & TREC Submissions)
Goal is to learn a recombination of outputs which improves mean average precision.

27. Moving Forward Approach also works (in theory) for other measures.
Some promising results when optimizing for NDCG (with only 1 level of relevance).
Currently working on optimizing for NDCG with multiple levels of relevance.
Preliminary MRR results not as promising.

28. Conclusions Principled approach to optimizing average precision.
(avoids difficult to control heuristics)
Performs at least as well as alternative SVM methods.
Can be generalized to a large class of rank-based performance measures.
Software available at