1. Learning to rank
11/04/2017

4. Ranking of job candidates

6. The ranking task The set of o1…n candidates and the query q is given
The pairs {oi,q} can be described by a (rich) feature set
Rank the o1…n items according to their relevance to the qurery q!
The output is an ordering of o1…n instances (i.e. a structure)

7. Learning to rank
Train database:
Model:

8. Learning2rank vs. regression
Baseline solution: use rank as a target regression value
Warning: The ranking is relative in a a given set! And the concrete values are not important, only the order counts!
Normalisation among sets is crucial:
pl. f(q1,o1,18) = f(q2, o2,72) = 1

9. Learning to rank – top K case
Train database:
Only a few relevant item is know:
Model:

10. Learning2rank vs. classification
Relevant/non-relevant binary classification?
Relevant is a local concept as we are looking for the most relevant ones! (What happens if a binary classifier predicts only non-relevant for each item?)
We’re looking for a relative ordering and not a global function.

11. Evaluation metric Kendall tau: topK case: reciproc ranking = 1/rank,
where rank is the place of the first relevant prediction
MRR: mean of the reciproc rank over a set of q,O pairs

12. Learning2rank approaches
Pointwise approach
Forget the original set memberships, global regressziós (full ranking) or classfication (topK case)
Pairwise approach
Take each pair inside an O. Define a binary classification task for predicting whether o1 or o2 is preferred (all-vs-all)
Listwise approach
Learn the ranking directly. A pair {q,O} is an instance.

13. Pairwise learning to rank

14. SVMrank

15. SVMrank

16. Listwise learning to rank

17. ListMLE – Plackett Luce Modell π is an ordering of O
π-1(i) is the item in the ith position
s a the score for a particular item
P is a distribution
The decreasing(increasing) ordering according to s has the
greatest(lowest) probability Ps

18. ListMLE
Training is a maximum likelihood parameter estimation (MLE) of the Plackett-Luce modell alapján (θ):

19. ListMLE
topK case
Yi is the set of relevant items for the ith query ?

20. Notes
Pair- and listwise approaches considerably outperform the pointwise approach
The pair- and listwise approaches are competitive
Number of training instances:
pairwise |Q||O|2
listwise |Q|

21. Summary Learning to rank Pointwise approach Pairwise approach
Full ranking OR
Only topK relevant item is known
Pointwise approach
Regression
Relevant/non-relevant classification
Pairwise approach
SVMrank
Listwise approach
ListMLE