1. Learning to Rank From Pairwise Approach to Listwise Approach

2. Agenda  Introduction Ranking Problem  Ranking Pairwise Ranking (Brief) Listwise Ranking  Probability Models Permutation Probability Top one Probability  Results

3. Introduction  Construct model or method that learns to rank  Area of use: Anti Spam Product Rating Expert Finding...

4. Introduction  Ranking Problem – Document retrieval Ranking System Query: q Documents: {d_1, d_2,... d_n} d_1^i d_2^i d_3^i. d_n^i Ranking of documents

5. Pairwise Ranking Classification Model Relevance label Query Q D = {d, d,...., d } 12n  Classification of objects Instance : (d, d ) 12

6. Pairwise Ranking  Support Vector Machine Ranking SVM  Boosting RankBoost  Neural Network RankNet

7. Listwise Ranking Q = {q,...., q }Queries d = {d,...., d }Documents y = {y,...., y }Judgements x = {x,...., x }Features f (x )Score Func. z = {f(x ),...., f(x )} Scores (1) (m) (i) 1 n 1 n 1 n T = {(x, y )} (i) i=1 m (i)  Training (i) j 1 n m i=1 L (y, z ) (i) Listwise loss function

8. Listwise Ranking  Ranking New Query : q Associated Docs. :d Feature vectors:x Trained rank. Func.:f (x ) Rank documents in descending order ( i’ ) j’

9. Permutation Probability  f : s probability distribution pi = s = (s, s,.... s ) 12n P (pi) = s j=1 n o(S ) Pi(j) k=j n o(S ) Pi(k) Probability of pi given s

10. Top one Probability  Probability of being ranked on top of list s P (j) = P (p). s P(1)=j,p n Top one prob. of j

11. ListNet  Optimizing loss function Neural Network as model Gradient Descent as optimization alg. w = neural network f = f x = f (x ) z (f ) = {f (x ),..., f (x )} w w (i) j j ww 1 1

12. Results  TREC Web pages from.gov domain  OSHUMED Documents and queries on medicine  CSearch Data from commerciel search engine

13. Results  NDCG – Normalized Discounted Cumulative Gain Relevance judgements > 2  Korrekt – Delvist korrekt - Ukorrekt  MAP – Mean Average Precision Relevance judgements = 2  Korrekt - Ukorrekt

14. Results  NDCG on TREC

15. Results  NDCG on OSHUMED