Giorgos Giannopoulos (IMIS/”Athena” R.C and NTU Athens, Greece) Theodore Dalamagas (IMIS/”Athena” R.C., Greece) Timos Sellis (IMIS/”Athena” R.C and NTU Athens, Greece) Collaborative Ranking Function Training for Web Search Personalization National Technical University of Athens School of Electrical and Computer Engineering Divison of Computer Science Institute for the Management of Information Systems Athena Research Center

2 Intro How to personalize search results?

Intro How to personalize search results? Step 1. Implicit (from user log clicks) or explicit feedback can give you relevance judgments, i.e. irrelevant, partially relevant, relevant 3 irrelevant unjudged relevant unjudged irrelevant Partially relevant

Intro How to personalize search results? Step 1. Implicit (from user log clicks) or explicit feedback can give you relevance judgments, i.e. irrelevant, partially relevant, relevant Step 2. Extract features from query-result pairs Text similarity between query-result, 2. rank of result in Google, 3. domain of the result url irrelevant unjudged relevant unjudged irrelevant Partially relevant

Intro How to personalize search results? Step 1. Implicit (from user log clicks) or explicit feedback can give you relevance judgments, i.e. irrelevant, partially relevant, relevant Step 2. Extract features from query-result pairs. Step 3. Feed a ranking function (i.e. RSVM) with judgments and features. TRAINED RANKING FUNCTION 5 1. Text similarity between query-result, 2. rank of result in Google, 3. domain of the result url irrelevant unjudged relevant unjudged irrelevant Partially relevant

6 Intro How to personalize search results? Step 1. Implicit (from user log clicks) or explicit feedback can give you relevance judgments, i.e. irrelevant, partially relevant, relevant Step 2. Extract features from query-result pairs. Step 3. Feed a ranking function (i.e. RSVM) with judgments and features. Step 4. Re-rank the results using trained function irrelevant unjudged relevant unjudged Re-rank the results irrelevant Partially relevant relevant Partially relevant relevant irrelevant

7 Problem I Usually users search in more than one different areas of interest Example scenario 1: A phd student searches for papers on “web search ranking” They would prefer clicking on results with “acm” in title or url They would also prefer pdf results The same student searches for info about “samsung omnia cellphone” They would prefer results with “review”, “specs” or “hands on” in title or abstract They would also prefer results from blogs or forums or video results

8 Problem I Usually users search in more than one different areas of interest Example scenario 1: A phd student searches for papers on “web search ranking” They would prefer clicking on results with “acm” in title or url They would also prefer pdf results The same student searches for info about “samsung omnia cellphone” They would prefer results with “review”, “specs” or “hands on” in title or abstract They would also prefer results from blogs or forums or video results Training a single ranking function model for this user: Could favor video results while searching for papers Could favor pdf results (cellphone manual) while searching for reviews about a cellphone

9 Problem II Even users with different search behaviors may share a common search area Example scenario 2: User A is a phd student in IR and searches mostly for papers on their search area User B is a linguist and searches mostly for papers on their search area However, they could both be interested in new cellphones

10 Problem II Even users with different search behaviors may share a common search area Example scenario 2: User A is a phd student in IR and searches mostly for papers on their search area User B is a linguist and searches mostly for papers on their search area However, they could both be interested in new cellphones Training a common ranking function model for both users: Would probably give a better model for searching on cellphones Would probably give a worse model for the rest of the searches

11 Problem II Even users with different search behaviors may share a common search area Example scenario 2: User A is a phd student in IR and searches mostly for papers on their search area User B is a linguist and searches mostly for papers on their search area However, they could both be interested in new cellphones Training a single ranking function model for each user: Would not utilize each user’s behavior on common search areas Example: User A is familiar with “ a very informative site about cellphones, while user B is not Training a common ranking function on this particular search area, would favor “ in both users’ searches As a result, user B would become aware of this site and use it in future searches

12 Solution Train multiple ranking functions Each ranking function corresponds: Not to a single user Not to a group of users But to a topic area: A group of search results With similar content Collected from all users When re-ranking search results: Check which topic areas match with each new query Re-rank the query’s results according to the ranking functions trained for those topic areas

13 Our method (phase 1) Clustering on clicked results of all queries of all users Clicked results are more informative than all results or the query Partitional clustering (repeated bisections) Result representation: Term vector of size N (= the number of distinct terms in all results) Every feature is represented by a weight w (relating a result with a term) w depends on term’s tf and idf Title and abstract are considered as result’s text Use cosine similarity on term vectors as metric to compare two results Clustering criterion function maximizes Output Clusters containing (clicked) results with similar content (topic clusters)

14 Our method (phase 2) Cluster Indexing To be able to calculate the similarity of each new query with each cluster Extraction of title and abstract text of all results belonging to each cluster Use of this text as textual representation of the cluster Indexing of clusters-documents Output Inverted index on clusters’ textual representations

15 Our method (phase 3) Multiple ranking function training Use of Ranking SVM Each ranking function model F i is trained with clickstream data only from the corresponding cluster i Features used: Textual similarity between query and (title, abstract,URL) of the result Domain of the result (.com,.edu, e.t.c.) Rank in Google Special words (“blog”, “forum”, “wiki”, “portal”, e.t.c.) found in result’s title, abstract or url Features denoting textual similarity of each word with title/abstract/url URL suffix (.html,.pdf,.ppt) 100 most frequent words in all result documents of all searches Features denoting textual similarity of each word with title/abstract/url

16 Our method (phase 4) For each new query q: We calculate its textual similarity w qi with each cluster i using the index from phase 2 We produce N different rankings R qi with r qij being the rank of result j, for query q after re-ranking results using the ranking function trained on cluster i Final rank: In other words: w qi represents how similar is the content of cluster i with query q r qij gives the result rank when using the ranking function of cluster I We combine all produced rankings according to how similar they are to the query

17 Our method

18 Evaluation Logging mechanism on google search engine: Queries Result lists Clicked results User IPs Date and time Search topics: Gadgets Cinema Auto/Moto Life & Health Science Users: 10 phd students and researchers from our lab 1-3 search topics per user 2 months search period 671 queries First 501 queries used as training set Last 170 queries used as test set

19 Evaluation Comparison of our method (T3) with: Training a common ranking function for all users (T1) Training one ranking function per user (T2) Results: Average change in rank between our method and the base methods: Percentages of clicked results belonging to each cluster for each user

20 Conclusion-Future work First cut approach to the problem “Collaborative” training of ranking functions for personalization based on topic areas Encouraging results Preliminary experiments Extensions More extended experiments with (much) larger datasets Experiments to verify the homogeneity of topic clusters Experiments to verify the efficiency in very large datasets More performance measures (precision/recall) Topic clusters inference Clustering on feature vectors (and not on text) of results Use of pre-defined topic hierarchies and classification techniques for detecting topic areas (ODP) Dynamic switching between topic clusters by the users themselves