Personalized Search Cheng Cheng (cc2999) Department of Computer Science Columbia University A Large Scale Evaluation and Analysis of Personalized Search Strategies

Your company slogan Contents Introduction 1 Evaluation Framework 2 Experiment Results 3 Conclusion 4

Your company slogan Introduction What is Personalized Search ? “Personalized Search is the fine-tuning of search results and advertising based on an individual’s preferences, information and other factors.”[Steve Johnson] “Personalized Search is the fine-tuning of search results and advertising based on an individual’s preferences, information and other factors.”[Steve Johnson] Personalized Search Engines Google ( Google ( Yahoo’s Myweb ( Yahoo’s Myweb (

Your company slogan Personalization Strategies Person-level Re-ranking Based on Historical Clicks P-Click : P-Click : Person-level Re-ranking Based on User Interests L-Profile : L-Profile : S-Profile : S-Profile : LS-Profile : Group-level Re-ranking G-Click : G-Click :

Your company slogan Evaluation Framework Step 1 Download the top 50 search Results from MSN search Engine for the test query. We denote the downloaded Web pages with U and deonte the rank list with R1. Step 2 Compute a personalized score for each web page in U using personalization strategy and generate a new rank list R2. (Five different strategies are given in the last slide.) Step 3 Combine the rank lists of R1 and R2 using Borda’ ranking fusion method and sort the page with combined rankings. The final rank list is personalized search result list denoted with R. Step 4 Using the measurement in the next slide to evaluate the personalization performance on R.

Your company slogan Evaluation Metrics Ranking Scoring (evaluate the accuracy of personalized search) The expected utility of a ranked list of web pages: The expected utility of a ranked list of web pages: The final rank scoring (reflecting the utility of all test queries) The final rank scoring (reflecting the utility of all test queries) Average Rank (evaluate the quality of personalized search) The average rank of a query s: The average rank of a query s: The final average rank on the test query set S: The final average rank on the test query set S:

Your company slogan Experiment Data Dataset (large scale) Randomly sample 10,000 distinct users from the MSN query logs for 12 days in August These users and their click-through logs are extracted as our dataset. Training Set and Testing Set training set: the log data of the first 11 days. training set: the log data of the first 11 days. testing set: the log data of the last day. testing set: the log data of the last day.

Your company slogan Experiment Results (1) Overall Performance of Strategies 1) 1) Click-based personalization methods G-Click and P-Click perform better than the method WEB on the whole. 2) Profile-based methods L-Profile, S-Profile, and LS-Profile perform less well. The not–optimal query is the query on which users select not only the top results returned by MSN search engine

Your company slogan Experiment Results (2) Overall Performance of Strategies 3) T 3) Though L-Profile, S-Profile, and LS-Profile methods improve the search accuracy on many queries, they also harm the performance on more queries, which makes them perform worse on average.

Your company slogan Click Entropy Click entropy is a direct indication of query click variation. ClickEntroy(q) is the click entropy of query q : P(p|q) is the percentage of the clicks on web page p among all the clicks on q : Smaller click entropy means that the majorities of users agree with each other on a small number of web pages. In such a case, there is no need to do personalization.

Your company slogan Experiment Results (3) Performance on Different Click Entropies 1)T 1)The improvement of the personalized search performance increases when the click entropy of query becomes larger, especially when ClickEntropy ≥ )All these results indicate that on the queries with small click entropy (which means that these queries are less ambiguous), the personalization is insufficient and thus personalization is unnecessary.

Your company slogan Analysis of Profile-based Strategies Profile-based personalization strategies perform less optimally, which contradict the existing investigation. This is probably caused by the rough implementation of our strategies. Method LS-Profile is more stable than methods L-Profile and S- Profile. In other words, both long-term and short-term search contexts are very important to personalize search results. The combination of the two type of search context can make the prediction of real user information need more reliable.

Your company slogan Conclusion All proposed methods have significant improvements over common web search on queries with large click entropy. Personalized search has different effectiveness on different queries and thus not all queries should be handled in the same manner. Click-based personalization strategies work well. And they are straightforward and stable. The appropriate combination of long term profile and short term profile can be more reliable than solely using either of them.

Your company slogan Thank you !