Learning to Suggest: A Machine Learning Framework for Ranking Query Suggestions Date: 2013/02/18 Author: Umut Ozertem, Olivier Chapelle, Pinar Donmez,

Slides:



Advertisements
Similar presentations
A Comparison of Implicit and Explicit Links for Web Page Classification Dou Shen 1 Jian-Tao Sun 2 Qiang Yang 1 Zheng Chen 2 1 Department of Computer Science.
Advertisements

Context-Sensitive Query Auto-Completion AUTHORS:NAAMA KRAUS AND ZIV BAR-YOSSEF DATE OF PUBLICATION:NOVEMBER 2010 SPEAKER:RISHU GUPTA 1.
Date: 2013/1/17 Author: Yang Liu, Ruihua Song, Yu Chen, Jian-Yun Nie and Ji-Rong Wen Source: SIGIR12 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang Adaptive.
ACM CIKM 2008, Oct , Napa Valley 1 Mining Term Association Patterns from Search Logs for Effective Query Reformulation Xuanhui Wang and ChengXiang.
Temporal Query Log Profiling to Improve Web Search Ranking Alexander Kotov (UIUC) Pranam Kolari, Yi Chang (Yahoo!) Lei Duan (Microsoft)
Psychological Advertising: Exploring User Psychology for Click Prediction in Sponsored Search Date: 2014/03/25 Author: Taifeng Wang, Jiang Bian, Shusen.
Term Level Search Result Diversification DATE : 2013/09/11 SOURCE : SIGIR’13 AUTHORS : VAN DANG, W. BRUCE CROFT ADVISOR : DR.JIA-LING, KOH SPEAKER : SHUN-CHEN,
Introduction to Information Retrieval
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
Date: 2012/8/13 Source: Luca Maria Aiello. al(CIKM’11) Advisor: Jia-ling, Koh Speaker: Jiun Jia, Chiou Behavior-driven Clustering of Queries into Topics.
Optimizing search engines using clickthrough data
Query Chains: Learning to Rank from Implicit Feedback Paper Authors: Filip Radlinski Thorsten Joachims Presented By: Steven Carr.
DOMAIN DEPENDENT QUERY REFORMULATION FOR WEB SEARCH Date : 2013/06/17 Author : Van Dang, Giridhar Kumaran, Adam Troy Source : CIKM’12 Advisor : Dr. Jia-Ling.
WSCD INTRODUCTION  Query suggestion has often been described as the process of making a user query resemble more closely the documents it is expected.
Toward Whole-Session Relevance: Exploring Intrinsic Diversity in Web Search Date: 2014/5/20 Author: Karthik Raman, Paul N. Bennett, Kevyn Collins-Thompson.
Vote Calibration in Community Question-Answering Systems Bee-Chung Chen (LinkedIn), Anirban Dasgupta (Yahoo! Labs), Xuanhui Wang (Facebook), Jie Yang (Google)
Query Dependent Pseudo-Relevance Feedback based on Wikipedia SIGIR ‘09 Advisor: Dr. Koh Jia-Ling Speaker: Lin, Yi-Jhen Date: 2010/01/24 1.
1 Learning User Interaction Models for Predicting Web Search Result Preferences Eugene Agichtein Eric Brill Susan Dumais Robert Ragno Microsoft Research.
Context-aware Query Suggestion by Mining Click-through and Session Data Authors: H. Cao et.al KDD 08 Presented by Shize Su 1.
Mining Query Subtopics from Search Log Data Date : 2012/12/06 Resource : SIGIR’12 Advisor : Dr. Jia-Ling Koh Speaker : I-Chih Chiu.
A Quality Focused Crawler for Health Information Tim Tang.
Evaluating Search Engine
Context-Aware Query Classification Huanhuan Cao 1, Derek Hao Hu 2, Dou Shen 3, Daxin Jiang 4, Jian-Tao Sun 4, Enhong Chen 1 and Qiang Yang 2 1 University.
Query Log Analysis Naama Kraus Slides are based on the papers: Andrei Broder, A taxonomy of web search Ricardo Baeza-Yates, Graphs from Search Engine Queries.
(ACM KDD 09’) Prem Melville, Wojciech Gryc, Richard D. Lawrence
Advisor: Hsin-Hsi Chen Reporter: Chi-Hsin Yu Date:
2008/06/06 Y.H.Chang Towards Effective Browsing of Large Scale Social Annotations1 Towards Effective Browsing of Large Scale Social Annotations WWW 2007.
Language Identification of Search Engine Queries Hakan Ceylan Yookyung Kim Department of Computer Science Yahoo! Inc. University of North Texas 2821 Mission.
Fast Webpage classification using URL features Authors: Min-Yen Kan Hoang and Oanh Nguyen Thi Conference: ICIKM 2005 Reporter: Yi-Ren Yeh.
Web Usage Mining with Semantic Analysis Date: 2013/12/18 Author: Laura Hollink, Peter Mika, Roi Blanco Source: WWW’13 Advisor: Jia-Ling Koh Speaker: Pei-Hao.
1 Context-Aware Search Personalization with Concept Preference CIKM’11 Advisor : Jia Ling, Koh Speaker : SHENG HONG, CHUNG.
Understanding and Predicting Graded Search Satisfaction Tang Yuk Yu 1.
When Experts Agree: Using Non-Affiliated Experts To Rank Popular Topics Meital Aizen.
CIKM’09 Date:2010/8/24 Advisor: Dr. Koh, Jia-Ling Speaker: Lin, Yi-Jhen 1.
Exploring Online Social Activities for Adaptive Search Personalization CIKM’10 Advisor : Jia Ling, Koh Speaker : SHENG HONG, CHUNG.
Understanding and Predicting Personal Navigation Date : 2012/4/16 Source : WSDM 11 Speaker : Chiu, I- Chih Advisor : Dr. Koh Jia-ling 1.
Presenter: Lung-Hao Lee ( 李龍豪 ) January 7, 309.
Implicit User Feedback Hongning Wang Explicit relevance feedback 2 Updated query Feedback Judgments: d 1 + d 2 - d 3 + … d k -... Query User judgment.
ON THE SELECTION OF TAGS FOR TAG CLOUDS (WSDM11) Advisor: Dr. Koh. Jia-Ling Speaker: Chiang, Guang-ting Date:2011/06/20 1.
Improving Cloaking Detection Using Search Query Popularity and Monetizability Kumar Chellapilla and David M Chickering Live Labs, Microsoft.
Date: 2012/4/23 Source: Michael J. Welch. al(WSDM’11) Advisor: Jia-ling, Koh Speaker: Jiun Jia, Chiou Topical semantics of twitter links 1.
Contextual Ranking of Keywords Using Click Data Utku Irmak, Vadim von Brzeski, Reiner Kraft Yahoo! Inc ICDE 09’ Datamining session Summarized.
Enhancing Cluster Labeling Using Wikipedia David Carmel, Haggai Roitman, Naama Zwerdling IBM Research Lab (SIGIR’09) Date: 11/09/2009 Speaker: Cho, Chin.
LANGUAGE MODELS FOR RELEVANCE FEEDBACK Lee Won Hee.
A Scalable Machine Learning Approach for Semi-Structured Named Entity Recognition Utku Irmak(Yahoo! Labs) Reiner Kraft(Yahoo! Inc.) WWW 2010(Information.
Algorithmic Detection of Semantic Similarity WWW 2005.
Jiafeng Guo(ICT) Xueqi Cheng(ICT) Hua-Wei Shen(ICT) Gu Xu (MSRA) Speaker: Rui-Rui Li Supervisor: Prof. Ben Kao.
Diversifying Search Results Rakesh AgrawalSreenivas GollapudiSearch LabsMicrosoft Research Alan HalversonSamuel.
Ranking Related Entities Components and Analyses CIKM’10 Advisor: Jia Ling, Koh Speaker: Yu Cheng, Hsieh.
A Classification-based Approach to Question Answering in Discussion Boards Liangjie Hong, Brian D. Davison Lehigh University (SIGIR ’ 09) Speaker: Cho,
Date: 2013/6/10 Author: Shiwen Cheng, Arash Termehchy, Vagelis Hristidis Source: CIKM’12 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang Predicting the Effectiveness.
Post-Ranking query suggestion by diversifying search Chao Wang.
More Than Relevance: High Utility Query Recommendation By Mining Users' Search Behaviors Xiaofei Zhu, Jiafeng Guo, Xueqi Cheng, Yanyan Lan Institute of.
Date: 2012/11/29 Author: Chen Wang, Keping Bi, Yunhua Hu, Hang Li, Guihong Cao Source: WSDM’12 Advisor: Jia-ling, Koh Speaker: Shun-Chen, Cheng.
Query Suggestions in the Absence of Query Logs Sumit Bhatia, Debapriyo Majumdar,Prasenjit Mitra SIGIR’11, July 24–28, 2011, Beijing, China.
Date: 2012/5/28 Source: Alexander Kotov. al(CIKM’11) Advisor: Jia-ling, Koh Speaker: Jiun Jia, Chiou Interactive Sense Feedback for Difficult Queries.
Identifying “Best Bet” Web Search Results by Mining Past User Behavior Author: Eugene Agichtein, Zijian Zheng (Microsoft Research) Source: KDD2006 Reporter:
Date: 2013/9/25 Author: Mikhail Ageev, Dmitry Lagun, Eugene Agichtein Source: SIGIR’13 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang Improving Search Result.
A Framework to Predict the Quality of Answers with Non-Textual Features Jiwoon Jeon, W. Bruce Croft(University of Massachusetts-Amherst) Joon Ho Lee (Soongsil.
1 Random Walks on the Click Graph Nick Craswell and Martin Szummer Microsoft Research Cambridge SIGIR 2007.
Predicting Short-Term Interests Using Activity-Based Search Context CIKM’10 Advisor: Jia Ling, Koh Speaker: Yu Cheng, Hsieh.
1 Context-Aware Ranking in Web Search (SIGIR 10’) Biao Xiang, Daxin Jiang, Jian Pei, Xiaohui Sun, Enhong Chen, Hang Li 2010/10/26.
Click Through Rate Prediction for Local Search Results
Improving Search Relevance for Short Queries in Community Question Answering Date: 2014/09/25 Author : Haocheng Wu, Wei Wu, Ming Zhou, Enhong Chen, Lei.
Personalized, Interactive Question Answering on the Web
A Large Scale Prediction Engine for App Install Clicks and Conversions
ISWC 2013 Entity Recommendations in Web Search
Struggling and Success in Web Search
Date : 2013/1/10 Author : Lanbo Zhang, Yi Zhang, Yunfei Chen
Date: 2012/11/15 Author: Jin Young Kim, Kevyn Collins-Thompson,
Presentation transcript:

Learning to Suggest: A Machine Learning Framework for Ranking Query Suggestions Date: 2013/02/18 Author: Umut Ozertem, Olivier Chapelle, Pinar Donmez, Emre Velipasaoglu Source: SIGIR12 Advisor: Jia-ling, Koh Speaker: Shun-Chen, Cheng

Outline Introduction Target Generation Suggestion Candidates Query logs Synthetic Suggestions Most Frequent Extensions Machine learning Features Learning Evaluation Conclusions

Introduction

Introduction Wisdom of Crowds Term semantic

The shortcomings of those methods: the utility of a query reformulation suggestion to the users search task is indirectly handled all co-occurrences in the query logs are treated equally models that are solely based on collocated queries in the past logs So, the goal is addressing these shortcomings.

Ways to address the problems: A utility model that takes into account positions of URLs that are common to result sets of the original query and the suggestions. An implicit task boundary method to model whether a following query is a continuation of the preceding queries Predict the utility of suggestion by lexical and result set features.

Outline Introduction Target Generation Suggestion Candidates Query logs Synthetic Suggestions Most Frequent Extensions Machine learning Features Learning Evaluation Conclusions

Target Generation Query co-occurrence : when two queries are manually1 issued by the same user within the same session. Session : all the user activity within a time window limited by 10 minutes of inactivity. The targets to be used in our machine learning model depend on Pr(q2, q1),namely the probability of query co- occurrence.

Two problem with Pr(q2, q1) : 1. query q2 is unrelated to q1 2. the result page associated with query q2 may not be useful So, given a query pair (q1, q2), the query q2 was a helpful reformulation of query q1 if and only if the following two conditions are satisfied: 1. The query q2 is a continuation of q1. If the query q2 is the beginning of a new task and has nothing to do with q1, we should not consider q2 to be a helpful reformulation for q1. 2. The query q2 has a positive utility, that is the search results returned for that query are useful to the user.

Utility of Reformulation Reformulations followed by a click are not always useful. For example Although co-occurrences like these lead to a click on the result set of the second query q2, they do not likely take the user to a destination URL that is not already directly accessible from the original query q1. bank of america bank ofamerica online facebook login facebook or

a reformulation to be useful only if it leads to a click on a URL that either is not existing in the search result page of q1 or that is ranked higher than that in the search result page of q1. When there was a click on q2, the reformulation is defined to be useful if Δ > 0. Dc : the set of clicked documents on the result page of q2 r(q2,d) : the rank of the given document d Dc for the given query q and returns +inf if the URL is not ranked.

It is important that q2 is a continuation of q1

Implicit task boundary detection q2 is a continuation of q1 (denoted by c = 1) q2 is useful (denoted by u = 1) implicitly conditioned on u = 1 But, this could lead to negative probabilities.

Outline Introduction Target Generation Suggestion Candidates Query logs Synthetic Suggestions Most Frequent Extensions Machine learning Features Learning Evaluation Conclusions

Query log for head queries in particular, we only consider the queries q2 which co-occured at least 3 times with q1. But it may limits the coverage of query suggestions and many queries with a low query log frequency will remain with a few or no suggestions.

Synthetic Suggestions The query big lots store does not have many good suggestions as compared to big lots, due to much lower frequency. So, the term store is decided to be dropped. segments the query into units and assigns importance weights to each unit. Afterwards, less important units of the query are dropped, or replaced with other contextually relevant units.

Most Frequent Extensions 20 most frequent queries that contains the original query as a suggestion source, but with one modification; we use a word boundary condition to bring these completions where the query is a full word

Outline Introduction Target Generation Suggestion Candidates Query logs Synthetic Suggestions Most Frequent Extensions Machine learning Features Learning Evaluation Conclusions

Lexical features

Result set features quality overlap

Outline Introduction Target Generation Suggestion Candidates Query logs Synthetic Suggestions Most Frequent Extensions Machine learning Features Learning Evaluation Conclusions

GBDT(Gradient Boosted Decision Trees)

Outline Introduction Target Generation Suggestion Candidates Query logs Synthetic Suggestions Most Frequent Extensions Machine learning Features Learning Evaluation Conclusions

Evaluation Query set: random sample of 912 fully anonymized queries issued on a commercial search engine Manual annotation: a group of eight professional search engine quality evaluators Annotation guidelines: 4-level ratings (excellent, good, fair, bad) Variation of the system: B: baseline, candidates are ranked according to mutual information 10M: ranked by Pr(q2 | q1, u = 1, c = 1); no machine learning model and only the suggestions from the query logs are considered. 10M-ML: ranked by GBDT model; only the suggestions from the query logs are considered. 10M-ML-SY: Same as 10M-ML; synthetic suggestions and most frequent specializations are also considered. 10MC,10MC-ML,10MC-ML-SY: Same as the above three models except that only consecutive pairs in a session.

Evaluation Offline human judgment Online Conducted an A/B test where we tested and compared the proposed system (10M-ML-SY implementation) against the baseline on live traffic. (3 is the best) =1 when labeled good or excellent =0 when labeled fair or bad

Evaluation depth = cut-off point Coverage: the ratio of the number of queries the system could bring suggestions for to the total number of test queries. Coverage, DCG, and Precision improvement compared to the baseline:

The system not only increases the chance of finding suggestions for a given query, it also ensures a significantly higher quality of these suggestions. The improvement compared to the baseline in the online test:

10M and 10MC-ML-SY do perform better grade distributions are more skewed towards better grades. the machine learning step has a more important contribution in bringing a lot more good quality suggestions without increasing the ratio of bad suggestions.

Outline Introduction Target Generation Suggestion Candidates Query logs Synthetic Suggestions Most Frequent Extensions Machine learning Features Learning Evaluation Conclusions

Conclusions Present an end-to-end query suggestion method that implements novel ideas such as incorporating usefulness of reformulations, an implicit session boundary model, and a machine learning model to further improve the suggestion relevance and be able to add more sources of suggestions beyond the co-occurrences in query logs. The click through rates on the online tests are very promising and we plan to extend this work with personalization and further diversification of suggestions.