CIKM Competition 2014 Second Place Solution

Slides:



Advertisements
Similar presentations
Pat Langley Computational Learning Laboratory Center for the Study of Language and Information Stanford University, Stanford, California
Advertisements

Using Large-Scale Web Data to Facilitate Textual Query Based Retrieval of Consumer Photos.
Florida International University COP 4770 Introduction of Weka.
Temporal Query Log Profiling to Improve Web Search Ranking Alexander Kotov (UIUC) Pranam Kolari, Yi Chang (Yahoo!) Lei Duan (Microsoft)
Overview of the TAC2013 Knowledge Base Population Evaluation: Temporal Slot Filling Mihai Surdeanu with a lot help from: Hoa Dang, Joe Ellis, Heng Ji,
1 Semi-supervised learning for protein classification Brian R. King Chittibabu Guda, Ph.D. Department of Computer Science University at Albany, SUNY Gen*NY*sis.
© 2014 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac.
Improving relevance prediction by addressing biases and sparsity in web search click data Qi Guo, Dmitry Lagun, Denis Savenkov, Qiaoling Liu
Paper presentation for CSI5388 PENGCHENG XI Mar. 23, 2005
Personal Name Classification in Web queries Dou Shen*, Toby Walker*, Zijian Zheng*, Qiang Yang**, Ying Li* *Microsoft Corporation ** Hong Kong University.
Bayesian Learning Rong Jin. Outline MAP learning vs. ML learning Minimum description length principle Bayes optimal classifier Bagging.
Lazy Learning k-Nearest Neighbour Motivation: availability of large amounts of processing power improves our ability to tune k-NN classifiers.
Ensemble Learning (2), Tree and Forest
For Better Accuracy Eick: Ensemble Learning
Data Mining Techniques
WEB FORUM MINING BASED ON USER SATISFACTION PAGE 1 WEB FORUM MINING BASED ON USER SATISFACTION By: Suresh Pokharel Information and Communications Technologies.
Transfer Learning From Multiple Source Domains via Consensus Regularization Ping Luo, Fuzhen Zhuang, Hui Xiong, Yuhong Xiong, Qing He.
Slide Image Retrieval: A Preliminary Study Guo Min Liew and Min-Yen Kan National University of Singapore Web IR / NLP Group (WING)
From Devices to People: Attribution of Search Activity in Multi-User Settings Ryen White, Ahmed Hassan, Adish Singla, Eric Horvitz Microsoft Research,
Language Identification of Search Engine Queries Hakan Ceylan Yookyung Kim Department of Computer Science Yahoo! Inc. University of North Texas 2821 Mission.
Zhangxi Lin ISQS Texas Tech University Note: Most slides are from Decision Tree Modeling by SAS Lecture Notes 6 Ensembles of Trees.
Detecting Semantic Cloaking on the Web Baoning Wu and Brian D. Davison Lehigh University, USA WWW 2006.
CIKM’09 Date:2010/8/24 Advisor: Dr. Koh, Jia-Ling Speaker: Lin, Yi-Jhen 1.
Semi-supervised Learning on Partially Labeled Imbalanced Data May 16, 2010 Jianjun Xie and Tao Xiong.
Presenter: Lung-Hao Lee ( 李龍豪 ) January 7, 309.
Shanda Innovations Context-aware Ensemble of Multifaceted Factorization Models for Recommendation Kevin Y. W. Chen.
Chapter 5. Adaptive Resonance Theory (ART) ART1: for binary patterns; ART2: for continuous patterns Motivations: Previous methods have the following problem:
Improving Classification Accuracy Using Automatically Extracted Training Data Ariel Fuxman A. Kannan, A. Goldberg, R. Agrawal, P. Tsaparas, J. Shafer Search.
Confidence-Aware Graph Regularization with Heterogeneous Pairwise Features Yuan FangUniversity of Illinois at Urbana-Champaign Bo-June (Paul) HsuMicrosoft.
Prediction of Molecular Bioactivity for Drug Design Experiences from the KDD Cup 2001 competition Sunita Sarawagi, IITB
Ensemble Learning Spring 2009 Ben-Gurion University of the Negev.
BAGGING ALGORITHM, ONLINE BOOSTING AND VISION Se – Hoon Park.
An Ensemble of Three Classifiers for KDD Cup 2009: Expanded Linear Model, Heterogeneous Boosting, and Selective Naive Bayes Members: Hung-Yi Lo, Kai-Wei.
Team Dogecoin: An Experience in Predicting Hospital Readmissions Acknowledgements The Problem Hospitals in the UK must keep track of which patients, once.
Konstantina Christakopoulou Liang Zeng Group G21
Post-Ranking query suggestion by diversifying search Chao Wang.
1 Accurate Product Name Recognition from User Generated Content Team: ISSSID Sen Wu, Zhanpeng Fang, Jie Tang Department of Computer Science Tsinghua University.
Classification Ensemble Methods 1
Context-Aware Query Classification Huanhuan Cao, Derek Hao Hu, Dou Shen, Daxin Jiang, Jian-Tao Sun, Enhong Chen, Qiang Yang Microsoft Research Asia SIGIR.
Competition II: Springleaf Sha Li (Team leader) Xiaoyan Chong, Minglu Ma, Yue Wang CAMCOS Fall 2015 San Jose State University.
Risk Solutions & Research © Copyright IBM Corporation 2005 Default Risk Modelling : Decision Tree Versus Logistic Regression Dr.Satchidananda S Sogala,Ph.D.,
Predicting Short-Term Interests Using Activity-Based Search Context CIKM’10 Advisor: Jia Ling, Koh Speaker: Yu Cheng, Hsieh.
Recommendation Systems By: Bryan Powell, Neil Kumar, Manjap Singh.
Cell Segmentation in Microscopy Imagery Using a Bag of Local Bayesian Classifiers Zhaozheng Yin RI/CMU, Fall 2009.
Transfer and Multitask Learning Steve Clanton. Multiple Tasks and Generalization “The ability of a system to recognize and apply knowledge and skills.
Coached Active Learning for Interactive Video Search Xiao-Yong Wei, Zhen-Qun Yang Machine Intelligence Laboratory College of Computer Science Sichuan University,
Tree and Forest Classification and Regression Tree Bagging of trees Boosting trees Random Forest.
Mustafa Gokce Baydogan, George Runger and Eugene Tuv INFORMS Annual Meeting 2011, Charlotte A Bag-of-Features Framework for Time Series Classification.
Vertical Search for Courses of UIUC Homepage Classification The aim of the Course Search project is to construct a database of UIUC courses across all.
Kaggle Winner Presentation Template. Agenda 1.Background 2.Summary 3.Feature selection & engineering 4.Training methods 5.Important findings 6.Simple.
Experience Report: System Log Analysis for Anomaly Detection
Supervised Time Series Pattern Discovery through Local Importance
Learning Triadic Influence in Large Social Networks
Cross-lingual Knowledge Linking Across Wiki Knowledge Bases
FRM: Modeling Sponsored Search Log with Full Relational Model
Introduction Feature Extraction Discussions Conclusions Results
Machine Learning Week 1.
CIKM Competition 2014 Second Place Solution
CLSciSumm-2018 What to submit Task Framework Task 1A Task 1B
Example: Academic Search
Weakly Learning to Match Experts in Online Community
Using decision trees and their ensembles for analysis of NIR spectroscopic data WSC-11, Saint Petersburg, 2018 In the light of morning session on superresolution.
Towards a Personal Briefing Assistant
iSRD Spam Review Detection with Imbalanced Data Distributions
Ryen White, Ahmed Hassan, Adish Singla, Eric Horvitz
CS412 – Machine Learning Sentiment Analysis - Turkish Tweets
Predicting Loan Defaults
CAMCOS Report Day December 9th, 2015 San Jose State University
THE ASSISTIVE SYSTEM SHIFALI KUMAR BISHWO GURUNG JAMES CHOU
Advisor: Dr.vahidipour Zahra salimian Shaghayegh jalali Dec 2017
Presentation transcript:

CIKM Competition 2014 Second Place Solution Team: FAndy Zhanpeng Fang, Jie Tang Department of Computer Science Tsinghua University

Task Given a sequence of query sessions Example Class1 Query1 – Class1 Query1 Title1 Class2 Query2 – Class2 Query2 Title2 Class2 Query2 Title3 Classify the class label of test queries

Challenges Encoding character Heterogeneous data User search behavior Only little prior knowledge can be used Heterogeneous data Query, title, session information User search behavior How to incorporate user search behavior to help the classification task? Unlabeled data How to utilize the large scale unlabeled data?

Result 0.9245(public score)/0.9245(private score) 2nd place winner Achieve in 4 days, from Sep. 27th to Sep. 30th EST 2018/11/22

Our Approach Feature extraction Learning models Ensemble Bag of words User search behavior Learning models Logistic regression Gradient boosted decision trees Factorization machines Ensemble

Feature Extraction – Bag of Words Given a query Q One gram, two grams, last gram of Q 0 -> 0.8452

Feature Extraction – Bag of Words Given a query Q One gram, two grams, last gram of Q 0 -> 0.8452 One gram, two grams of the clicked titles 0.8452 -> 0.9091, top 12 in the leaderboard!

Feature Extraction – Bag of Words Given a query Q One gram, two grams, last gram of Q 0 -> 0.8452 One gram, two grams of the clicked titles 0.8452 -> 0.9091, top 12 in the leaderboard! More bag of words features? Queries in the same session of Q? Titles in the same session of Q?

Feature Extraction – Bag of Words Given a query Q One gram, two grams, last gram of Q 0 -> 0.8452 One gram, two grams of the clicked titles 0.8452 -> 0.9091, top 12 in the leaderboard! More bag of words features? Queries in the same session of Q? Titles in the same session of Q? Performance decreases, 0.9091 -> 0.89x How to use the session information?

Feature Extraction – Search Behavior Given a query Q Macro features #total search, average length of clicked titles, length of the query 0.9091 -> 0.9105

Feature Extraction – Search Behavior Given a query Q Macro features #total search, average length of clicked titles, length of the query 0.9091 -> 0.9105 Session class features For each potential class C, calculate: #class C queries in the same session #class C queries in the next/previous query 0.9105 -> 0.9145

Feature Extraction – Search Behavior Same session’s queries can help but might contain noises

Feature Extraction – Search Behavior Same session’s queries can help but might contain noises Only use similar queries! Same session’s queries feature Bag of words feature for same session’s queries that are similar to the query Q Use Jaccard to measure similarity between queries 0.9145 -> 0.9182, utilizing the large scale unlabeled data!

Feature Extraction – Search Behavior Further add clicked titles of same session’s similar queries Performance decrease, 0.9182 -> 0.9176

Factorization Machine Learning Models Logistic regression Use the implementation of Liblinear Factorization machine Use the implementation of LibFM Gradient boosted decision trees Use the implementation of XGBoost Method Implementation Score on leaderboard Logistic Regression Liblinear 0.9182 Factorization Machine LibFM 0.9151 GBDT XGBoost 0.9225

Factorization Machine Ensemble Ensemble prediction results from different models by logistic regression Ensemble can significantly improves the performance Method Implementation Score on Validation Logistic Regression Liblinear 0.9182 Factorization Machine LibFM 0.9151 GBDT XGBoost 0.9225 Ensemble 0.9245

“Tricks” on how to win 2nd place Summary “Tricks” on how to win 2nd place Use unlabeled data Train multiple models Ensemble different results

Thank you! Questions?