Active Learning and Collaborative Filtering

Slides:



Advertisements
Similar presentations
Google News Personalization: Scalable Online Collaborative Filtering
Advertisements

Diversity Maximization Under Matroid Constraints Date : 2013/11/06 Source : KDD’13 Authors : Zeinab Abbassi, Vahab S. Mirrokni, Mayur Thakur Advisor :
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
LEARNING INFLUENCE PROBABILITIES IN SOCIAL NETWORKS Amit Goyal Francesco Bonchi Laks V. S. Lakshmanan University of British Columbia Yahoo! Research University.
Combining Classification and Model Trees for Handling Ordinal Problems D. Anyfantis, M. Karagiannopoulos S. B. Kotsiantis, P. E. Pintelas Educational Software.
A P2P REcommender system based on Gossip Overlays (PREGO) ‏ R.Baraglia, P.Dazzi M.Mordacchini, L.Ricci A P2P REcommender system based on Gossip Overlays.
Dong Liu Xian-Sheng Hua Linjun Yang Meng Weng Hong-Jian Zhang.
Preference Elicitation Partial-revelation VCG mechanism for Combinatorial Auctions and Eliciting Non-price Preferences in Combinatorial Auctions.
Search Engines Information Retrieval in Practice All slides ©Addison Wesley, 2008.
Jeff Howbert Introduction to Machine Learning Winter Collaborative Filtering Nearest Neighbor Approach.
The Wisdom of the Few A Collaborative Filtering Approach Based on Expert Opinions from the Web Xavier Amatriain Telefonica Research Nuria Oliver Telefonica.
Evaluating Search Engine
Memory-Based Recommender Systems : A Comparative Study Aaron John Mani Srinivasan Ramani CSCI 572 PROJECT RECOMPARATOR.
Optimizing Online Auction Bidding Strategies with Genetic Programming Ekaterina “Kate” Smorodkina.
The AutoSimOA Project Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School OR49 Sept 07 A 3 year, EPSRC funded project in collaboration with.
Collaborative Ordinal Regression Shipeng Yu Joint work with Kai Yu, Volker Tresp and Hans-Peter Kriegel University of Munich, Germany Siemens Corporate.
Copyright © 2010 Pearson Education, Inc. Publishing as Prentice Hall Statistics for Business and Economics 7 th Edition Chapter 9 Hypothesis Testing: Single.
Experimental Evaluation
CONTENT-BASED BOOK RECOMMENDING USING LEARNING FOR TEXT CATEGORIZATION TRIVIKRAM BHAT UNIVERSITY OF TEXAS AT ARLINGTON DATA MINING CSE6362 BASED ON PAPER.
Learning From Mistakes—A Comprehensive Study on Real World Concurrency Bug Characteristics Shan Lu, Soyeon Park, Eunsoo Seo and Yuanyuan Zhou Appeared.
A Study of Computational and Human Strategies in Revelation Games 1 Noam Peled, 2 Kobi Gal, 1 Sarit Kraus 1 Bar-Ilan university, Israel. 2 Ben-Gurion university,
Item-based Collaborative Filtering Recommendation Algorithms
Performance of Recommender Algorithms on Top-N Recommendation Tasks
Chapter 10 Hypothesis Testing
1 Reading Report 9 Yin Chen 29 Mar 2004 Reference: Multivariate Resource Performance Forecasting in the Network Weather Service, Martin Swany and Rich.
Discovering Interesting Subsets Using Statistical Analysis Maitreya Natu and Girish K. Palshikar Tata Research Development and Design Centre (TRDDC) Pune,
Performance of Recommender Algorithms on Top-N Recommendation Tasks RecSys 2010 Intelligent Database Systems Lab. School of Computer Science & Engineering.
Tag Clouds Revisited Date : 2011/12/12 Source : CIKM’11 Speaker : I- Chih Chiu Advisor : Dr. Koh. Jia-ling 1.
Distributed Networks & Systems Lab. Introduction Collaborative filtering Characteristics and challenges Memory-based CF Model-based CF Hybrid CF Recent.
Fundamentals of Data Analysis Lecture 4 Testing of statistical hypotheses.
Philosophy of IR Evaluation Ellen Voorhees. NIST Evaluation: How well does system meet information need? System evaluation: how good are document rankings?
A Comparative Study of Search Result Diversification Methods Wei Zheng and Hui Fang University of Delaware, Newark DE 19716, USA
Introduction to Inferential Statistics. Introduction  Researchers most often have a population that is too large to test, so have to draw a sample from.
Group Recommendations with Rank Aggregation and Collaborative Filtering Linas Baltrunas, Tadas Makcinskas, Francesco Ricci Free University of Bozen-Bolzano.
A Hybrid Recommender System: User Profiling from Keywords and Ratings Ana Stanescu, Swapnil Nagar, Doina Caragea 2013 IEEE/WIC/ACM International Conferences.
1 Applying Collaborative Filtering Techniques to Movie Search for Better Ranking and Browsing Seung-Taek Park and David M. Pennock (ACM SIGKDD 2007)
Exploring Online Social Activities for Adaptive Search Personalization CIKM’10 Advisor : Jia Ling, Koh Speaker : SHENG HONG, CHUNG.
Experimental Evaluation of Learning Algorithms Part 1.
1 Nasser Alsaedi. The ultimate goal for any computer system design are reliable execution of task and on time delivery of service. To increase system.
Online Learning for Collaborative Filtering
Measuring Association Rules Shan “Maggie” Duanmu Project for CSCI 765 Dec 9 th 2002.
Evaluation of Recommender Systems Joonseok Lee Georgia Institute of Technology 2011/04/12 1.
1 Collaborative Filtering & Content-Based Recommending CS 290N. T. Yang Slides based on R. Mooney at UT Austin.
EigenRank: A ranking oriented approach to collaborative filtering By Nathan N. Liu and Qiang Yang Presented by Zachary 1.
1 Privacy-Enhanced Collaborative Filtering Privacy-Enhanced Personalization workshop July 25, 2005, Edinburgh, Scotland Shlomo Berkovsky 1, Yaniv Eytani.
Intelligent DataBase System Lab, NCKU, Taiwan Josh Jia-Ching Ying, Eric Hsueh-Chan Lu, Wen-Ning Kuo and Vincent S. Tseng Institute of Computer Science.
Performance Measures. Why to Conduct Performance Evaluation? 2 n Evaluation is the key to building effective & efficient IR (information retrieval) systems.
Pairwise Preference Regression for Cold-start Recommendation Speaker: Yuanshuai Sun
Collaborative Filtering via Euclidean Embedding M. Khoshneshin and W. Street Proc. of ACM RecSys, pp , 2010.
Chapter 7 Introduction to Sampling Distributions Business Statistics: QMIS 220, by Dr. M. Zainal.
Chapter 9: Introduction to the t statistic. The t Statistic The t statistic allows researchers to use sample data to test hypotheses about an unknown.
Fundamentals of Data Analysis Lecture 4 Testing of statistical hypotheses pt.1.
Hypothesis Testing. Steps for Hypothesis Testing Fig Draw Marketing Research Conclusion Formulate H 0 and H 1 Select Appropriate Test Choose Level.
The Wisdom of the Few Xavier Amatrian, Neal Lathis, Josep M. Pujol SIGIR’09 Advisor: Jia Ling, Koh Speaker: Yu Cheng, Hsieh.
Collaborative Filtering - Pooja Hegde. The Problem : OVERLOAD Too much stuff!!!! Too many books! Too many journals! Too many movies! Too much content!
Collaborative Filtering With Decoupled Models for Preferences and Ratings Rong Jin 1, Luo Si 1, ChengXiang Zhai 2 and Jamie Callan 1 Language Technology.
Trust-aware Recommender Systems
A Collaborative Quality Ranking Framework for Cloud Components
Recommendation in Scholarly Big Data
Evaluation of IR Systems
Preface to the special issue on context-aware recommender systems
Location Recommendation — for Out-of-Town Users in Location-Based Social Network Yina Meng.
Collaborative Filtering Nearest Neighbor Approach
M.Sc. Project Doron Harlev Supervisor: Dr. Dana Ron
Introduction to Instrumentation Engineering
by Octavio A Ramirez and J. Scott Shonkwiler
Movie Recommendation System
Probabilistic Latent Preference Analysis
Retrieval Performance Evaluation - Measures
Presentation transcript:

Active Learning and Collaborative Filtering Valdemaras Repsys Francesco Ricci Mehdi Elahi Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

Introduction Recommender Systems (RSs): Propose interesting items to the user Predict user’s preferences (ratings) on new items exploiting ratings on known items Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

Active Learning with Collaborative Filtering If during the learning process – rating prediction - some preferences are not available, the system can actively and selectively ask the user their value Collaborative Filtering: A technique used to predict the ratings exploiting ratings given by the users to items. Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

Objectives To simulate the evolution of a RS and its performance exploiting rating elicitation strategies, i.e., algorithms for choosing the items to be presented to the user for rating To understand the benefits and drawbacks of different strategies with respect to various measures of a recommender system effectiveness (e.g. Mean Absolute Error, precision, ranking quality, or coverage) To study whether the rating elicitation strategy must take into account the size and the state of the rating database. Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

Pure Strategies Popularity: chooses the most popular items, hence it is more likely that a request of such a rating will really increase the size of the rating database. Binary Prediction: tries to predict what items the user has experienced, to maximize the probability that the user be able to rate the item. Highest Predicted: the best recommendations are more likely to have been experienced by the user and their ratings reveal what the user likes (default strategy for RSs). Lowest Predicted: reveals what the user does not like, but can actually collect a few ratings, since the user is unlikely to have experienced all the items that he does not like. Highest and Lowest Predicted: combines "highest predicted" and "lowest predicted" strategies. Random: selects randomly the items to ask - a baseline strategy used for comparison. Variance: collects the opinion of the user on items with more diverse ratings - assuming that these ratings are more useful. Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

Partially Randomized Strategies In a partially randomized strategy : The list of ratings returned by a pure strategy S is modified, introducing some random ratings, simulating the free addition of some rating values not explicitly requested by the system but known by the user. For instance, we note that if S is the highest predicted strategy, there are cases where no rating predictions can be computed by the RS for the user u, and hence S would not be able to identify the ratings to request. This happens when u is a new user and none of his ratings are known. In this case the randomized version of this strategy generates purely random ratings to ask to the user. Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

Evaluation Methodology K known T test X unknown elicit Movielens No. of users: 943 No. of items: 1682 No. of ratings: 100K Time span: 1997 - 1998 Netflix No. of users: 480189 No. of items: 17770 No. of ratings: 100M* Time span: 1998 – 2005 *We used 1st 100K ratings Datasets are partitioned into three subsets: K: contains the rating values that are considered to be known by the system at a certain point in time. X: contains the rating values that are considered to be known by the users but not to the system. These ratings are incrementally elicited, i.e., their values are transferred into K if the system asks them to the (simulated) users. T: contains the ratings that are never elicited and are used only to test the recommendation effectiveness after the system has acquired the new elicited ratings. Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

b)Partially randomized strategies Evaluation: MAE Mean Absolute Error (MAE): Measures the average absolute deviation of the predicted rating from the user's true rating: a)Pure strategies b)Partially randomized strategies Where rui stands for a real rating value and r^ui represents the predicted value for user u and item i. T is the testing data set. Results are shown for Movielens data set – with Netflix data there are similar results. Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

b) Partially randomized strategies Evaluation: Precision a) Pure strategies b) Partially randomized strategies Precision: percentage of the items with rating values (in T ) equal to 4 or 5 in the top 10 recommended items. Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

b) Partially randomized strategies Evaluation: Coverage a) Pure strategies b) Partially randomized strategies Coverage: proportion of the full set of items over which the system can form predictions or make recommendations. Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

b) Partially randomized strategies Evaluation: NDCG Normalized Discounted Cumulative Gain (NDCG): The recommendations for u are sorted according to the predicted rating values, then DCGu is defined as: a) Pure strategies b) Partially randomized strategies Where IDCGu stands for the maximum possible value of DCGu, i.e., obtained if the recommended items were ordered by decreasing value of their true ratings Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

Conclusion This work is a first attempt to evaluate a set of strategies that are either used or could be used in a RS to elicit ratings. Our results will help selecting the right strategy for a given effectiveness metric - in fact, there is no single best strategy, among those that are evaluated, that dominates the others for all the evaluation measures. The random strategy is the best for NDCG. The lo-high predicted is the best for MAE and precision. In addition: Prediction-based strategies neither address the problem of new users, nor of new items. Popularity and variance strategies are able to select items for new users, but cannot select items that have no ratings. Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy

Thank you! Free University of Bolzano Bolzano, Italy Active Learning and Experimental Design Workshop May 16, 2010, Sardinia, Italy