1 Recommender Systems and Collaborative Filtering Jon Herlocker Assistant Professor School of Electrical Engineering and Computer Science Oregon State.

Slides:



Advertisements
Similar presentations
Recommender System A Brief Survey.
Advertisements

Recommender Systems & Collaborative Filtering
Google News Personalization: Scalable Online Collaborative Filtering
Item Based Collaborative Filtering Recommendation Algorithms
1 Evaluation Rong Jin. 2 Evaluation  Evaluation is key to building effective and efficient search engines usually carried out in controlled experiments.
Collaborative Filtering Sue Yeon Syn September 21, 2005.
COMP423 Intelligent Agents. Recommender systems Two approaches – Collaborative Filtering Based on feedback from other users who have rated a similar set.
1 RegionKNN: A Scalable Hybrid Collaborative Filtering Algorithm for Personalized Web Service Recommendation Xi Chen, Xudong Liu, Zicheng Huang, and Hailong.
COLLABORATIVE FILTERING Mustafa Cavdar Neslihan Bulut.
CNI 2003/Herlocker, Jung, and Webster1 Collaborative Filtering: Possibilities for Digital Libraries Jon Herlocker Janet Webster Seikyung Jung Oregon State.
1 Learning User Interaction Models for Predicting Web Search Result Preferences Eugene Agichtein Eric Brill Susan Dumais Robert Ragno Microsoft Research.
Web Search - Summer Term 2006 II. Information Retrieval (Basics Cont.)
Recommender Systems Aalap Kohojkar Yang Liu Zhan Shi March 31, 2008.
CS345 Data Mining Recommendation Systems Netflix Challenge Anand Rajaraman, Jeffrey D. Ullman.
Retrieval Evaluation. Brief Review Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.
Computing Trust in Social Networks
CS276B Web Search and Mining Winter 2005 Lecture 5 (includes slides borrowed from Jon Herlocker)
Recommender systems Ram Akella February 23, 2011 Lecture 6b, i290 & 280I University of California at Berkeley Silicon Valley Center/SC.
Sparsity, Scalability and Distribution in Recommender Systems
Retrieval Evaluation. Introduction Evaluation of implementations in computer science often is in terms of time and space complexity. With large document.
Recommender systems Ram Akella November 26 th 2008.
Personalized Ontologies for Web Search and Caching Susan Gauch Information and Telecommunications Technology Center Electrical Engineering and Computer.
Tomer Sagi and Avigdor Gal Technion - Israel Institute of Technology Non-binary Evaluation for Schema Matching ER 2012 October 2012, Florence.
CS 277: Data Mining Recommender Systems
Chapter 12 (Section 12.4) : Recommender Systems Second edition of the book, coming soon.
Item-based Collaborative Filtering Recommendation Algorithms
Mining and Summarizing Customer Reviews
Performance of Recommender Algorithms on Top-N Recommendation Tasks
Graph-RAT Overview By Daniel McEnnis. 2/32 What is Graph-RAT  Relational Analysis Toolkit  Database abstraction layer  Evaluation platform  Robustly.
Performance of Recommender Algorithms on Top-N Recommendation Tasks RecSys 2010 Intelligent Database Systems Lab. School of Computer Science & Engineering.
Distributed Networks & Systems Lab. Introduction Collaborative filtering Characteristics and challenges Memory-based CF Model-based CF Hybrid CF Recent.
1 Information Filtering & Recommender Systems (Lecture for CS410 Text Info Systems) ChengXiang Zhai Department of Computer Science University of Illinois,
EMIS 8381 – Spring Netflix and Your Next Movie Night Nonlinear Programming Ron Andrews EMIS 8381.
GAUSSIAN PROCESS FACTORIZATION MACHINES FOR CONTEXT-AWARE RECOMMENDATIONS Trung V. Nguyen, Alexandros Karatzoglou, Linas Baltrunas SIGIR 2014 Presentation:
Google News Personalization: Scalable Online Collaborative Filtering
Toward the Next generation of Recommender systems
Collaborative Information Retrieval - Collaborative Filtering systems - Recommender systems - Information Filtering Why do we need CIR? - IR system augmentation.
LATENT SEMANTIC INDEXING Hande Zırtıloğlu Levent Altunyurt.
EigenRank: A Ranking-Oriented Approach to Collaborative Filtering IDS Lab. Seminar Spring 2009 강 민 석강 민 석 May 21 st, 2009 Nathan.
Collaborative Filtering  Introduction  Search or Content based Method  User-Based Collaborative Filtering  Item-to-Item Collaborative Filtering  Using.
Badrul M. Sarwar, George Karypis, Joseph A. Konstan, and John T. Riedl
Evaluation of Recommender Systems Joonseok Lee Georgia Institute of Technology 2011/04/12 1.
1 Collaborative Filtering & Content-Based Recommending CS 290N. T. Yang Slides based on R. Mooney at UT Austin.
Improving Recommendation Lists Through Topic Diversification CaiNicolas Ziegler, Sean M. McNee,Joseph A. Konstan, Georg Lausen WWW '05 報告人 : 謝順宏 1.
Recommender Systems Debapriyo Majumdar Information Retrieval – Spring 2015 Indian Statistical Institute Kolkata Credits to Bing Liu (UIC) and Angshul Majumdar.
Data Mining: Knowledge Discovery in Databases Peter van der Putten ALP Group, LIACS Pre-University College LAPP-Top Computer Science February 2005.
Recommender Systems. Recommender Systems (RSs) n RSs are software tools providing suggestions for items to be of use to users, such as what items to buy,
Collaborative Filtering Zaffar Ahmed
Pairwise Preference Regression for Cold-start Recommendation Speaker: Yuanshuai Sun
Google News Personalization Big Data reading group November 12, 2007 Presented by Babu Pillai.
SCHOOL OF INFORMATION UNIVERSITY OF MICHIGAN si.umich.edu Author(s): Rahul Sami, 2009 License: Unless otherwise noted, this material is made available.
Collaborative Filtering via Euclidean Embedding M. Khoshneshin and W. Street Proc. of ACM RecSys, pp , 2010.
Personalization Services in CADAL Zhang yin Zhuang Yuting Wu Jiangqin College of Computer Science, Zhejiang University November 19,2006.
Identifying “Best Bet” Web Search Results by Mining Past User Behavior Author: Eugene Agichtein, Zijian Zheng (Microsoft Research) Source: KDD2006 Reporter:
Online Evolutionary Collaborative Filtering RECSYS 2010 Intelligent Database Systems Lab. School of Computer Science & Engineering Seoul National University.
User Modeling and Recommender Systems: recommendation algorithms
Text Information Management ChengXiang Zhai, Tao Tao, Xuehua Shen, Hui Fang, Azadeh Shakery, Jing Jiang.
Item-Based Collaborative Filtering Recommendation Algorithms Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl GroupLens Research Group/ Army.
Reputation-aware QoS Value Prediction of Web Services Weiwei Qiu, Zhejiang University Zibin Zheng, The Chinese University of HongKong Xinyu Wang, Zhejiang.
Collaborative Filtering - Pooja Hegde. The Problem : OVERLOAD Too much stuff!!!! Too many books! Too many journals! Too many movies! Too much content!
Analysis of massive data sets Prof. dr. sc. Siniša Srbljić Doc. dr. sc. Dejan Škvorc Doc. dr. sc. Ante Đerek Faculty of Electrical Engineering and Computing.
COMP423 Intelligent Agents. Recommender systems Two approaches – Collaborative Filtering Based on feedback from other users who have rated a similar set.
Recommender Systems 11/04/2017
Data Mining: Concepts and Techniques
Recommender Systems & Collaborative Filtering
CS728 The Collaboration Graph
Methods and Metrics for Cold-Start Recommendations
Adopted from Bin UIC Recommender Systems Adopted from Bin UIC.
Collaborative Filtering Nearest Neighbor Approach
ITEM BASED COLLABORATIVE FILTERING RECOMMENDATION ALGORITHEMS
Presentation transcript:

1 Recommender Systems and Collaborative Filtering Jon Herlocker Assistant Professor School of Electrical Engineering and Computer Science Oregon State University Corvallis, OR (also President, MusicStrands, Inc.)

2 Personalized Recommender Systems and Collaborative Filtering (CF)

3 Outline The recommender system space Pure collaborative filtering (CF) CF algorithms for prediction Evaluation of CF algorithms CF in web search (if time)

4 Recommender Systems Help people make decisions –Examples: Where to spend attention Where to spend money Help maintain awareness –Examples: New products New information In both cases –Many options, limited resources

5 Stereotypical Integrator of RS Has: Large product (item) catalog –With product attributes Large user base –With user attributes (age, gender, city, country, …) Evidence of customer preferences –Explicit ratings (powerful, but harder to elicit) –Observations of user activity (purchases, page views, s, prints, …)

6 UsersItems Observed preferences The RS Space Item-Item Links User-User Links Links derived from similar attributes, similar content, explicit cross references Links derived from similar attributes, explicit connections (Ratings, purchases, page views, laundry lists, play lists)

7 UsersItems Observed preferences Individual Personalization Item-Item Links User-User Links

8 UsersItems Observed preferences Classic CF Item-Item Links User-User Links In the end, most models will be hybrid

9 Collaborative Filtering Process Community Opinions Items you’ve experienced Predictions Unseen items Your Opinions You

10 Find a Restaurant!

11 Find a Restaurant!

12 Find a Restaurant!

13 Find a Restaurant!

14 Find a Restaurant!

15 Find a Restaurant!

16 Find a Restaurant!

17 Advantages of Pure CF No expensive and error-prone user attributes or item attributes Incorporates quality and taste Works on any rate-able item One data model => many content domains Serendipity Users understand it!

18 Predictive Algorithms for Collaborative Filtering

19 Predictive Algorithms for Collaborative Filtering Frequently proposed taxonomy for collaborative filtering systems –Model-based methods Build a model offline Use model to generate recommendations Original data not needed at predict-time –Instance-based methods Use the ratings directly to generate recommendations

20 Model-Based Algorithms Probabilistic Bayesian approaches, clustering, PCA, SVD, etc. Key ideas –Reduced dimension representations (aggregations) of original data –Ability to reconstruct an approximation of the original data

21 Stereotypical model- based approaches Lower dimensionality => faster performance Can explain recommendations Can over-generalize Not using the latest data Force a choice of aggregation dimensions ahead of time

22 Instance-Based Methods Primarily nearest neighbor approaches Key ideas –Predict over raw ratings data (sometimes called memory- based methods) –Highly personalized recommendations

23 Stereotypical model- based approaches Use most up-to-date ratings Are simple and easy to explain to users Are unstable when there are few ratings Have linear (w.r.t. users and items) run-times Allow a different aggregation method for each user, possibly chosen at runtime

24 Evaluating CF Recommender Systems

25 Evaluation – User Tasks Evaluation depends on the user task Most common tasks –Annotation in context Predict ratings for individual items –Find good items Produce top-N recommendations Other possible tasks –Find all good items –Recommend sequence –Many others…

26 Novelty and Trust - Confidence Tradeoff –High confidence recommendations Recommendations are obvious Low utility for user However, they build trust –Recommendations with high prediction yet lower confidence Higher variability of error Higher novelty => higher utility for user

27 Test Users Community Ratings Data Training Set Test Set

28 Predictive Accuracy Metrics Mean absolute error (MAE) Most common metric Characteristics –Assumes errors at all levels in the ranking have equal weight –Sensitive to small changes –Good for “Annotate in Context” task –May not be appropriate for “Find Good Items” task

29 Classification Accuracy Metrics Precision/Recall –Precision: Ratio of “good” items recommended to number of items recommended –Recall: Ratio of “good” items recommended to the total number of “good” items Characteristics –Simple, easy to understand –Binary classification of “goodness” –Appropriate for “Find Good Items” –Can be dangerous due to lack of ratings for recommended items

30 ROC Curves “Relative Operating Characteristic” or “Receiver Operating Characteristic” Characteristics –Binary classification –Not a single number metric –Covers performance of system at all points in the recommendation list –More complex

31

32

33 Prediction to Rating Correlation Metrics Pearson, Spearman, Kendall Characteristics –Compare non-binary ranking to non-binary ranking –Rank correlation metrics suffer from “weak orderings” –Can only be computed on rated items –Provide a single score

34 Half-life Utility Metric Characteristics –Explicitly incorporates idea of decreasing user utility –Tuning parameters reduce comparability –Weak orderings can result in different utilities for the same system ranking –All items rated less than the max contribute equally –Only metric to really consider non-uniform utility

35 Does it Matter What Metric You Use? An empirical study to gain some insight…

36 Analysis of 432 variations of an algorithm on a 100,000 rating movie dataset

37 Comparison among results provided by all the per-user correlation metrics and the mean average precision per user metric. These metrics have strong linear relationships with each other.

38

39 Comparison between metrics that are averaged overall rather than per- user. Note the linear relationship between the different metrics.

40 A comparison of representative metrics from the three subsets that were depicted in the previous slides. Within each of the subsets, the metrics strongly agree, but this figure shows that metrics from different subsets do not correlate well

41 Does it Matter What Metric You Use? Yes.

42 Want to try CF? CoFE “Collaborative Filtering Engine” –Open source Java –Easy to add new algorithms –Includes testing infrastructure (this month) –Reference implementations of many popular CF algorithms –One high performance algorithm Production ready (see Furl.net)

43 Improving Web Search Using CF With Janet Webster, OSU Libraries

44 Controversial Claim Improvements in text analysis will substantially improve the search experience Focus on improving results of Mean Average Precision (MAP) metric

45 Evidence of the Claim Human Subjects Study by Turpin and Hersh (SIGIR 2001) –Compared human performance of 1970s search model (basic TF/IDF) Recent OKAPI search model with greatly improved MAP –Task: locating medical information –No statistical difference

46 Bypass the Hard Problem! The hard problem – Automatic analysis of text –Software “understanding” language We propose: Let humans assist with the analysis of text! –Enter Collaborative Filtering

47

50 The Human Element Capture and leverage the experience of every user –Recommendations are based on human evaluation Explicit votes Inferred votes (implicit) Recommend (question, document) pairs –Not just documents –Human can determine if questions have similarity System gets smarter with each use –Not just each new document

51 Research Issues Basic Issues –Is the concept sound? –What are the roadblocks? More mathematical issues –Algorithms for ranking recommendations (question, document, votes) –Robustness with unreliable data Text/Content Analysis –Improved NLP for matching questions –Incorporating more information into information context More social issues –Training users for new paradigm –Privacy –Integrating with existing reference library practices and systems –Offensive material in questions –Most effective user interface metaphors

52 Initial Results

53 Initial Results

Average visited documents: First click - recommendation (141 – 71.6%) First click - Google result (56 – 28.4%) Average ratings: Average ratings: Only Google Results ( %) Google results + recommendations ( %) Average visited documents: Clicked (172 – 24.4%) No clicks ( %) Clicked (197 – 40.4%) No click (291 – 59.6%) Three months SERF usage – 1194 search transactions

Average visited documents: First click - recommendation (141 – 71.6%) First click - Google result (56 – 28.4%) Average ratings: Average ratings: Only Google Results ( %) Google results + recommendations ( %) Average visited documents: Clicked (172 – 24.4%) No clicks ( %) Clicked (197 – 40.4%) No click (291 – 59.6%) Three months SERF usage – 1194 search transactions

Average visited documents: First click - recommendation (141 – 71.6%) First click - Google result (56 – 28.4%) Average ratings: Average ratings: Only Google Results ( %) Google results + recommendations ( %) Average visited documents: Clicked (172– 24.4%) No clicks ( %) Clicked (197 – 40.4%) No click (291 – 59.6%) Three months SERF usage – 1194 search transactions

Average visited documents: First click - recommendation (141 – 71.6%) First click - Google result (56 – 28.4%) Average rating: (49% Voted as Useful) Average rating: (69% Voted as Useful) Only Google Results ( %) Google results + recommendations ( %) Average visited documents: Clicked (172 – 24.4%) No clicks ( %) Clicked (197 – 40.4%) No click (291 – 59.6%) Three months SERF usage – 1194 search transactions Vote of yes = 30, vote of no = 0

58 SERF Project Summary No large leaps in language understanding expected –Understanding the meaning of language is *very* hard Collaborative filtering (CF) bypasses this problem –Humans do the analysis Technology is widely applicable

59 Talk Messages Model for learning options in recommender systems Survey of popular predictive algorithms for CF Survey of evaluation metrics Empirical data showing metrics can matter Evidence showing CF could significantly improve web search

60 Links & Contacts CoFE – SERF – Jon Herlocker + 1 (541)