Hybrid Web Recommender Systems Robin Burke Presentation by Jae-wook Ahn 10/04/05

Hybrid Web Recommender Systems References Entrée system & dataset Burke, R. (2002). Semantic ratings and heuristic similarity for collaborative filtering. AAAI Workshop on Knowledge-based Electronic Markets 2000. Feature augmentation, mixed hybrid example Torres, R., McNee, S., Abel, M., Konstan J., & Riedl J. (2004). Enhancing Digital Libraries with TechLens+. Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries. Hybrid recommender system UI issue Schafer, J. (2005). DynamicLens: A Dynamic User-Interface for a Meta-Recommendation System. Workshop: Beyond Personalization 2005, IUI’05. Collaborative filtering algorithm Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001). Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web. 10/5/05 Hybrid Web Recommender Systems

Concepts and Techniques

Hybrid Recommender Systems Mix of recommender systems Recommender system classification – knowledge source Collaborative (CF) User’s ratings “only” Content-based (CN) Product features, user’s ratings Classifications of user’s likes/dislikes Demographic User’s ratings, user’s demographics Knowledge-based (KB) Domain knowledge, product features, user’s need/query Inferences about a use’s needs and preferences 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems CF vs. CN User-based CF Searches for similar users in user-item “rating” matrix Item-based CF Searches for similar items in user-item “rating” matrix CN Searches for similar items in item-feature matrix Example – TF*IDF term weight vector for news recommendation Items Ratings Users 10/5/05 Hybrid Web Recommender Systems

Recommender System Problems Cold-start problem Learning based techniques Collaborative, content-based, demographic  Hybrid techniques Stability vs. plasticity problem Difficulty to change established user’s profile Temporal discount – older rating with less influence KB – fewer cold start problem (no need of historical data) CF/Demographic – cross-genre niches, jump outside of the familiar (novelty, serendipity) 10/5/05 Hybrid Web Recommender Systems

Strategies for Hybrid Recommendation Combination of multiple recommendation techniques together for producing output Different techniques of different types Most common implementations Most promise to resolve cold-start problem Different techniques of the same type Ex) NewsDude – naïve Bayes + kNN 10/5/05 Hybrid Web Recommender Systems

Seven Types of Recommender Systems Taxonomy by Burke (2002) Weighted Switching Mixed Feature combination Feature augmentation Cascade Meta-level 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Weighted Hybrid Concept Each component of the hybrid scores a given item and the scores are combined using a linear formula When recommenders have consistent relative accuracy across the product space Uniform performance among recommenders (otherwise  other hybrids) 10/5/05 Hybrid Web Recommender Systems

Weighted Hybrid Procedure Training Joint rating Intersection – candidates shared between the candidates Union – case with no possible rating  neutral score (neither liked nor disliked) Linear combination 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Mixed Hybrid Concepts Presentation of different components side-by-side in a combined list If lists are to be combined, how are rankings to be integrated? Merging based on predicted rating or on recommender confidence Not fit with retrospective data Cannot use actual ratings to test if right items ranked highly Example CF_rank(3) + CN_rank(2)  Mixed_rank(5) 10/5/05 Hybrid Web Recommender Systems

Mixed Hybrid Procedure Candidate generation Multiple ranked lists Combined display 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Switching Hybrid Concepts Selects a single recommender among components based on recommendation situation Different profile  different recommendation Components with different performance for some types of users Existence of criterion for switching decision Ex) confidence value, external criteria 10/5/05 Hybrid Web Recommender Systems

Switching Hybrid Procedure Switching decision Candidate generation Scoring No role for unchosen recommender 10/5/05 Hybrid Web Recommender Systems

Feature Combination Hybrid Concepts Inject features of one source into a different source for processing different data Features of “contributing recommender” are used as a part of the “actual recommender” Adding new features into the mix Not combining components, just combining knowledge source 10/5/05 Hybrid Web Recommender Systems

Feature Combination Hybrid Procedure  In training stage Candidate generation Scoring 10/5/05 Hybrid Web Recommender Systems

Feature Augmentation Hybrid Concepts Similar to Feature Combination Generates new features for each item by contributing domain Augmentation/combination – done offline Comparison with Feature Combination Not raw features (FC), but the result of computation from contribution (FA) More flexible to apply Adds smaller dimension 10/5/05 Hybrid Web Recommender Systems

Feature Augmentation Hybrid Procedure 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Cascade Hybrid Concepts Tie breaker Secondary recommender Just tie breaker Do refinements Primary recommender Integer-valued scores – higher probability for ties Real-valued scores – low probability for ties Precision reduction Score: 0.8348694  0.83 10/5/05 Hybrid Web Recommender Systems

Cascade Hybrid Procedure Primary recommender Ranks Break ties by secondary recommender 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Meta-level Hybrid Concepts A model learned by contributing recommender  input for actual recommender Contributing recommender completely replaces the original knowledge source with a learned model Not all recommenders can produce the intermediary model 10/5/05 Hybrid Web Recommender Systems

Meta-level Hybrid Procedure Contributing recommender  Learned model Knowledge Source Replacement Actual Recommender 10/5/05 Hybrid Web Recommender Systems

Experiments

Testbed – Entrée Restaurant Recommender Entrée System Case-based reasoning Interactive critiquing dialog Ex) Entry  Candidates  “Cheaper”  Candidates  “Nicer”  Candidates  Exit Not “narrowing” the search by adding constrains, but changing the focus in the feature space 10/5/05 Hybrid Web Recommender Systems

Testbed – Entrée Restaurant Recommender (cont’d) Entrée Dataset Rating Entry, ending point – “positive” rating Critiques – “negative” rating Mostly negative ratings Validity test for positive ending point assumption – strong correlation between original vs. modified (entry points with positive ratings) Small in size 10/5/05 Hybrid Web Recommender Systems

Evaluation Methodology Measures ARC (Average Rank of the Correct recommendations) Accuracy of retrieval At different size retrieval set Fraction of the candidate set (0 ~ 1.0) Training & Test set 5 fold cross validation – random partition of training/test set “Leave one out” methodology – randomly remove one item and check whether the system can recommend it Sessions Sizes Single visit profiles – 5S, 10S, 15S Multiple visit profiles – 10M, 20M, 30M 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Baseline Algorithms Collaborative Pearson (CFP) Pearson’s correlation coefficient for similarity Collaborative Heuristic (CFH) Heuristics for calculating distances between critiques “nicer” and “cheaper”  dissimilar “nicer” & “quieter”  similar Content-based (CN) Naïve Bayes algorithm – compute probability that a item is “liked” / “disliked” Too few “liked” items  modified candidate generation Retrieve items with common features with the “liked” vector of the naïve Bayes profile Knowledge-based (KB) Knowledge-based comparison metrics of Entrée Nationality, price, atmosphere, etc. 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Baseline Evaluations Techniques vary in performance on the Entrée data Content-based (CN) – weak Knowledge-based (KB) – better on single-session than multi-session Heuristic collaborative (CFH) – better than correlation-based (CFP) for short profiles Room for improvement Multi-session profiles 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Baseline Evaluations 10/5/05 Hybrid Web Recommender Systems

Hybrid Comparative Study Missing components Mixed hybrid Not possible with retrospective data Demographic recommender No demographic data 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Results – Weighted Hybrid performance better in only 10 of 30 CN/CFP – consistent synergy (5 of 6) Lacks uniform performance KB, CFH Linear weighting scheme assumption – fault 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Results – Switching KB hybrids – best switching hybrids 10/5/05 Hybrid Web Recommender Systems

Results – Feature Combination CN/CFH, CN/CFP Contributing CN Identical to CFH, CFP CFH maintains accuracy with reduced dataset CF/CN Winnow – modest improvement 10/5/05 Hybrid Web Recommender Systems

Results – Feature Augmentation Best performance so far Particularly CN*/CF* Good for multi-session profiles 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Results – Cascade CFP/KB, CFP/CN Great improvement Also good for multi-profile sessions 10/5/05 Hybrid Web Recommender Systems

Results – Meta-level Hybrids CN/CF, CN/KB, CF/KB, CF/CN Not effective No synergy Weakness of KB/CN in Entrée dataset Both components should be strong 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Discussion Dominance of the hybrids over basic recommenders Synergy was found under Smaller profile size Sparse recommendation density  hybridization conquers cold start problem 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Discussion (cont’d) Best hybrids Feature augmentation, cascade FA allows a contributing recommender to make a positive impact without interfering with the performance of the better algorithm 10/5/05 Hybrid Web Recommender Systems

Hybrid Web Recommender Systems Conclusions Knowledge-based recommendation is not limited Numerously combined to build hybrids Good for secondary or contributing components Cascade hybrids are effective Though rare in literatures Effective for combining recommender with different strengths Different performance characteristics Six hybridization techniques Relative accuracy & consistency of hybrid components 10/5/05 Hybrid Web Recommender Systems

System Example & Related Issues

System Example – TechLens+ Hybrid recommender system Recommenders – CF, CN Hybrid algorithms – CF/CN FA, CN/CF FA, Fusion (Mixed) Corpus CiteSeer Title, abstract (CN), citations (CF) Methodology Offline experiment, Online user study with questionnaire (by asking satisfaction on the recommendation) Results Fusion was the best Some FA were not good due the their sequential natures Different algorithms should be used for recommending different papers Users with different levels of experiences perceive recommendations differently 10/5/05 Hybrid Web Recommender Systems

Meta-recommender – DynamicLens Can user provided information improve hybrid recommender system output? Meta-recommender Provide users with personalized control over the generation of a recommendation list from hybrid recommender system MetaLens IF (Information Filtering), CF 10/5/05 Hybrid Web Recommender Systems

Meta-recommender – DynamicLens (cont’d) Dynamic query Merges preference & recommendation interfaces Immediate feedback Discover why a given set of ranking recommendations were made 10/5/05 Hybrid Web Recommender Systems

Questions & Comments