A NON-IID FRAMEWORK FOR COLLABORATIVE FILTERING WITH RESTRICTED BOLTZMANN MACHINES Kostadin Georgiev, VMware Bulgaria Preslav Nakov, Qatar Computing Research Institute ICML, June 17, 2013, Atlanta

Overview 1. Non-IID framework for collaborative filtering based on Restricted Botzmann Machines (RBMs) with user ratings modeled as real values (vs. multinomials) 2. A neighborhood method boosted by RBM 2 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

COLLABORATIVE FILTERING

Introduction Recommender systems predict user preferences for new items content-based vs. collaborative Collaborative filtering (CF) predictions inferred from the preferences of other users N x M user-item matrix of rating values large and highly sparse (e.g., 95% of values are missing) 4 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

User-based vs. Item-based CF User-based most of the early CF systems Item-based e.g., (Sarwar et al., 2001) Joint user-item based matrix factorization, joint latent factor space (Salakhutdinov & Mnih, 2008; Koren et al., 2009; Lawrence & Urtasun,2009); probabilistic latent model (Langseth & Nielsen, 2012) 5 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Boltzmann Machines for CF Restricted Botzmann Machines (Salakhutdinov et al., 2007) user-based ratings as multinomial variables outperforms SVD Unrestricted Boltzmann Machines (Truyen et al., 2009) joint user-item based connections between the visible units preprocessing, correlations computation, neighborhood formation ordinal modeling of ratings better than categorical 6 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

THIS WORK

Outline User-based RBM (U-RBM) Item-based RBM (I-RBM) Hybrid non-IID RBM (UI-RBM) Neighborhood method boosted by I-RBM (I-RBM+INB) 8 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

User/Item-based RBM Model The visible layer represents either all ratings by a user or all rating for an item units model ratings as real values (vs. multinomial) noise-free reconstruction is better 9 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Non-IID Hybrid RBM Model (1) We remove the IID assumption for the training data Topology: Unit v ij is connected to two independent hidden layers: one user-based and another item-based. 10 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Non-IID Hybrid RBM Model (2) Missing values (ratings): the generated predictions are used during testing, but are ignored during training Training procedure: we average the predictions of the user-based and of the item-based RBM models 11 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Neighborhood Boosted by I-RBM Use the I-RBM predictions from a neighborhood-based (NB) algorithm However, compute the averages from the original ratings 12 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

EXPERIMENTS AND EVALUATION

Data Two MovieLens datasets: 100k: 1,682 movies assigned 943 users 100,000 ratings sparseness: 93.7% 1M: 3,952 movies 6,040 users 1 million ratings sparseness: 95.8% Each rating is an integer between 1 (worst) and 5 (best) 14 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Evaluation Mean Absolute Error (MAE): Cross-validation 5-fold 80%:20% training:testing data splits 15 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Experiments Evaluated three RBM-based models: User-based RBM (U-RBM) Item-based RBM (I-RBM) Hybrid non-IID RBM model (UI-RBM) Tested real-valued vs. multinomial visible units for all above models Neighborhood model boosted by I-RBM (I-RBM+INB) 16 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Types of Visible Units: the IID Case 17 In the IID case, multinomial visible units are better than real-valued. Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Types of Visible Units: the non-IID Case 18 In the non-IID case: real-valued visible units outperform multinomial. Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Number of Units: the IID Case 19 - Item-based RBM model outperforms user-based, but not by much. - Hybrid item-based RBM + NB model is relatively insensitive to number of units. Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Results: MovieLens 100k 20 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

Results: MovieLens 1M 21 Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines

CONCLUSION AND FUTURE WORK

Conclusion and Future Work Conclusion proposed a non-IID RBM framework for CF results rival the best CF algorithms, which are more complex Future work add an additional layer to model higher-order correlations add content-based features, e.g., demographic 23 Thank you! Georgiev & Nakov: A non-IID Framework for Collaborative Filtering with Restricted Boltzmann Machines