Missing Value Prediction Using Co-clustering and RBF for Collaborative Filtering Department of Automation Xiamen University Youchun Ji, Wenxing Hong*, Jianwei Qi November, 2015 Good morning, everyone. My name is Ji Youchun. It is a privilege to have the opportunity to report on behalf of our group. The title of my presentation is Missing Value Prediction Using Co-clustering and RBF for Collaborative Filtering.

I come from the beautiful coastal city Xiamen I come from the beautiful coastal city Xiamen. Our school Xiamen University is known as one of the most beautiful universities in China. 1

Case Website 5501.cn i.xmrc.com.cn 17du.info Interest Expert finding Job recommendation News recommendation My major is automation. Our group focus on studying datamining, recommendation system, and so on. Our case include the fist one expert finding, the second one job recommendation, and the third one news recommendation. I focus on news recommendation. Our research have done on how to modeling the users, how to improve the precision of recommendation and how to alleviate the problem of sparsity. Today I want to talk about our work on how to reduce the sparseness of user-item rating matrix. 2012-2014 2014-now 2014-now 2

Outline 1 Introduction 2 The Problem Definition 3 Algorithms & Experiments Conclusion 4 The outline of my talk is as follows. The first part I want to introduce the background of this research. The second part defines the problem. The third part suggests the algorithms for the problem and then introduces the experiments. Finally, a simple conclusion is given. 3

Introduction As we know, with the rapid development of the Internet, the quantity of news and information increase explosively. People can conveniently obtain information on the Internet, but produce a lot of redundant information at the same time. Help users find interesting articles that match the users’ preference as much as possible. Jannach, D., M. Zanker, A. Felfernig, &G. Friedrich, Recommender systems: an introduction. 2010: Cambridge University Press. Zheng, L., L. Li, W. Hong, &T. Li, PENETRATE: Personalized news recommendation using ensemble hierarchical clustering. Expert Systems with Applications, 2013. 40(6): p. 2127-2136. Das, A.S., M. Datar, A. Garg, &S. Rajaram. Google news personalization: scalable online collaborative filtering. in Proceedings of the 16th international conference on World Wide Web. 2007. ACM. Breese, J.S., D. Heckerman, &C. Kadie. Empirical analysis of predictive algorithms for collaborative filtering. in Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence. 1998. Morgan Kaufmann Publishers Inc. 4

Introduction Therefore, a variety of strategies are proposed to make choices about what to read. News recommendation systems automate some of these strategies with the goal of providing affordable, personal, and high-quality recommendation. Nowadays, approaches oriented from content filtering, collaborative filtering are proposed and widely used by existing news recommender systems. Collaborative filtering is one of the most successful methods for news recommendation systems. Collaborative filtering is one of the most successful methods for news recommendation systems. Pazzani, M.J., A framework for collaborative, content-based and demographic filtering. Artificial Intelligence Review, 1999. 13(5-6): p. 393-408. Huang, Z., H. Chen, &D. Zeng, Applying associative retrieval techniques to alleviate the sparsity problem in collaborative filtering. ACM Transactions on Information Systems (TOIS), 2004. 22(1): p. 116-142. Hofmann, T., Latent semantic models for collaborative filtering. ACM Transactions on Information Systems (TOIS), 2004. 22(1): p. 89-115. Blei, D.M., A.Y. Ng, &M.I. Jordan, Latent dirichlet allocation. The Journal of machine Learning research, 2003. 3: p. 993-1022. 5

Motivation Scenario 2 Scenario 3 Scenario 1 In order to overcome the problem, we predict the values of user-item rating matrix combining two approaches: co-clustering and Radial Basis Function network (RBF). The sparsity of user-item rating matrix will lead to the negative effect of collaborative filtering algorithm. The number of news which users have read is far less than the news published on the website. But, in reality, the number of news which users have read is far less than the news published on the website, and the latest published news do not have enough evaluation ratings. The sparsity of user-item rating matrix will reduce the performance of collaborative filtering algorithm in news recommendation system. In order to overcome the problem, we predict the values of user-item rating matrix combining two approaches: co-clustering and Radial Basis Function network (RBF). 6 Zhang, S., W. Wang, J. Ford, &F. Makedon. Learning from Incomplete Ratings Using Non-negative Matrix Factorization. in SDM. 2006. SIAM. Dhillon, I.S. Co-clustering documents and words using bipartite spectral graph partitioning. in Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. 2001. ACM.

Outline 1 Introduction 2 The Problem Definition 3 Algorithms & Experiments Conclusion 4 These are the motivation of our research. But how to defines the problem? 7

The Problem Definition           It is defined as an optimization problem. The optimization goal is minimizing the Root Mean Square Error of R and R^' apostrophe. The process of news recommendation is the process of missing value prediction. We predict the missing values in user-item rating matrix, and recommend the top N rating news that the users have not read. Using a matrix representation, we transform this problem to a weighted matrix approximation problem. U is the set of users and V is the set of news. is the real rating of the user i for the news j. ’apostrophe is the prediction value. We can use this mathematical model to describe the problem. 8 George, T., &S. Merugu. A scalable collaborative filtering framework based on co-clustering. in Data Mining, Fifth IEEE International Conference on. 2005. IEEE.

Outline 1 Introduction 2 The Problem Definition 3 Algorithms & Experiments Conclusion 4 And how do we solve this problem? In the paper, we combine co-clustering and RBF to predict the missing values of the user-item rating matrix. 9

Data sample – Reading History News ID v1 v2 v3 v4 v5 v6 v7 v8 u1 3 u2 2 u3 u4 4 u5 1 u6 5 u7 u8 u9 u10 u11 u12 u13 User ID This is the data sample of our experiment. The data sample comes from the log of Xiamen University news reading website. We assume that the times a user access to a page represent the preference of the user. And the IP address represents users’ ID. We calculate the item ratings by the users’ click times .For example, we choose 13 users and 8 news. If an article is accessed by a reader 5 times is assigned the score of 5. Similarly, an article is accessed by a reader 1 times is assigned the score of 1. If an article haven’t been read by a reader, it is assigned the score of zero. The data sample like this. 10

Algorithms – Flow chart The figure shows the workflow of the proposed algorithm. The input is the user-item rating matrix and the number of clusters. Then co-cluster the matrix into several small matrix with high similarity. Then use Radial Basis Function network to predict the missing values. The output is the matrix after predicting. For example, user one user three and user five have the similar behavior in item one and item two. They can cluster the similar users and items into a cluster. Then we choose this cluster. If user three haven’t read news one. We predict the score of user three for news one. The prediction value is 2. 11

Algorithms – Co-clustering 0.8756 0.7535 Now we introduce the details of the algorithm. The first part is co-clustering. Co-clustering simultaneous cluster the rows and columns of the user-item rating matrix and reduce the dimensions of the user-item rating matrix. It can cluster the rating matrix into some small matrices. The user or item in the same matrix have high similarity. Co-clustering calculate the probability of the first formula , it is the probability that the user-item rating belongs to the k cluster. It is based on the probability of the second formula that user belongs to the k cluster and the probability of the third formula that item belongs to the k cluster. We can know the probability of the forth formula that the probability of rating in the k cluster. We calculate the probability of the first formula until it converges. The procedure value input and output like this. Take user one, news one and the third cluster for example. The input of this one is 0.21, it calculate by these three formular ,the output is 0.8756. the value of the second formula is changing according to the first formula. 0.9400 0.2241 1. Hu, W., W. Yong-Ji, W. Zhe, W. Xiu-Li, et al., Two-Phase Collaborative Filtering Algorithm Based on Co-Clustering. Journal of Software. 21: p. 1042-1054 (in Chinese). 2. George, T., &S. Merugu. A scalable collaborative filtering framework based on co-clustering. in Data Mining, Fifth IEEE International Conference on. 2005. IEEE. 3. Dhillon, I.S. Co-clustering documents and words using bipartite spectral graph partitioning. in Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. 2001. ACM. 12

Algorithms - RBF The second part is the Radial Basis Function network. The input is one small matrix which has a high similarity after co-clustering. And the expected output is the mean rating vector which only calculate the non-null values. The output of the network is a scalar function of the input vector. It is given by We used Gaussian Function as the radial basis function. Because the value is changing according to the input matrix, so I don’t want to describe in detail, 13 1. https://en.wikipedia.org/wiki/Radial_basis_function_network. 2. Fuliang, X., &Z. Huiying, A Research of Collaborative Filtering Recommender MethodBased on SOM and RBFN Filling Missing Values. XIANDAI TUSHU QINGBAO JISHU, 2014. 7/8: p. 56-63 (in Chinese).

Platform - http://yiqidu.xmu.edu.cn/ In order to do the experiments, we built an online website yiqidu. The platform like this. It includes the function of search and login. And show the campus news ,popular news and recommendation list. It is an online system. Welcome to visit the website. We are also improving it. 14

Data set – XMU News Rating:: UserID:: NewsID:: News title The experiment data comes from Xiamen University news reading website which is focus on campus news. It includes 9502 users, 6372 news and 932640 rating. The sparseness of the user-item rating matrix is 98.46%. The data set was divided into testing set and training sets. The experiment data comes from Xiamen University news reading website which is focus on campus news. It includes about 9502 users, 6372 news and 932640 rating. The sparseness of the user-item rating matrix is 98.46%. The data set was divided into testing set and training set. This is the original fragments of data. It includes UserID News ID, News title and the rating. Rating:: UserID:: NewsID:: News title 15 1. Jiang, S., &W. Hong. A vertical news recommendation system: CCNS—An example from Chinese campus news reading system. in Computer Science & Education (ICCSE), 2014 9th International Conference on. 2014. IEEE.

Experiments The number of co-clustering is 36 in the experiment. After prediction the missing values, the sparseness of the user-item rating matrix reduce to about 60%. sparseness Before >0.95 The number of co-clustering is 36 in the experiment. After prediction the missing values, the sparseness of the user-item rating matrix reduce from 98% to about 60%. After <0.65 co- clustering number 16

Experiments As the experiment result shows, the prediction method that combine co-clustering and RBF work effective on XMUNEWS data set. The root mean square error is 1.553. Algorithm RMSE Time(s) Co-clustering 2.455 40 RBF 2.092 150 Co-clustering & RBF 1.553 330 As the experiment result shows, the prediction method that combine co-clustering and RBF work effective on XMUNEWS data set. The root mean square error is 1.553. Although the algorithm cost more time, we can predict the missing value offline. We concentrate on the Root Mean Square Error. We observe that the combing method is better than the individual algorithms on XMUNEWS data set. 17

Outline 1 Introduction 2 The Problem Definition 3 Algorithms & Experiments Conclusion 4 Finally, I want to conclude my presentation. 18

Conclusion Before prediction, the sparseness of the user-item rating matrix is above 96%. But after prediction, it reduce to below 60%. The root mean square error of true rating values and prediction rating values is 1.553 on XMUNEWS data set. As the experiment result shows, the combining algorithm is better than the separate algorithm. We built an online website to collect data and do experiments (http://yiqidu.xmu.edu.cn/). For future work, we will concentrate on how to improve the computational efficiency and how to choose the number of clusters. we use co-clustering and Radial Basis Function network (RBF) to predict the missing values of user-item rating matrix. Before prediction, the sparseness of the user-item rating matrix is above 96%. But after prediction, it reduce to below 60%. The root mean square error of true rating values and prediction rating values is 1.553 on XMUNEWS data set. As the experiment result shows, the combining algorithm is better than the separate algorithm. We built an online website to collect data and do experiments (http://yiqidu.xmu.edu.cn/). For future work, we will concentrate on how to improve the computational efficiency and how to choose the number of clusters. 19

References Jannach, D., M. Zanker, A. Felfernig, &G. Friedrich, Recommender systems: an introduction. 2010: Cambridge University Press. Zheng, L., L. Li, W. Hong, &T. Li, PENETRATE: Personalized news recommendation using ensemble hierarchical clustering. Expert Systems with Applications, 2013. 40(6): p. 2127-2136. Das, A.S., M. Datar, A. Garg, &S. Rajaram. Google news personalization: scalable online collaborative filtering. in Proceedings of the 16th international conference on World Wide Web. 2007. ACM. Breese, J.S., D. Heckerman, &C. Kadie. Empirical analysis of predictive algorithms for collaborative filtering. in Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence. 1998. Morgan Kaufmann Publishers Inc. Pazzani, M.J., A framework for collaborative, content-based and demographic filtering. Artificial Intelligence Review, 1999. 13(5-6): p. 393-408. Huang, Z., H. Chen, &D. Zeng, Applying associative retrieval techniques to alleviate the sparsity problem in collaborative filtering. ACM Transactions on Information Systems (TOIS), 2004. 22(1): p. 116-142. Hofmann, T., Latent semantic models for collaborative filtering. ACM Transactions on Information Systems (TOIS), 2004. 22(1): p. 89-115. Blei, D.M., A.Y. Ng, &M.I. Jordan, Latent dirichlet allocation. The Journal of machine Learning research, 2003. 3: p. 993-1022. Zhang, S., W. Wang, J. Ford, &F. Makedon. Learning from Incomplete Ratings Using Non-negative Matrix Factorization. in SDM. 2006. SIAM. Dhillon, I.S. Co-clustering documents and words using bipartite spectral graph partitioning. in Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. 2001. ACM. George, T., &S. Merugu. A scalable collaborative filtering framework based on co-clustering. in Data Mining, Fifth IEEE International Conference on. 2005. IEEE. https://en.wikipedia.org/wiki/Radial_basis_function_network. Fuliang, X., &Z. Huiying, A Research of Collaborative Filtering Recommender MethodBased on SOM and RBFN Filling Missing Values. XIANDAI TUSHU QINGBAO JISHU, 2014. 7/8: p. 56-63 (in Chinese). Jiang, S., &W. Hong. A vertical news recommendation system: CCNS—An example from Chinese campus news reading system. in Computer Science & Education (ICCSE), 2014 9th International Conference on. 2014. IEEE.  20

Thanks! Q&A Acknowledgment The research was supported by the National Natural Science Foundation of China under Grant No.61303081 and by the Fundamental Research Funds for the Xiamen University under Grant No. 20720152008. Thanks! Q&A Ok, these are my representation. Once again I would like to thank you for the opportunity of talking to you on the subject.   Do you have any question? 21