CoNMF: Exploiting User Comments for Clustering Web2.0 Items Presenter: He Xiangnan 28 June School of Computing National University of Singapore

Xiangnan He Introduction Motivations: – Users comment on items based on their own interests. – Most users’ interests are limited. – The categories of items can be inferred from the comments. Proposed problem: – Clustering items by exploiting user comments. Applications: – Improve search diversity. – Automatic tag generation from comments. – Group-based recommendation 2WING, NUS

Xiangnan He Challenges Traditional solution: – Represent items as a feature space. – Apply any clustering algorithm, e.g. k-means. Key challenges: – Items have heterogeneous features: 1.Own features (e.g. words for articles, pixels for images) 2.Comments  Usernames  Textual contents – Simply concatenate all features does not preform well. – How to meaningfully combine the heterogeneous views to produce better clustering (i.e. multi-view clustering)? 3WING, NUS

Xiangnan He Proposed solution Extend NMF (Nonnegative Matrix Factorization) to support multi-view clustering… 4WING, NUS

Xiangnan He NMF (Non-negative Matrix Factorization) 5WING, NUS Factorize data matrix V (#doc×#words) as: –where W is #doc × k and H is k × #words, and each entry is non- negative Alternating optimization: – With Lagrange multipliers, differentiate on W and H respectively. Local optimum, not global! Goal is minimizing the objective function: –where || || denotes the Frobenius norm

Xiangnan He Difference with SVD(LSI): Characteristics of NMF Matrix Factorization with a non-negative constraint Reduce the dimension of the data; derive the latent space CharacteristicSVDNMF Orthogonal basisYesNo Negative entryYesNo Post clusteringYesNo Theoretically proved suitable for clustering (Chis et al. 2005) Practically shown superior performance than SVD and k-means in document clustering (Xu et al. 2003)

Xiangnan He Extensions of NMF Relationships with other clustering algorithms: – K-means: Orthogonal NMF = K-means – PLSI: KL-Divergence NMF = PLSI – Spectral clustering Extensions: –Tri-factor of NMF( V = W S H ) (Ding et al. 2006) –NMF with sparsity constraints (Hoyer 2004) –NMF with graph regularization (Cai et al. 2011) – However, studies on NMF-based multi-view clustering approaches are quite limited. (Liu et al. 2013) My proposal: – Extend NMF to support multi-view clustering 7WING, NUS

Xiangnan He Proposed solution - CoNMF Idea: – Couple the factorization process of NMF Example: – Single NMF:  Factorization equation ：  Objective function:  Constraints: all entries of W and H are non-negative. 8WING, NUS - 2-view CoNMF:  Factorization equation:  Objective function:

Xiangnan He CoNMF Framework – Mutual-based:  Point-wise:  Cluster-wise: 9WING, NUS Objective function: –Similar alternating optimization with Lagrange multipliers can solve it. Coupling the factorization process of multiple matrices(i.e. views) via regularization. Different options of regularization: – Centroid-based (Liu et al. 2013):

Xiangnan He Experiments Last.fm dataset: 3-views: Ground-truth: – Music type of each artist provided by Last.fm Evaluation metrics: – Accuracy and F1 Average performance of 20 runs. 10WING, NUS #Items#Users#Comments#Clusters 9,694131,8982,500,27121 View#Items#FeaturesToken type Items-Desc. words9,69414,076TF – IDF Items-Comm. words9,69431,172TF – IDF Items-Users9,694131,898Boolean

Xiangnan He Statistics of datasets 11WING, NUS Statistics of #items/userStatistics of #clusters/user P(T<=3) = P(T<=5) = P(T<=10) = Verify our assumption: each user usually comments on limited music types.

Xiangnan He Experimental results (Accuracy) InitializationMethodDesc.Comm.UsersComb. Randomkmeans WING, NUS SVD RandomNMF K-meansNMF K-means CoNMF – point K-means CoNMF – cluster NMF Multi-NMF (SDM'13) Random MM-LDA (WSDM'09) Users>Comm.>Desc., while combined is best. 2. SVD performs badly on users (non-textual). 3. Users>Comm.>Desc., while combined does worse. 4. Initialization is important for NMF. 5. CoNMF-point performs best. 6. Other two state-of-the-art baselines.

Xiangnan He Experimental results (F1) 13WING, NUS InitializationMethodDesc.Comm.UsersCombined Randomkmeans SVD RandomNMF K-meansNMF K-means CoNMF – point K-means CoNMF – cluster NMF Multi-NMF (SDM'13) Random MM-LDA (WSDM'09) 0.286

Xiangnan He Conclusions Comments benefit clustering. Mining different views from the comments is important: – The two views (commenting words and users) contribute differently for clustering. – For this Last.fm dataset, users is more useful. – Combining all views works best. For NMF-based methods, initialization is important. 14WING, NUS

Xiangnan He Ongoing More experiments on other datasets. Improve the CoNMF framework through adding the sparseness constraints. The influence of normalization on CoNMF. 15WING, NUS

Xiangnan He Thanks! QA? 16WING, NUS

Xiangnan He References(I) Ding Chris, Xiaofeng He, and Horst D. Simon On the equivalence of nonnegative matrix factorization and spectral clustering. In Proc. SIAM Data Mining Conf Wei Xu, Xin Liu, and Yihong Gong Document clustering based on non-negative matrix factorization. In Proc. of SIGIR 2003 Chris Ding, Tao Li, Wei Peng Orthogonal nonnegative matrix tri-factorizations for clustering. In Proc. of SIGKDD 2006 Patrik O. Hoyer Non-negative Matrix Factorization with Sparseness Constraints. Journal of Machine Learning Researh 2004 Deng Cai, Xiaofei He, Jiawei Han, and Thomas S. Huang Graph Regularized Nonnegative Matrix Factorization for Data Representation. IEEE Trans. Pattern Anal. Mach. Intell Jialu Liu, Chi Wang, Jing Gao and Jiawei Han Multi-View Clustering via Joint Nonnegative Matrix Factorization, In Proceedings of SIAM Data Mining Conference (SDM’13) 17WING, NUS