1. Collaborative Filtering Rong Jin Dept. of Computer Science and Engineering Michigan State University

2. 2 Short vs. Long Term Info. Need Short-term information need (Ad hoc retrieval)Short-term information need (Ad hoc retrieval) –“Temporary need”, e.g., info about used cars –Information source is relatively static –User “pulls” information –Application example: library search, Web search Long-term information need (Filtering)Long-term information need (Filtering) –“Stable need”, e.g., new data mining algorithms –Information source is dynamic –System “pushes” information to user –Applications: news filter

3. 3 Information Filtering Basic filtering question: Will user U like item X?Basic filtering question: Will user U like item X? Two different ways of answering itTwo different ways of answering it –Look at what U likes  characterize X  content-based filtering –Look at who likes X  characterize U  collaborative filtering

4. 4 Collaborative Filtering (Resnick et al., 1994) Make recommendation decisions for a specific user based on the judgments of users with similar interests.

5. 5 Collaborative Filtering (Resnick et al., 1994) Make recommendation decisions for a specific user based on the judgments of users with similar interests. User 115345 User 241523 User 32?354

6. 6 A General Strategy (Resnick et al., 1994) 1.Identify the training users that share similar interests as the test user. 2.Predict the ratings of the test user as the average of ratings by the training users with similar interests

7. 7 A General Strategy (Resnick et al., 1994) User 115345 User 241523 User 32?354 5 1.Identify the training users that share similar interests as the test user. 2.Predict the ratings of the test user as the average of ratings by the training users with similar interests

8. 8 Important Problems in Collaborative Filtering How to estimate users’ similarity if rating data is sparse?How to estimate users’ similarity if rating data is sparse? –Most users only rate a few items How to identify interests of a test user if he/she only provides ratings for a few items?How to identify interests of a test user if he/she only provides ratings for a few items? –Most users are inpatient to rate many items How to combine collaborative filtering with content filtering?How to combine collaborative filtering with content filtering? –For movie ratings, both the content information and the user ratings are available

9. Problem I: How to Estimate Users’ Similarity based on Sparse Rating Data?

10. 10 Sparse Data Problem (Breese et al., 1998) User 1?5342 User 2415?5 User 35?425 User 41535? Most users only rate a small number of items and leave most items unrated Most users only rate a small number of items and leave most items unrated

11. 11 Flexible Mixture Model (FMM) (Si & Jin, 2003) Cluster training users of similar interestsCluster training users of similar interests User 1?5342 User 2415?5 User 35?425 User 41535?

12. 12 Flexible Mixture Model (FMM) (Si & Jin, 2003) Cluster training users of similar interestsCluster training users of similar interests Cluster items with similar ratingsCluster items with similar ratings User 1?5342 User 2415?5 User 35?425 User 41535?

13. 13 Flexible Mixture Model (FMM) (Si & Jin, 2003) User Class I1p(4)=1/4 p(5)=3/4 3 User Class IIp(4)=1/4 p(5)=3/4 p(1)=1/2 p(2)=1/2 p(4)=1/2 p(5)=1/2 Movie Type I Movie Type II Movie Type III Unknown ratings are gone! Unknown ratings are gone!

14. 14 Flexible Mixture Model (FMM) (Si & Jin, 2003) Introduce rating uncertaintyIntroduce rating uncertainty User Class I1p(4)=1/4 p(5)=3/4 3 User Class IIp(4)=1/4 p(5)=3/4 p(1)=1/2 p(2)=1/2 p(4)=1/2 p(5)=1/2 Movie Type I Movie Type II Movie Type III Unknown ratings are gone! Unknown ratings are gone! Cluster both users and items simultaneouslyCluster both users and items simultaneously

15. 15 Flexible Mixture Model (FMM) (Si & Jin, 2003) Z o Z u O U R Z u : user class Z o : item class U: user O: item R: rating Cluster variable Observed variable An Expectation Maximization (EM) algorithm can be used for identifying clustering structure for both users and items

16. 16 Rating Variance (Jin et al., 2003a) The Flexible Mixture Model is based on the assumption that users of similar interests will have similar ratings for the same itemsThe Flexible Mixture Model is based on the assumption that users of similar interests will have similar ratings for the same items But, different users of similar interests may have different rating habitsBut, different users of similar interests may have different rating habits

17. 17 Rating Variance (Jin et al., 2003a) The Flexible Mixture Model is based on the assumption that users of similar interests will have similar ratings for the same itemsThe Flexible Mixture Model is based on the assumption that users of similar interests will have similar ratings for the same items But, different users of similar interests may have different rating habitsBut, different users of similar interests may have different rating habits User 135443 User 213221 User 351515 User 414231

18. 18 Rating Variance (Jin et al., 2003a) The Flexible Mixture Model is based on the assumption that users of similar interests will have similar ratings for the same itemsThe Flexible Mixture Model is based on the assumption that users of similar interests will have similar ratings for the same items But, different users of similar interests may have different rating habitsBut, different users of similar interests may have different rating habits User 135443 User 213221 User 351515 User 414231

19. 19 Hidden variable Observed variable Decoupling Model (DM) (Jin et al., 2003b) Z o Z u O U Z u : user class Z o : item class U: user O: item R: rating R

20. 20 Decoupling Model (DM) (Jin et al., 2003b) Z pref : whether users like items Z pref Z o Z u O U Z u : user class Z o : item class U: user O: item R: rating R Hidden variable Observed variable

21. 21 Decoupling Model (DM) (Jin et al., 2003b) Z pref : whether users like items Z R : rating class Z u : user class Z o : item class U: user O: item R: rating Z R Z pref Z o Z u O U R Hidden variable Observed variable

22. 22 Decoupling Model (DM) (Jin et al., 2003b) Z pref : whether users like items Z R : rating class Z u : user class Z o : item class U: user O: item R: rating Z R Z pref Z o Z u O U R Hidden variable Observed variable

23. 23 Empirical Studies EachMovie dataset:EachMovie dataset: –2000 users and 1682 movie items –Avg. # of rated items per user is 130 –Rating range: 0-5 Evaluation protocolEvaluation protocol –400 training users, and 1600 testing users –Numbers of items rated by a test user: 5, 10, 20 Evaluation metric: MAEEvaluation metric: MAE MAE: mean absolute error between true ratings and predicted ratingsMAE: mean absolute error between true ratings and predicted ratings The smaller the MAE, the better the performanceThe smaller the MAE, the better the performance

24. 24 Baseline Approaches Ignore unknown ratingsIgnore unknown ratings –Vector similarity (Breese et al., 1998) Fill out unknown ratings for individual users with their average ratingsFill out unknown ratings for individual users with their average ratings –Personal diagnosis (Pennock et al., 2000) –Pearson correlation coefficient (Resnick et al., 1994) Only cluster usersOnly cluster users –Aspect model (Hofman & Puzicha, 1999)

25. 25 Experimental Results

26. 26 Summary The sparse data problem is important to collaborative filteringThe sparse data problem is important to collaborative filtering Flexible Mixture Model (FMM) is effectiveFlexible Mixture Model (FMM) is effective –Cluster both users and items simultaneously Decoupling Model (DM) provides additional improvement for collaborative filteringDecoupling Model (DM) provides additional improvement for collaborative filtering –Take into account rating variance among users of similar interests

27. Problem II: How to Identify Users’ Interests based on A Few Rated Items?

28. 28 Identify Users’ Interests To identify the interests of a user, the system needs to ask the user to rate a few itemsTo identify the interests of a user, the system needs to ask the user to rate a few items Given a user is only willing to rate a few items, which one should be asked to solicit rating?Given a user is only willing to rate a few items, which one should be asked to solicit rating?

29. 29 Identify Users’ Interests User 115135 User 231545 User 334324 User 4?5??? To identify the interests of a user, the system needs to ask the user to rate a few itemsTo identify the interests of a user, the system needs to ask the user to rate a few items Given a user is only willing to rate a few items, which one should be asked to solicit rating?Given a user is only willing to rate a few items, which one should be asked to solicit rating?

30. 30 Identify Users’ Interests User 115135 User 231545 User 334324 User 4?5??? To identify the interests of a user, the system needs to ask the user to rate a few itemsTo identify the interests of a user, the system needs to ask the user to rate a few items Given a user is only willing to rate a few items, which one should be asked to solicit rating?Given a user is only willing to rate a few items, which one should be asked to solicit rating?

31. 31 Identify Users’ Interests User 115135 User 231545 User 334324 User 4?5??? To identify the interests of a user, the system needs to ask the user to rate a few itemsTo identify the interests of a user, the system needs to ask the user to rate a few items Given a user is only willing to rate a few items, which one should be asked to solicit rating?Given a user is only willing to rate a few items, which one should be asked to solicit rating?

32. 32 Identify Users’ Interests User 115135 User 231545 User 334324 User 4?5??? To identify the interests of a user, the system needs to ask the user to rate a few itemsTo identify the interests of a user, the system needs to ask the user to rate a few items Given a user is only willing to rate a few items, which one should be asked to solicit rating?Given a user is only willing to rate a few items, which one should be asked to solicit rating?

33. 33 Active Learning Approaches (Ross & Zemel, 2002) Selective samplingSelective sampling – Ask a user to rate the items that are most distinguishable for users’ interests A general strategyA general strategy –Define a loss function that represents the uncertainty in determining users’ interests –Choose the item whose rating will result in the largest reduction in the loss function

34. 34 Active Learning Approach (I) (Jin & Si, 2004) Select the items that have the largest variance in the ratings by the most similar usersSelect the items that have the largest variance in the ratings by the most similar users User 115135 User 231545 User 334324 User 4?5???

35. 35 Active Learning Approach (II) (Jin & Si, 2004) Consider all the training users when selecting items Weight training users by their similarities when computing the “uncertainty” of items User 115135 User 251545 User 324324 User 4?5???

36. 36 A Bayesian Approach for Active Learning (Jin & Si, 2004) Flexible Mixture ModelFlexible Mixture Model –Key is to determine the user class for a test user Let D be the ratings already provided by test user yLet D be the ratings already provided by test user y –D = {(x 1, r 1 ), …, (x k, r k )} Let  be the distribution of user class for test user y estimated based on DLet  be the distribution of user class for test user y estimated based on D –  = {  z = p(z|y)|1  z  m}

37. 37 A Bayesian Approach for Active Learning (Jin & Si, 2004) When the user class distribution  true of the test user is given, we will select the item x* thatWhen the user class distribution  true of the test user is given, we will select the item x* that

38. 38 A Bayesian Approach for Active Learning (Jin & Si, 2004) When the user class distribution  true of the test user is given, we will select the item x* thatWhen the user class distribution  true of the test user is given, we will select the item x* that –  x,r be the distribution of user class for test user y estimated based on D + (x,r)

39. 39 A Bayesian Approach for Active Learning (Jin & Si, 2004) When the user class distribution  true of the test user is given, we will select the item x* thatWhen the user class distribution  true of the test user is given, we will select the item x* that –  x,r be the distribution of user class for test user y estimated based on D + (x,r) –Take into account the uncertainty in rating prediction

40. 40 A Bayesian Approach for Active Learning (Jin & Si, 2004) But, in reality, we never know the true user class distribution  true of the test userBut, in reality, we never know the true user class distribution  true of the test user Replace  true with the distribution p(  |D)Replace  true with the distribution p(  |D) Two types of uncertainties 1.Uncertainty in user class distribution  2.Uncertainty in rating prediction

41. 41 Computational Issues Estimating p(  |D) is computationally expensiveEstimating p(  |D) is computationally expensive Calculating the expectation is also expensiveCalculating the expectation is also expensive

42. 42 Approximate Posterior Distribution (Jin & Si, 2004) Approximate p(  |D) by Laplacian approximationApproximate p(  |D) by Laplacian approximation –Expand the log-likelihood function around its maximum point  *

43. 43 Compute Expectation (Jin & Si, 2004) Expectation can be computed analytically using the approximate posterior distribution p(  |D)Expectation can be computed analytically using the approximate posterior distribution p(  |D)

44. 44 Empirical Studies EachMovie datasetEachMovie dataset –400 training users, and 1600 test users For each test userFor each test user –Initially provides 3 rated items –5 iterations, and 4 items are selected for each iteration Evaluation metricEvaluation metric –Mean Absolute Error (MAE)

45. 45 Baseline Approaches The random selection methodThe random selection method –Randomly select 4 items for each iteration The model entropy methodThe model entropy method –Select items that result in the largest reduction in the entropy of distribution p(  |D) –Only considers the uncertainty in model distribution The prediction entropy methodThe prediction entropy method –Select items that result in the largest reduction in the uncertainty of rating prediction –Only considers the uncertainty in rating prediction

46. 46 Experimental Results

47. 47 Summary Active learning is effective for identifying users’ interestsActive learning is effective for identifying users’ interests It is important to take into account every bit of uncertainty when applying active learning methodsIt is important to take into account every bit of uncertainty when applying active learning methods

48. Problem III How to Combine Collaborative Filtering with Content Filtering?

49. 49 Collaborative Filtering + Content Info. User 1User 2User 3Content 142 A high-school boy is given the chance to write a story about an up-and-coming rock band as he accompanies it on their concert tour. 515 A homicide detective and a fire marshall must stop a pair of murderers who commit videotaped crimes to become media darlings 353 A biography of sports legend, Muhammad Ali, from his early days to his days in the ring 425 A young adventurer named Milo Thatch joins an intrepid group of explorers to find the mysterious lost continent of Atlantis. 554 Benjamin Martin is drawn into the American revolutionary war against his will when a brutal British commander kills his son.

50. 50 Collaborative Filtering + Content Info. User 1User 2User 3Content Information 142 A high-school boy is given the chance to write a story about an up-and-coming rock band as he accompanies it on their concert tour. 515 A homicide detective and a fire marshall must stop a pair of murderers who commit videotaped crimes to become media darlings 353 A biography of sports legend, Muhammad Ali, from his early days to his days in the ring 425 A young adventurer named Milo Thatch joins an intrepid group of explorers to find the mysterious lost continent of Atlantis. 554 Benjamin Martin is drawn into the American revolutionary war against his will when a brutal British commander kills his son.

51. 51 Linear Combination (Good et al., 1999) Build a different prediction model for content information and collaborative informationBuild a different prediction model for content information and collaborative information Linearly combine their predictions togetherLinearly combine their predictions together

52. 52 User 1User 2User 3Content InformationTest user 142 A high-school boy is given the chance to write a story about an up-and-coming rock band as he accompanies it on their concert tour. 1 515 A homicide detective and a fire marshall must stop a pair of murderers who commit videotaped crimes to become media darlings 4 p cf (r|x) p cont (r|x) + Linear Combination (Good et al., 1999)

53. 53 The Co-Training Approach (Hoi, Lyu & Jin, 2005) The linear combination approach ignores the correlation between content information and collaborative informationThe linear combination approach ignores the correlation between content information and collaborative information We propose a Co-training approach for exploiting the correlation between these two types of informationWe propose a Co-training approach for exploiting the correlation between these two types of information

54. 54 Collaborative Filtering + Content Info. User 1 User 2 User 3 Content Information Testuser124 A high-school boy is given the chance to write a story about an up-and-coming rock band as he accompanies it on their concert tour. 1 515 A homicide detective and a fire marshall must stop a pair of murderers who commit videotaped crimes to become media darlings ??3 A biography of sports legend, Muhammad Ali, from his early days to his days in the ring 4 425 A young adventurer named Milo Thatch joins an intrepid group of explorers to find the mysterious lost continent of Atlantis. 543 Benjamin Martin is drawn into the American revolutionary war against his will when a brutal British commander kills his son. 5

55. 55 Collaborative Filtering + Content Info. User 1 User 2 User 3 Content Information TestUser124 A high-school boy is given the chance to write a story about an up-and-coming rock band as he accompanies it on their concert tour. 1 515 A homicide detective and a fire marshall must stop a pair of murderers who commit videotaped crimes to become media darlings ??3 A biography of sports legend, Muhammad Ali, from his early days to his days in the ring 4 425 A young adventurer named Milo Thatch joins an intrepid group of explorers to find the mysterious lost continent of Atlantis. 543 Benjamin Martin is drawn into the American revolutionary war against his will when a brutal British commander kills his son. 5

56. 56 Collaborative Filtering + Content Info. User 1User 2User 3Content Information Test User 124 A high-school boy is given the chance to write a story about an up-and-coming rock band as he accompanies it on their concert tour. 1 515 A homicide detective and a fire marshall must stop a pair of murderers who commit videotaped crimes to become media darlings ??3 A biography of sports legend, Muhammad Ali, from his early days to his days in the ring 4 425 A young adventurer named Milo Thatch joins an intrepid group of explorers to find the mysterious lost continent of Atlantis. 543 Benjamin Martin is drawn into the American revolutionary war against his will when a brutal British commander kills his son. 5

57. 57 Collaborative Filtering + Content Info. User 1User 2User 3Content Information Test User 124 A high-school boy is given the chance to write a story about an up-and-coming rock band as he accompanies it on their concert tour. 1 515 A homicide detective and a fire marshall must stop a pair of murderers who commit videotaped crimes to become media darlings 4 ??3 A biography of sports legend, Muhammad Ali, from his early days to his days in the ring 4 425 A young adventurer named Milo Thatch joins an intrepid group of explorers to find the mysterious lost continent of Atlantis. 543 Benjamin Martin is drawn into the American revolutionary war against his will when a brutal British commander kills his son. 5

58. 58 Collaborative Filtering + Content Info. User 1User 2User 3Content Information Test User 124 A high-school boy is given the chance to write a story about an up-and-coming rock band as he accompanies it on their concert tour. 1 515 A homicide detective and a fire marshall must stop a pair of murderers who commit videotaped crimes to become media darlings 4 ??3 A biography of sports legend, Muhammad Ali, from his early days to his days in the ring 5 425 A young adventurer named Milo Thatch joins an intrepid group of explorers to find the mysterious lost continent of Atlantis. 543 Benjamin Martin is drawn into the American revolutionary war against his will when a brutal British commander kills his son. 5

59. 59 Collaborative Filtering + Content Info. User 1User 2User 3Content Information Test User 124 A high-school boy is given the chance to write a story about an up-and-coming rock band as he accompanies it on their concert tour. 1 515 A homicide detective and a fire marshall must stop a pair of murderers who commit videotaped crimes to become media darlings 4 ??3 A biography of sports legend, Muhammad Ali, from his early days to his days in the ring 5 425 A young adventurer named Milo Thatch joins an intrepid group of explorers to find the mysterious lost continent of Atlantis. 4 543 Benjamin Martin is drawn into the American revolutionary war against his will when a brutal British commander kills his son. 5

60. 60 Coupled Support Vector Machine (Hoi, Lyu & Jin, 2005) Content info.Collaborative info. Representationxixi riri Weightswiwi uiui Rated iterms Unrated iterms Require both the content information and collaborative information to provide consistent prediction for rated items

61. 61 Coupled Support Vector Machine (Hoi, Lyu & Jin, 2005) Require both the content information and collaborative information to provide coherent prediction on unrated items Content info.Collaborative info. Representationxixi riri Weightswiwi uiui Rated iterms Unrated iterms

62. 62 Alternating Optimization (Hoi, Lyu & Jin, 2005) Fix (u, b u ) and estimate optimal (w, b w )Fix (u, b u ) and estimate optimal (w, b w ) Fix (w, b w ) and estimate optimal (u, b u )Fix (w, b w ) and estimate optimal (u, b u ) Quadratic programming

63. 63 Alternating Optimization (Hoi, Lyu & Jin, 2005) Fix (u, b u ) and (w, b w ), estimate ratings Y’ for the unrated itemsFix (u, b u ) and (w, b w ), estimate ratings Y’ for the unrated items –It can be decomposed into a set of optimization problems involved in single variables

64. 64 Empirical Studies DatasetDataset –Images in 20 categories of the COREL dataset –100 images randomly selected from each category –Totally 2000 images Content informationContent information –Image features: colors, edges, and texture Collaborative informationCollaborative information –Log of relevance judgments in the history –150 user sessions, 20 images are judged for each session

65. 65 Evaluation Methodology Evaluation is based on online relevance feedbackEvaluation is based on online relevance feedback 1.A query image is randomly generated 2.20 images are retrieved by a content-based image retrieval (CBIR) system for the given query image 3.A user is asked to judge the relevance of the 20 images to the query image 4.The CBIR system refines the given query using the feedback information from the user, and returns a new set of images 5.The mean average precision of the top returned images is used as the evaluation metric

66. 66 Baseline Methods Euclidean distance (‘Euclidean’)Euclidean distance (‘Euclidean’) –Measure the similarity between images using the Euclidean distance in low-level image features –Neither relevance feedback nor log information is used Relevance feedback by a support vector machine (‘RF-SVM’)Relevance feedback by a support vector machine (‘RF-SVM’) –Build a support vector machine (SVM) based on the users’ feedback –Only utilizes relevance feedback information Linear combination approach (‘LRF-2SVM’)Linear combination approach (‘LRF-2SVM’) –Build SVM models that are based on relevance feedback information and log information –Linearly combine their predictions

67. 67 Experimental Results Coupled Support Vector Machine

68. 68 Summary Combining content information and collaborative filtering is important for predicting users’ interestsCombining content information and collaborative filtering is important for predicting users’ interests It is important to exploit the correlation between content information and collaborative information.It is important to exploit the correlation between content information and collaborative information.

69. Conclusion

70. 70 Conclusion Collaborative judgments are extremely valuable informationCollaborative judgments are extremely valuable information –Provide alternative representation of items in addition to their content –Are more related to human perception than content information It is particularly usefulIt is particularly useful –When content information is not available –When content information is difficult to analyze e.g., imagese.g., images

71. 71 Conclusion Carefully designed learning algorithms are the key to exploit collaborative informationCarefully designed learning algorithms are the key to exploit collaborative information –Sparse data & rating variance  mixture models –Identify users’ interests  active learning –Exploit content information  co-training

72. 72 Existing Challenges Large-sized data for collaborative filtering –Scalability –Large diversity in users’ interests –Large diversity in the content of items Mixed types of users’ feedback –Ratings, ranking, textual notations, … The privacy issue

73. 73 Acknowledgement Luo SiLuo Si Chengxiang ZhaiChengxiang Zhai Jamie CallanJamie Callan Alex G. HauptmannAlex G. Hauptmann Joyce Y. ChaiJoyce Y. Chai Steven C.H. HoiSteven C.H. Hoi Michael R. LyuMichael R. Lyu

74. 74 Reference Hoi, C.H., M. R. Lyu, and R. Jin (2005), Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval, in the 1st IEEE International Workshop on Managing Data for Emerging Multimedia Applications (EMMA 2005) (invited paper)Hoi, C.H., M. R. Lyu, and R. Jin (2005), Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval, in the 1st IEEE International Workshop on Managing Data for Emerging Multimedia Applications (EMMA 2005) (invited paper) Jin, R. and L. Si (2004), A Study of Methods for Normalizing User Ratings in Collaborative Filtering, in the Proceedings of The 27th Annual International ACM SIGIR Conference (SIGIR 2004) Sheffield, UK.Jin, R. and L. Si (2004), A Study of Methods for Normalizing User Ratings in Collaborative Filtering, in the Proceedings of The 27th Annual International ACM SIGIR Conference (SIGIR 2004) Sheffield, UK. Jin, R., J. Y. Chai, and L. Si (2004), An Automated Weighting Scheme for Collaborative Filtering, in the Proceedings of the 27th Annual International ACM SIGIR Conference (SIGIR 2004) Sheffield, UK.Jin, R., J. Y. Chai, and L. Si (2004), An Automated Weighting Scheme for Collaborative Filtering, in the Proceedings of the 27th Annual International ACM SIGIR Conference (SIGIR 2004) Sheffield, UK. Jin, R. and L. Si (2004), A Bayesian Approach toward Active Learning for Collaborative Filtering, in the Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence (UAI 2004) Banff, Alberta, Canada.Jin, R. and L. Si (2004), A Bayesian Approach toward Active Learning for Collaborative Filtering, in the Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence (UAI 2004) Banff, Alberta, Canada. Jin, R., L. Si, C.X. Zhai, and J. Callan (2003), Collaborative Filtering with Decoupled Models for Preferences and Ratings, the Twelfth International Conference on Information and Knowledge Management (CIKM 2003), 2003Jin, R., L. Si, C.X. Zhai, and J. Callan (2003), Collaborative Filtering with Decoupled Models for Preferences and Ratings, the Twelfth International Conference on Information and Knowledge Management (CIKM 2003), 2003 L. Si, and R. Jin (2003), Product Space Mixture Model for Collaborative Filtering, the Twentieth International Conference on Machine Learning (ICML 2003),Washington, DC USA.L. Si, and R. Jin (2003), Product Space Mixture Model for Collaborative Filtering, the Twentieth International Conference on Machine Learning (ICML 2003),Washington, DC USA. Jin, R., L. Si and C.X. Zhai (2003), Preference-based Graphic Models for Collaborative Filtering, the 19th Conference on Uncertainty in Artificial Intelligence (UAI 2003), Acapulco,Mexico.Jin, R., L. Si and C.X. Zhai (2003), Preference-based Graphic Models for Collaborative Filtering, the 19th Conference on Uncertainty in Artificial Intelligence (UAI 2003), Acapulco,Mexico.

75. 75 Reference Ross, D. A. and R. S. Zemel (2002). Multiple-cause Vector Quantization. In Advances in Neural Information Processing Systems 15.Ross, D. A. and R. S. Zemel (2002). Multiple-cause Vector Quantization. In Advances in Neural Information Processing Systems 15. Good, N., J. Schafer, J. Konstan, S. Borchers, B. Sarwar, J. Herlocker and J. Riedl. (1999). Combining collaborative filtering with personal agents for better recommendations. In Proceedings of the 16th National Conference on Artificial Intelligence.Good, N., J. Schafer, J. Konstan, S. Borchers, B. Sarwar, J. Herlocker and J. Riedl. (1999). Combining collaborative filtering with personal agents for better recommendations. In Proceedings of the 16th National Conference on Artificial Intelligence. Hofmann, T., & J. Puzicha (1999). Latent Class Models for Collaborative Filtering. In the Proceedings of International Joint Conference on Artificial Intelligence.Hofmann, T., & J. Puzicha (1999). Latent Class Models for Collaborative Filtering. In the Proceedings of International Joint Conference on Artificial Intelligence. Breese, J. S., D. Heckerman, C. Kadie (1998). Empirical Analysis of Predictive Algorthms for Collaborative Filtering. In the Proceeding of the Fourteenth Conference on Uncertainty in Artificial IntelligenceBreese, J. S., D. Heckerman, C. Kadie (1998). Empirical Analysis of Predictive Algorthms for Collaborative Filtering. In the Proceeding of the Fourteenth Conference on Uncertainty in Artificial Intelligence Resnick, P., N. Iacovou, M. Suchak, P. Bergstrom, & J. Riedl (1994) Grouplens: An Open Architecture for Collaborative Filtering of Netnews. In Proceeding of the ACM 1994 Conference on Computer Supported Cooperative Work.Resnick, P., N. Iacovou, M. Suchak, P. Bergstrom, & J. Riedl (1994) Grouplens: An Open Architecture for Collaborative Filtering of Netnews. In Proceeding of the ACM 1994 Conference on Computer Supported Cooperative Work. Pennock, D. M., E. Horvitz, S. Lawrence, & C.L. Giles (2000) Collaborative Filtering by Personality Diagnosis: A Hybrid Memory- and Model-Based Approach. In the Proceeding of the Sixteenth Conference on Uncertainty in Artificial Intelligence.Pennock, D. M., E. Horvitz, S. Lawrence, & C.L. Giles (2000) Collaborative Filtering by Personality Diagnosis: A Hybrid Memory- and Model-Based Approach. In the Proceeding of the Sixteenth Conference on Uncertainty in Artificial Intelligence.