Predicting Sequential Rating Elicited from Humans Aviv Zohar & Eran Marom

2 Motivation Recommender systems base recommendations on information collected from users Humans are not consistent when rating items Framing problem (scaling) Relative ratings, not absolute Short memory Most recommender systems usually ignore the sequential aspect of the rating process We wanted to find out if we can use this information to better predict unseen future ratings

3 Collected Data 12 quotes of different qualities were chosen Each user rates all quotes in a random order Quotes are easy and quick to rank A total of 500 people completed the survey Binned the ratings to 5 values

4 Simple Analysis of the Data A naïve classifier that picks the most common rating is right 33.8% of the time Ordered data seems to have added information

5 The Graphical Model q r t Quote type/quality (unknown) Quote rating Person type (unknown) q1q1 q2q2 q3q3 t1t1 r11r11 r21r21 r31r31 r41r41 q4q4 t2t2 r12r12 r22r22 r32r32 r42r42

6 Dependencies in the Model V-Structures q2q2 t1t1 r21r21 t2t2 r32r32 Given the data, and not the quotes qualities, every path between types and qualities is open When quotes qualities are given types are independent and vice versa Conclusion: types are hard to learn together with qualities The learning is exponential in min(N People,N Quotes )

7 Likelihood of the Data

8 EM - Learning Parameters (known types) Expectation Maximization

9 Selecting Strict People Types Cleverly Using additional information on users Clustering using ratings

10 How to Predict the Next Rating q1q1 q2q2 q3q3 ? r1r1 r2r2 ? ? q4q4 We want to predict this Unknown, can be estimated from previous ratings We have a prior from past data

11 How to Predict the Next Rating Use the previous ratings by the user to estimate the user type, and then predict the following rating

12 Experiments Learning Process: Initialized with random, almost uniform, priors Final network selected by max likelihood from 100 EM runs Score: percentage of correct predictions 5-fold cross validation was used

13 Results for Learning with Strictly Selected People Types To check if the order information helps us with prediction, tests were repeated with scrambled order Only a slight difference was found True order – 37.98% Permuted order – 37.40% The type of each person is better estimated with each new rating given – so we ’ ll make less mistakes over time

14 Adaptively Assigning People Types Multi-phase EM Alternating between optimal selection of types and optimal selection of parameters

15 Results for Learning with Dynamic Adaptation of People Types Average score is still 0.5% better for the ordered run Could not see improvements in predictions compared to constant people types deduced from k-means General rise in score with progress of ratings is seen, though other underlying aspects of the data have effect Permutated Ordered

16 Numerical Problems when Increasing the Data Size Insufficient data for learning the parameter space 11 rating values, 3 quote types, 2 person types Number of parameters to learn: 11  12  3  2  3  795 Number of predictions from 500 people: 500  12  6000 Precision problems Alas, using more than 400 people implies such small probabilities that we have to use infinite precision tools

17 Conclusions and Possible Extensions A significant improvement is already gained by very naïve predictors, but still we can do better Adding order dependencies improves predictions only slightly Multi-phase learning could be done using strict quotes qualities while using types probabilities for each person Model can be extended to encompass more aspects of sequential information More data may yield better results Algorithms ran fast – system is scalable to larger data sets