1. Algorithms for Position Bias Correction in Ranking Anirban Majumder Machine Learning, Amazon

2. User Created Content @Amazon

3. Content Ranking Rank content to improve customer experience Ranking Algorithms Rank Content User feedback Clicks, purchase …

4. Content Ranking #Impressions = 1000 #clicks = 20 CTR = 0.02 #Impressions = 1000 #clicks = 10 CTR = 0.01 #Impressions = 1000 #clicks = 1 CTR = 0.001

5. Content Ranking #Impressions = 1000 #clicks = 12 CTR = 0.012 #Impressions = 1000 #clicks = 10 CTR = 0.010 #Impressions = 1000 #clicks = 5 CTR = 0.005 What happens if we change the ordering ?

6. Presentation Bias Eye-tracking experiment 1 on search result listing 1 “Google Eye Tracking Report”, by Enquiro, Eyetools, Did-It, 2005 ImpressionAction View Implication Self-fulfilling prophecy !

7. Examination Model 2 Sample a position p Examine the item k at the position Take an action (purchase/no-purchase, click/no-click) Item Factor Position Factor

8. Beta-Poisson Model A k,p : #actions (of item k from position p) I k,p : #impressions

9. Beta-Poisson : Inference Exact posterior computation is difficult – interaction between α and β Variational Bayes approximation

10. Data Sparsity Position data is often sparse Items appear in few positions Not possible to estimate the bias Use feature-based representation of Items Product category information Item price Review rating, review text

11. Leveraging Side Information Items with similar feature vectors can share position data Relevance Factor Position Factor

12. Experiments Experiments were performed on Amazon deals data. Data set DealsImpressionsSessionsPositions 9.3k315M11M350

13. Offline Experiments : Predicting Purchase Rate Baselines – Click-over-Expected-Click (COEC 6 ) – Gamma-Poisson 7 6 “Comparing Click Logs and Editorial Labels for Training Query Rewriting”, Zhang et al, WWW’07 7 “Position-Normalized Click Prediction in Search Advertising”, by Chen et al, KDD ‘12

14. Predicting Purchase Rate : Results Gamma- Poisson Beta-PoissonBeta-Poisson + features RMSE12.5x1.2x1.0 KL11.0x1.1x1.0 Accuracy of different bias correction models in predicting purchase rate at position 1 as measured by RMSE and KL divergence. MRRP@2P@5 COEC 0.85x0.94x0.96x Gamma-Poisson 0.91x0.94x0.95x Beta-Poisson 0.94x0.95x0.96x Beta-Poisson + features 1.0 The predicted purchase rate is used to rank deals. The performance is measured in terms of Mean Reciprocal Rank and precision@k

15. Conclusion Position bias correction is important for many ranking problems. Position data can be sparse and will affect bias correction – use a feature based representation.

16. Backup

17. Methodology Held- out set Training Data number of Impressions, purchases Position 1