Algorithms for Position Bias Correction in Ranking Anirban Majumder Machine Learning, Amazon

User Created

Content Ranking Rank content to improve customer experience Ranking Algorithms Rank Content User feedback Clicks, purchase …

Content Ranking #Impressions = 1000 #clicks = 20 CTR = 0.02 #Impressions = 1000 #clicks = 10 CTR = 0.01 #Impressions = 1000 #clicks = 1 CTR = 0.001

Content Ranking #Impressions = 1000 #clicks = 12 CTR = #Impressions = 1000 #clicks = 10 CTR = #Impressions = 1000 #clicks = 5 CTR = What happens if we change the ordering ?

Presentation Bias Eye-tracking experiment 1 on search result listing 1 “Google Eye Tracking Report”, by Enquiro, Eyetools, Did-It, 2005 ImpressionAction View Implication Self-fulfilling prophecy !

Examination Model 2 Sample a position p Examine the item k at the position Take an action (purchase/no-purchase, click/no-click) Item Factor Position Factor

Beta-Poisson Model A k,p : #actions (of item k from position p) I k,p : #impressions

Beta-Poisson : Inference Exact posterior computation is difficult – interaction between α and β Variational Bayes approximation

Data Sparsity Position data is often sparse Items appear in few positions Not possible to estimate the bias Use feature-based representation of Items Product category information Item price Review rating, review text

Leveraging Side Information Items with similar feature vectors can share position data Relevance Factor Position Factor

Experiments Experiments were performed on Amazon deals data. Data set DealsImpressionsSessionsPositions 9.3k315M11M350

Offline Experiments : Predicting Purchase Rate Baselines – Click-over-Expected-Click (COEC 6 ) – Gamma-Poisson 7 6 “Comparing Click Logs and Editorial Labels for Training Query Rewriting”, Zhang et al, WWW’07 7 “Position-Normalized Click Prediction in Search Advertising”, by Chen et al, KDD ‘12

Predicting Purchase Rate : Results Gamma- Poisson Beta-PoissonBeta-Poisson + features RMSE12.5x1.2x1.0 KL11.0x1.1x1.0 Accuracy of different bias correction models in predicting purchase rate at position 1 as measured by RMSE and KL divergence. COEC 0.85x0.94x0.96x Gamma-Poisson 0.91x0.94x0.95x Beta-Poisson 0.94x0.95x0.96x Beta-Poisson + features 1.0 The predicted purchase rate is used to rank deals. The performance is measured in terms of Mean Reciprocal Rank and

Conclusion Position bias correction is important for many ranking problems. Position data can be sparse and will affect bias correction – use a feature based representation.

Backup

Methodology Held- out set Training Data number of Impressions, purchases Position 1