1. Thompson Sampling for learning in online decision making
Shipra Agrawal
IEOR and Data Science Institute
Columbia University

2. Learning from sequential interactions
Tradeoff between
information and revenue
learning and optimization
Exploration and exploitation

3. Managing exploitation-exploitation tradeoff
The multi-armed bandit problem (Thompson 1933; Robbins 1952)
Multiple rigged slot machines in a casino.
Which one to put money on?
Try each one out
An old crisp and clean formulation of the problem in statistics
WHEN TO STOP TRYING (EXPLORATION) AND START PLAYING (EXPLOITATION)?

4. Stochastic multi-armed bandit problem
Online decisions
At every time step 𝑡=1,…, 𝑇, pull one arm out of 𝑁 arms
Stochastic feedback
For each arm 𝑖, reward is generated i.i.d from a fixed but unknown distribution support [0,1], mean 𝜇 𝑖
Bandit feedback
Only the reward of the pulled arm can be observed

5. Measure of Performance
Maximize expected reward in time 𝑇
𝐸[ 𝑡=1 𝑇 𝜇 𝑖 𝑡 ]
Minimize expected regret in time 𝑇
Optimal arm is the arm with expected reward 𝜇 ∗ = max 𝑗 𝜇 𝑗
Expected regret for playing arm 𝑖: Δ 𝑖 = 𝜇 ∗ − 𝜇 𝑖
Expected regret in any time 𝑇,
𝑅𝑒𝑔𝑟𝑒𝑡 𝑇 = 𝑖 Δ 𝑖 𝑡 = 𝑖 Δ 𝑖 𝐸[ 𝑘 𝑖 𝑇 ]
= 𝑖=1 𝑁 𝜇 𝑖 𝐸[𝑘 𝑖 (𝑇)
Any time algorithm: the time horizon 𝑇 is not known

6. Summary of the talk
Introduction to Thompson Sampling as a very general algorithmic methodology for MAB
Provably near-optimal adaptations for
Linear Contextual Bandits
Assortment Selection
Reinforcement Learning
Focus on algorithmic methodology

7. References Papers which this talk derives from.. Provably optimal
[A., and Goyal COLT 2012, AISTATS 2013, JACM 2017]
Handling personalization, contexts, large number of products
[A. and Goyal ICML 2013] [Kocák, Valko, Munos, A. AAAI 2014]
Assortment selection: consumer choice behavior
[A., Avandhanula, Goyal, Zeevi EC 2016, COLT 2017]
Reinforcement Learning
[A. Jia, NIPS 2017]
(Many) Other related research articles
[Li et al. 2011, Kaufmann et al. 2012, Russo and Van Roy, 2013, 2014, 2016, Osband and Van Roy 2016,….]
A central piece of my investigation into this problem has been analysis of a very old and popular Bayesian heusristic for this problem called Thompson Sampling. Despite being easy to implement and good empirical performance, this heuristic did not have rigorous theoretical undetrstanding. And, our work showed that this simple and popular heuristic achieves the optimal regret bounds possible for every instance of the multi-armed bandit problem
Of course when this heuristic was proposed the popular concerns of today – personalization and millions of products were not considered. Several of my work focus on extensions and analysis of this heuristic to incorporate these important aspects.
In addition to extensions in the direction I mentioned so far, one exciting direction I am working is in handling exploration-exploitation in reinforcement learning. This is a more complex and general model of computers learning from their own past mistakes, which forms a popular models for learning to play games.

8. The need for exploration
Two arms black and red
Random rewards with unknown mean 𝝁 𝟏 =𝟏.𝟏, 𝝁 𝟐 =𝟏
Optimal expected reward in 𝑇 time steps is 1.1×𝑇
Exploit only strategy: use the current best estimate (MLE/empirical mean) of unknown mean to pick arms
Initial few trials can mislead into playing red action forever
1.1, 1, 0.2,
1, 1, 1, 1, 1, 1, 1, ……
Expected regret in 𝑇 steps is close to 0.1×𝑇
2/4/2019

9. Exploration-Exploitation tradeoff
Exploitation: play the empirical mean reward maximizer
Exploration: play less explored actions to ensure empirical estimates converge
2/4/2019

10. Thompson Sampling [Thompson, 1933]
Natural and Efficient heuristic
Maintain belief about effectiveness (mean reward) of each arm
Observe feedback, update belief of pulled arm i in Bayesian manner
Pull arm with posterior probability of being best arm
NOT same as choosing the arm that is most likely to be best
If a customer likes the product update the belief to be more favorable for the product

11. Bernoulli rewards, Beta priors
Uniform distribution 𝐵𝑒𝑡𝑎(1,1)
𝐵𝑒𝑡𝑎 𝛼,𝛽 prior ⇒ Posterior
𝐵𝑒𝑡𝑎 𝛼+1,𝛽 if you observe 1
𝐵𝑒𝑡𝑎 𝛼,𝛽+1 if you observe 0
Start with 𝐵𝑒𝑡𝑎(1,1) prior belief for every arm
In round t,
For every arm 𝑖, sample 𝜃 𝑖,𝑡 independently from posterior B𝑒𝑡𝑎 𝑆 𝑖,𝑡 +1, 𝐹 𝑖,𝑡 +1
Play arm 𝑖 𝑡 = max 𝑖 𝜃 𝑖,𝑡
Observe reward and update the Beta posterior for arm 𝑖 𝑡

12. Thompson Sampling with Gaussian Priors
Suppose reward for arm 𝑖 is i.i.d. 𝑁( 𝜇 𝑖 ,1)
Starting prior N(0,1)
Gaussian Prior, Gaussian likelihood → Gaussian posterior 𝑁( 𝜇 𝑖,𝑡 , 1 𝑛 𝑖,𝑡 +1 )
𝜇 𝑖,𝑡 is empirical mean of 𝑛 𝑖,𝑡 observations for arm 𝑖
Algorithm
Sample 𝜇 𝑖 from posterior 𝑁 𝜇 𝑖,𝑡 , 1 𝑛 𝑖,𝑡 +1 for arm 𝑖
Play arm 𝑖 𝑡 = arg⁡ max 𝑖 ⁡ 𝜇 𝑖
Observe reward, update empirical mean for arm 𝑖 𝑡

13. Thompson Sampling [Thompson 1933]
With more trials posteriors concentrate on the true parameters
Mode is MLE: enables exploitation
Less trials means more uncertainty in estimates
Spread/variance captures uncertainty: enables exploration
Gives benefit of doubt those less explored
OPTIMAL benefit of doubt
[A. and Goyal, COLT 2012, AISTATS 2013, JACM 2017]
A sample from the posterior is used as an estimate for unknown parameters to make decisions
2/4/2019

14. Our Results [A. Goyal COLT 2012, AISTATS 2013, JACM 2017]
Optimal instance-dependent bounds for Bernoulli rewards
Regret 𝑇 ≤ 𝒍𝒏(𝑻) 1+𝜖 𝑖 Δ 𝑖 𝐾𝐿( 𝜇 ∗ || 𝜇 𝑖 ) +𝑂( 𝑁 𝜖 2 )
Matches asymptotic lower bound [Lai Robbins 1985]
UCB algorithm achieves this only after careful tuning [Kaufmann et al. 2012]
Arbitrary bounded reward distribution
Regret 𝑇 ≤ 𝑂(𝒍𝒏(𝑻) 𝑖 1 Δ 𝑖 )
Matches the best available for UCB for general reward distributions
Instance-independent bounds
Regret 𝑇 ≤ 𝑂( 𝑁𝑇ln 𝑇 )
Lower bound Ω( 𝑁𝑇 )
Prior and likelihood mismatch allowed!
Simple extension to arbitrary distribution with [0,1] support

15. Further challenges Scalability and contextual response
TS for Linear contextual bandits [A. and Goyal ICML 2013, Kocák, Valko, Munos, A., AAAI 2014]
Externality in assortment of arms
TS for dynamic assortment selection in MNL choice model [Avadhanula, A., Goyal, Zeevi, COLT 2017]
State-dependent response
Learning MDPs (Reinforcement learning) [A. and Jia, NIPS 2017]
Other
Budget/supply constraints, nonlinear utilities [EC 2014, SODA 2015, NIPS 2016, COLT 2016]
Delayed anonymous feedback [ICML 2018]

16. Further challenges #1 Scalability and context
Large number of products and customer types, external conditions
Utilize similarity?
Content based recommendation
Customers and products described by their features
Similar features means similar preferences
Parametric models mapping customer and product features to customer preferences
Contextual bandits
Exploration-exploitation to learn the parametric models
Large number of arms but related
Customization ``Recommended just for you”
Response depends not just on assortment but on customer profile (context)
Customer preference for product depends on (customer,product) features and unknown parameters

17. Linear Contextual Bandits
N arms, possibly very large N
A d-dimensional context (feature vector) 𝑥 𝑖,𝑡 for every arm 𝑖, time 𝑡
Linear parametric model
Unknown parameter 𝜃
Expected reward for arm 𝑖 at time 𝑡 is 𝑥 𝑖,𝑡 ⋅ 𝜃
Algorithm picks 𝑥 𝑡 ∈ {𝑥 1,𝑡 , …, 𝑥 𝑁,𝑡 }, observes 𝑟 𝑡 = 𝑥 𝑡 ⋅𝜃+ 𝜂 𝑡
Optimal arm depends on context: 𝑥 𝑡 ∗ =arg max 𝑥 𝑖,𝑡 𝑥 𝑖,𝑡 ⋅𝜃
Goal: Minimize regret
Regret 𝑇 = 𝑡 ( 𝑥 𝑡 ∗ ⋅𝜃− 𝑥 𝑡 ⋅𝜃 )
Arm are products,
every time 𝑡 a new user arrives
Optimal arm at any time is the arm with the best features at that time

18. Thompson Sampling for linear contextual bandits
Least square solution 𝜃 𝑡 of set of 𝑡−1 equations
𝑥 𝑠 ⋅𝜃= 𝑟 𝑠 , 𝑠=1, …, 𝑡−1
𝜃 𝑡 ≃ 𝐵 𝑡 −1 ( 𝑠=1 𝑡−1 𝑥 𝑠 𝑟 𝑠 ) where 𝐵 𝑡 =𝐼+ 𝑠=1 𝑡−1 𝑥 𝑠 𝑥 𝑠 ′
𝐵 𝑡 −1 covariance matrix of this estimator
Gaussian Prior, Gaussian Likelihood, Gaussian Posterior
𝑁 0, 𝐼 starting prior on 𝜃,
Reward distribution given 𝜃, 𝑥 𝑖,𝑡 : 𝑁 𝜃 𝑇 𝑥 𝑖,𝑡 , 1 ,
posterior on 𝜃 at time t is 𝑁 𝜃 𝑡 , 𝐵 𝑡 −1
An observation that ties Least squares estimation nicely with TS is that

19. Thompson Sampling for linear contextual bandits [A. Goyal ICML 2013]
Algorithm: At Step t,
Sample 𝜃 𝑡 from 𝑁( 𝜃 𝑡 , 𝑣 2 𝐵 𝑡 −1 )
Pull arm with feature 𝑥 𝑡 where
𝑥 𝑡 = max 𝑥 𝑖,𝑡 𝑥 𝑖,𝑡 ⋅ 𝜃 𝑡
Apply this algorithm for any likelihood , starting prior 𝑁 0, 𝑣 2 𝐼 !
Use slightly scaled varianace

20. Our Results [A. Goyal ICML 2013]
With probability 1−𝛿, regret
𝑅𝑒𝑔𝑟𝑒𝑡 𝑇 ≤ 𝑂 𝑑 3/2 𝑇
Any likelihood, unknown prior, only assumes bounded or sub-Gaussian noise
No dependence on number of arms
Lower bound Ω 𝑑 𝑇
For UCB, best bound 𝑂 𝑑 𝑇 [Dani et al 2008, Abbasi-Yadkori et al 2011]
Best earlier bound for a polynomial time algorithm 𝑂 𝑑 3/2 𝑇 [Dani et al ]
𝑂 𝑑 3/2 𝑇 ln⁡(𝑇) ln 𝑇 𝛿

21. Further challenges #2 Assortment selection as multi-armed bandit
Arms are products
Limited display space, k products at a time
Challenge: Customer response on one product is influenced by other products in assortment
Arms are no longer independent
Amazon product example
Combinatorially large number of product combinations
Cannot learn about every product combination
How to learn about individual product?

24. Assortment optimization
Multinomial logit choice model
𝑝 𝑖 𝑆 = 𝑒 𝑣 𝑖 𝑗 𝑒 𝑣 𝑗
Independence of irrelevant alternatives
Expected revenue on offering assortment 𝑆
𝑅 𝑆,𝒗 = 𝑖=1 𝑁 𝑟 𝑖 𝑝 𝑖 𝑆
Optimal assortment 𝑆 ∗ = arg max 𝑆 ≤𝑘 𝑅(𝑆, 𝒗)
More generally: product i (feature vector 𝑥 𝑖,𝑡 ) in assortment S
𝑝 𝑖,𝑡 𝑆 = 𝑒 𝜃 𝑇 𝑥 𝑖 ,𝑡 1+ 𝑗 𝑒 𝜃 𝑇 𝑥 𝑗,𝑡
Probability of choosing Log ratio is linear in features

25. Online assortment selection
N products, Unknown parameters 𝑣 1 , 𝑣 2 , …, 𝑣 𝑁
At every step 𝒕,
recommend an assortment 𝑆 𝑡 of size at most K,
observe customer choice 𝑖 𝑡 , revenue 𝑟 𝑖 𝑡 (expected value 𝑅 𝑆 𝑡 ,𝒗 )
update parameter estimates
Goal:
optimize total expected revenue E[ 𝑡 𝑟 𝑖 𝑡 ]
or minimize regret compared to optimal assortment
S ∗ = arg max S 𝑖 𝑟 𝑖 𝑝 𝑖 𝑆 =R(S,𝐯)

26. Thompson Sampling structure
Initialization: Specify belief on parameters 𝐯
Prior distribution P(𝐯) on 𝐯
Step t:
Sample 𝐯 from posterior belief P(𝐯| H t−1 )
Offer optimal assortment consistent with beliefs.
S = argmax S R(S, 𝐯 )
Observe purchase/no purchase 𝑟 𝑡 , update belief
P 𝐯 H t ) α P 𝑟 𝑡 𝐯)×P(𝐯| 𝐇 t−1 )

27. Main challenge and techniques
Censored feedback: Feedback for product 𝑖 effected by other products in assortment
Getting unbiased estimate of 𝑣 𝑖 / 𝑣 0 by repeated offering until no purchase
What is a good structure of prior/posterior?
Correlated priors
Boosted sampling
Difficult to update posterior
Approximate Gaussian posterior updates

28. TS Based Algorithm Initialize 𝜇 𝑖 =1, 𝜎 𝑖 2 =1 for 𝑖=1,…, 𝑁, Epoch ℓ=1
(Correlated Sampling)
Sample 𝐾 samples from Gaussian θ k ~ N(0,1).
v i = max k μ i + θ k 𝜎 𝑖
Offer repeatedly 𝑆 ℓ = arg max 𝑆 𝑅(𝑆, 𝑣 ) till no purchase occurs.
Track 𝑛 𝑖,ℓ : number of times product i is chosen
For 𝑖∈ 𝑆 ℓ , update 𝜇 𝑖 = ℓ 𝑛 𝑖,ℓ L i , σ 𝑖 2 =Var ℓ n i,𝑙 𝐿 𝑖 ; Repeat
That question is the focus of this talk. Specifically, can we design a policy that suggests S_t as a function of history, to learn v and maximize the cumulative revenue over the selling period.

29. Results: Dynamic assortment selection [A
Results: Dynamic assortment selection [A., Avadhanula, Goyal, Zeevi 2016, 2017]
Assumption: 𝑣 𝑖 ≤ 𝑣 0 (no purchase is more likely than single product purchase)
Upper Bound: For any 𝐯, Reg T,𝐯 ≤𝑂( NT log T )
Lower Bound [Chen and Wang 2017]: Reg T,𝐯 ≥Ω( NT )

30. Different versions of TS (N=1000, K=10, random instances)

31. Homepage Optimization at Flipkart
Assortment of Widgets
Widgets
Hence for this home page team, the other teams are pretty much black-boxes and their role is in picking the right subset of widgets. Often the requests can be competing and the screen space very limited.

32. Substitution Effects Demand Cannibalization: Similar recommendations.
For example, two different teams can push similar products like offers for you/discounts for you. If we don’t account for substitution we might end up wrongly estimating the attractiveness of these banners.

33. Numerical simulation

34. Numerical simulations
(assortment optimization)
(assortment optimization)

35. Further challenges #3 State dependent response: Reinforcement learning
Rounds 𝑡=1,…, 𝑇
Observe state take an action
Observe response: reward and new state
Markov model: Response to an action depends on the state of the system
Applications: autonomous vehicle control, robot navigation, personalized medical treatments, inventory management, dynamic pricing…
2/4/2019

36. The reinforcement learning problem
System dynamics given by an unknown MDP 𝑆,𝐴, 𝑃, 𝑟, 𝑠 0
Rounds 𝑡=1,…, 𝑇. In round 𝑡,
observe state s 𝑡 , take action 𝑎 𝑡 ,
observe reward 𝑟 𝑡 ∈[0,1], E 𝑟 𝑡 = 𝒓 𝒔 𝒕 , 𝒂 𝒕
observe the transition to next state 𝑠 𝑡+1 with probability 𝐏 𝒔 𝒕 , 𝒂 𝒕 , ( 𝒔 𝒕+𝟏 ) S4 S6 S5 S2 S3 S1
0.5, $1
0.5, $2
1, $0
6 states,
2 actions
2/4/2019

37. The reinforcement learning problem
Solution concept: optimal policy
Which action to take in which state
𝜋 𝑡 :𝑆→𝐴 , Action 𝜋 𝑡 𝑠 in state 𝑠
Unknown reward distributions 𝑟 𝑠,𝑎 and unknown transition function 𝑃 𝑠,𝑎 , unknown optimal policy 𝜋 ∗
Learn to solve the MDP from observations over 𝑇 rounds
Goal: maximize total reward over 𝑇 rounds
Minimize regret in 𝑇 rounds: difference between total reward for playing optimal policy for all rounds vs. algorithm’s total reward
2/4/2019

38. The need for exploration
Uncertainty in rewards, state transitions
Unknown reward distribution, transition probabilities
Exploration-exploitation:
Explore actions/states/policies, learn reward distributions and transition model
Exploit the (seemingly) best policy
0.6, $0
1.0, $1
0.4, $0 S1 S2
0.1, $0
Here is a simple example to illustrate this problem. We have a simple underlying MDP. Let’s say the game starts in state s1. and There are two actions available action 1, which is in black can take the process in state2 with some probability, where the rewards are very high, therefore clearly it is a better action. However, its expected immediate reward is bad. Therefore, the optimal policy should be to take action 1 in state s1, and s2.
lets see what would Q-learning do.
Initially, say all the Q values are initialized to 0 by the algorithm.
Q(s1,a1)=0
Q(s1,a2)=0
Q(s2,a1)=0
At least the algorithm does not know that state s2 is so good, or even that there is a way to go from state s1 to s2. Suppose the algorithm tries the first action (the black action) initially, but in these few trials it does not observe a transfer to s2, so Q value of s2 never gets updated and so that the Q-value of s1,a1 is updated again and again to -1.
Q(s1,a1)=-1
Therefore, the algorithm tries a2, and observes +1 reward, updates Q value to +1.
Q(s1,a2)=1
So, the algorithm chooses a2 again, the Q value is updated to
1+gamma 1 + \gamma^2 1 + …. = \frac{1}{1-\gamma}
Since a2 is not played and s2 is not visited its Q value doesn’t get updated.
This example may seem extreme, but this situation can actually easily occur, even in more reasonable looking examples. Before enough samples are obtained, the less explored states or actions might appear bad due to empirical errors. If these estimates are used greedily to decide actions, they will inevitably pick the suboptimal actions. And since the estimates are updated only when a state is visited or an action is played, the estimates for those unexplored states and actions never improve.
1.0, $1
0.9, $100
2/4/2019

39. Posterior Sampling Assume for simplicity: Known reward distribution
Needs to learn the unknown transition probability vector 𝑃 𝑠,𝑎 =(𝑃 𝑠,𝑎 1 , …, 𝑃 𝑠,𝑎 (𝑆)) for all 𝑠,𝑎
𝑆 2 𝐴 parameters
In any state 𝑠 𝑡 =𝑠, 𝑎 𝑡 =𝑎, observes new state 𝑠 𝑡+1
outcome of a Multivariate Bernoulli trial with probability vector 𝑃 𝑠,𝑎
2/4/2019

40. Posterior Sampling with Dirichlet priors
Given prior Dirichlet 𝛼 1 , 𝛼 2 , …, 𝛼 𝑆 on 𝑃 𝑠,𝑎
After a Multinoulli trial with outcome (new state) 𝑖, Bayesian posterior on 𝑃 𝑠,𝑎
Dirichlet 𝛼 1 , 𝛼 2 , …, 𝛼 𝑖 +1, … , 𝛼 𝑆
After 𝑛 𝑠,𝑎 = 𝛼 1 +…+ 𝛼 𝑆 observations for a state-action pair 𝑠,𝑎
Posterior mean vector is empirical mean
𝑃 𝑠,𝑎 (𝑖)= 𝛼 𝑖 𝛼 1 +…+ 𝛼 𝑆 = 𝛼 𝑖 𝑛 𝑠,𝑎
variance bounded by 1 𝑛 𝑠,𝑎
With more trials of 𝑠,𝑎, the posterior mean concentrates around true probability
2/4/2019

41. Posterior Sampling for RL (Thompson Sampling)
Learning
Maintain a Dirichlet posterior for 𝑃 𝑠,𝑎 for every 𝑠,𝑎
Start with an uninformative prior e.g., Dirichlet(1,1,…,1)
After round 𝑡, on observing outcome 𝑠 𝑡+1 , update for state 𝑠 𝑡 and action 𝑎 𝑡
To decide action
Sample a 𝑃 𝑠,𝑎 for every 𝑠,𝑎
Compute the optimal policy 𝝅 for sample MDP (𝑆,𝐴, 𝑃 , 𝑟, 𝑠 0 )
Choose 𝑎 𝑡 = 𝜋 𝑠 𝑡
Exploration-exploitation
Exploitation: With more observations Dirichlet posterior concentrates, 𝑃 𝑠,𝑎 approaches empirical mean 𝑃 𝑠,𝑎
Exploration: Anti-concentration of Dirichlet ensures exploration for states/actions/policies less explored
2/4/2019

42. Main result [A., Jia NIPS 2017]
An algorithm based on posterior sampling with high probability near- optimal worst-case regret upper bound
Regret 𝑀, 𝑇 =𝑇 𝜆 ∗ − 𝑡=1 𝑇 𝑟( 𝑠 𝑡 , 𝑎 𝑡 )
𝜆 ∗ is optimal infinite horizon average reward (gain), achieved by best single policy 𝜋 ∗
Theorem: For any communicating MDP M with (unknown) diameter 𝐷, and for T≥ 𝑆 5 𝐴, with high probability:
Regret 𝑀, 𝑇 ≤ 𝑂 (𝐷 𝑆𝐴𝑇 )
Improved state-of-the-art bounds and matched the lower bound
First worst-case/minimax regret analysis of posterior sampling for RL
Analyzed in known prior Bayesian regret setting earlier
[Osband and Van Roy, 2013, 2016, 2017, Abbasi-Yadkori and Szepesvari [2014]
First algorithm to achieve 𝑂 (𝐷 𝑆𝐴𝑇 ) bound in non-episodic setting
Episodic setting 𝑂 ( 𝐻𝑆𝐴𝑇 ), episodes of length 𝐻 [Azar, Osband, Munos, 2017]
2/4/2019

43. Some further directions
Large scale structured MDPs with small parameter space
Scalable learning algorithms for inventory management problems
Learning to price in presence of state-dependent demand
Transition probability depends on time varying context in addition to state
Logistic normal priors

44. Other related work on Thompson Sampling
Known likelihood
Exponential families (with Jeffreys prior) [Korda et al. 2013]
Known prior (Bayesian regret)
Near-optimal regret bounds for any prior [Russo and Van Roy 2013, 2014], [Bubeck and Liu 2013]
Extensions for many variations of MAB
side information, delayed feedback, sleeping bandits, sparse bandits, spectral bandits