1. The Nonstochastic Multiarmed Bandit Problem
Seminar on Experts and Bandits, Fall
Barak Itkin

2. Problem setup 𝐾 slot machines Rewards bounded in 0,1
No experts
Rewards bounded in 0,1
Can also be generalized to other bounds
Partial information
You only learn about the arm you pulled
“No assumptions at all” on slot machines
Not even a fixed distribution
In-fact, can be adversarial

3. Motivation Think of packet routing Rewards are the round-trip time
Multiple routes exist to reach the destination
Rewards are the round-trip time
Will change depending on load in different parts of the network
Partial information
You only learn about the packet you sent
“No assumptions at all” on load behavior
Load can change dramatically over-time

4. Back to the problem Arm assignment is determined in advance
I.e. before the first arm is pulled
Adversarial = Assignment can be picked after strategy is already known
But still, before the game begins
We want to minimize the regret
“How much better could have we done?”
We’ll start with “Weak” Regret
Comparing to always pulling one arm (the “best” arm)

5. Our Goal
We would like to show an algorithm that has the following bound on the “weak” regret:
𝑂 𝐾 𝐺 max ln 𝐾
𝐺 max is the return of the best arm
This should work for any setup of arms
Including random/adversarial setups

6. Difference from last week?
In the second half we saw:
𝐾 experts, each loses a certain fraction of what they get
We can split between multiple experts
We always learn about all the experts
Regret compare to single best expert (“weak” regret)
Where’s the difference?
Solo investment – we chose only one expert
Partial information – we learn only about chosen expert
Bounds:
Last week: ln 𝐾 + 2𝑇 ln 𝐾
Ours: 𝑂 𝐾 𝐺 max ln 𝐾

7. Notations 𝑡 – The time step. Also known as “trial”
𝑡∈ 1,2,…,𝑇
𝑥 𝑖 𝑡 – The reward of arm 𝑖 at trial 𝑡
𝑥 𝑖 𝑡 ∈ 0,1
𝐴 – an algorithm, choosing the arm 𝑖 𝑡 at trial 𝑡
The input at time 𝑡 is the arms chosen and their rewards till now
Formally ,2,…,𝐾 × 0,1 𝑡−1

8. Notations++ 𝐺 𝐴 𝑇 – The return of algorithm 𝐴 at time horizon 𝑇
𝐺 𝐴 𝑇 = 𝑡=1 𝑇 𝑥 𝑖 𝑡 𝑡
Will be abbreviated as 𝐺 𝐴 where the 𝑇 is obvious
𝐺 max 𝑇 – The best single-arm reward at time horizon 𝑇
𝐺 max 𝑇 = max 𝑗 𝑡=1 𝑇 𝑥 𝑗 𝑡
Will also be abbreviated as 𝐺 max

9. Recalling our baseline
The naïve approach

10. Simplistic approach Explore Exploit Profit? Check each arm 𝐵 times
Continue pulling only the best arm
Profit?
Only with “fixed distribution” (no significant changes over time)
May fail with arbitrary/adversary rewards

11. Multiplicative Weights approach
Maintain weights 𝑤 𝑖 𝑡 for arm 𝑖 at trial 𝑡
Start with uniform weights
Use the weights to define a distribution 𝑝 𝑖 𝑡
Typically 𝑝 𝑖 𝑡 = 𝑤 𝑖 𝑡 / 𝑗=1 𝐾 𝑤 𝑗 𝑡
Pick an arm by sampling the distribution
Update weights based on rewards of each action
When rewards are known, multiply by a function 𝑢 of the reward: 𝑤 𝑖 𝑡+1 ← 𝑤 𝑖 𝑡 ⋅𝑢 𝑥 𝑖 𝑡
We’ll discuss partial information setup today

12. First attempt
The Exp3 algorithm

13. Exp3 – Exploration Start with uniform weights
No surprise here
Always encourage the algorithm to explore more arms
Add an “exploration factor” 𝛾 to the probability
Each arm will always have a probability of at least 𝛾/𝐾
The exploration factor is controlled by 𝛾∈ 0,1
Can be fine-tuned later
The exploration factor does not change over time
Rewards may be arbitrary or even adversarial
So, we must always continue exploring

14. Estimation rational Question story time: Answer: More generally:
You randomly visit a shop on 20% of the days
Only on those days you know their profit
However, you need to give a profit estimation for every day
What do you do?
Answer:
On days you visit, estimate 5⋅ profit of that day
On other days, estimate 0
More generally:
observation / chance to observe if seen
0 otherwise

15. Exp3 – Weight updates
To update the weights, use “estimated rewards” instead of the actual rewards
𝑥 𝑖 𝑡 = 𝑥 𝑖 𝑡 / 𝑝 𝑖 𝑡
For actions that weren’t chosen, consider as if 𝒙 𝒊 𝒕 =𝟎
This will create an unbiased estimator:
𝐸 𝑥 𝑖 𝑡 𝑖 1 , 𝑖 2 ,…, 𝑖 𝑡−1 = 𝑝 𝑖 𝑡 ⋅ 𝑥 𝑖 𝑡 / 𝑝 𝑖 𝑡 + 1− 𝑝 𝑖 𝑡 ⋅0 = 𝑥 𝑖 𝑡
This equality holds for all actions – not just the one chosen
This helps with the weight updates on partial information

16. Exp3 Initialize: Set 𝑤 𝑖 1 ←1for all actions 𝑖 For each 𝒕:
Sample 𝑖 𝑡 from
𝑝 𝑖 𝑡 ≝ 1−𝛾 ⋅ 𝑤 𝑖 𝑡 𝑗=1 𝐾 𝑤 𝑗 𝑡 + 𝛾 𝐾
Receive reward 𝑥 𝑖 𝑡 𝑡
Update the weights with
𝑤 𝑖 𝑡+1 ← 𝑤 𝑖 𝑡 ⋅ exp 𝛾 𝑥 𝑖 𝑡 𝐾
𝑥 𝑖 𝑡 =0 for all but the selected action
Practically, we only update 𝑤 𝑖 𝑡

17. Bounds
Since we have a probabilistic argument, we care about the expected “weak” regret:
𝐸 𝐺 max − 𝐺 Exp3 = 𝐺 max −𝐸 𝐺 Exp3
Theorem:
𝐺 max −𝐸 𝐺 Exp3 ≤ 𝑒−1 𝐺 max ⋅𝛾+ 𝐾 ln 𝐾 𝛾
No, this is not what we wanted ( 𝐾 𝐺 max ln 𝐾 )
We’ll fix that later

18. Bound intuition 𝐺 max −𝐸 𝐺 Exp3 ≤ 𝑒−1 𝐺 max ⋅𝛾+ 𝐾 ln 𝐾 𝛾
𝛾 is the exploration factor
As 𝛾→1, we stop exploiting our weights
Instead we prefer “randomly” looking around
Thus we come closer and closer to 𝐺 max
As 𝛾→0, we exploit our weights too much
We check for less changes in other arms
Becomes worse when we have more arms (𝐾)
We must find the “sweet spot”

19. The sweet spot 𝐺 max −𝐸 𝐺 Exp3 ≤ 𝑒−1 𝐺 max ⋅𝛾+ 𝐾 ln 𝐾 𝛾
For some 𝑔≥ 𝐺 max , consider
𝛾= min 1, 𝐾 ln 𝐾 𝑒−1 𝑔 𝛾∈ 0,1
This yields
…≤2 𝐾 ln 𝐾 𝑒−1 𝑔 =2 𝑒−1 𝑔𝐾 ln 𝐾 =2.63 𝑔𝐾 ln 𝐾
Which is similar to the bound we wanted (𝑂 𝐾 𝐺 max ln 𝐾 )

20. Proof 𝐺 max −𝐸 𝐺 Exp3 ≤ 𝑒−1 𝐺 max ⋅𝛾+ 𝐾 ln 𝐾 𝛾
Enough talking, let’s start the proof!
We assume 0<𝛾<1
The bound trivially holds for 𝛾=1 (as 𝑒−1≥1)
Furthermore, we’ll want to rely on some facts
As outlined in the next slides

21. Fact #1 Irrelevant to our cause
We just want numbering (starting at 2) to be consistent with the paper 

22. Fact #2 𝒙 𝒊 𝒕 ≤𝑲/𝜸 Proof: 𝑥 𝑖 𝑡 ≝ 𝑥 𝑖 𝑡 / 𝑝 𝑖 𝑡
𝒙 𝒊 𝒕 ≤𝑲/𝜸
Proof:
𝑥 𝑖 𝑡 ≝ 𝑥 𝑖 𝑡 / 𝑝 𝑖 𝑡
Since 𝑥 𝑖 𝑡 ≤1, we get:
≤1/ 𝑝 𝑖 𝑡
Since 𝑝 𝑖 𝑡 =…+𝛾/𝐾, we get:
≤𝐾/𝛾

23. Fact #3
𝒊=𝟏 𝑲 𝒑 𝒊 𝒕 𝒙 𝒊 𝒕 = 𝒙 𝒊 𝒕 𝒕
Since 𝑥 𝑖 𝑡 ≝ 𝑥 𝑖 𝑡 / 𝑝 𝑖 𝑡 , the sum becomes:
𝑖=1 𝐾 𝑥 𝑖 𝑡
We defined 𝑥 𝑖 𝑡 =0 for all unobserved actions
And so we remain only with 𝑖 𝑡

24. Fact #4 𝒊=𝟏 𝑲 𝒑 𝒊 𝒕 𝒙 𝒊 𝒕 𝟐 ≤ 𝒊=𝟏 𝑲 𝒙 𝒊 𝒕
𝒊=𝟏 𝑲 𝒑 𝒊 𝒕 𝒙 𝒊 𝒕 𝟐 ≤ 𝒊=𝟏 𝑲 𝒙 𝒊 𝒕
From the previous slide (fact #3) we get
= 𝑥 𝑖 𝑡 𝑡 ⋅ 𝑥 𝑖 𝑡 𝑡
Since 𝑥 𝑖 𝑡 ≤1 we get
≤ 𝑥 𝑖 𝑡 𝑡
As before, since un-observed actions cancel out we can do:
= 𝑖=1 𝐾 𝑥 𝑖 𝑡

25. Proof - outline Denote 𝑊 𝑡 ≝ 𝑖=1 𝐾 𝑤 𝑖 𝑡
Establish a relation between 𝑊 𝑡+1 / 𝑊 𝑡 and 𝑥 𝑖 𝑡 𝑡
Establish a relation between ln 𝑊 𝑡+1 / 𝑊 𝑡 and 𝑥 𝑖 𝑡 𝑡
Obtain with 1+𝑥≤ 𝑒 𝑥
Establish a relation between ln 𝑊 𝑇+1 / 𝑊 1 and 𝐺 Exp3
Obtain by summing over 𝒕
Compute a direct bound over ln 𝑊 𝑇+1 / 𝑊 1
Apply bound to 𝐺 Exp3

26. Recall the weight update definition
Proof
We begin with the definition of the weight sums:
𝑊 𝑡+1 𝑊 𝑡 = 𝑖=1 𝐾 𝑤 𝑖 𝑡+1 𝑊 𝑡
Recall the weight update definition
𝑤 𝑗 𝑡+1 ← 𝑤 𝑗 𝑡 ⋅ exp 𝛾 𝑥 𝑗 𝑡 𝐾
= 𝑖=1 𝐾 𝑤 𝑖 𝑡 𝑊 𝑡 exp 𝛾 𝐾 𝑥 𝑖 𝑡

27. Proof
𝑊 𝑡+1 𝑊 𝑡 =…= 𝑖=1 𝐾 𝑤 𝑖 𝑡 𝑊 𝑡 exp 𝛾 𝐾 𝑥 𝑖 𝑡 Recall the probability definition: 𝑝 𝑖 𝑡 ≝ 1−𝛾 ⋅ 𝑤 𝑖 𝑡 𝑊 𝑡 + 𝛾 𝐾 Some algebra and we get: 𝑝 𝑖 𝑡 − 𝛾 𝐾 1−𝛾 = 𝑤 𝑖 𝑡 𝑊 𝑡 = 𝑖=1 𝐾 𝑝 𝑖 𝑡 − 𝛾 𝐾 1−𝛾 exp 𝛾 𝐾 𝑥 𝑖 𝑡

28. Proof
𝑊 𝑡+1 𝑊 𝑡 =…= 𝑖=1 𝐾 𝑝 𝑖 𝑡 − 𝛾 𝐾 1−𝛾 exp 𝛾 𝐾 𝑥 𝑖 𝑡 Note the following inequality for 𝒙≤𝟏 𝑒 𝑥 ≤1+𝑥+ 𝑒−2 𝑥 2 (No, it’s not a famous one. Yes it works, I tested it) ≤ 𝑖=1 𝐾 𝑝 𝑖 𝑡 − 𝛾 𝐾 1−𝛾 1+ 𝛾 𝐾 𝑥 𝑖 𝑡 + 𝑒−2 𝛾 𝐾 𝑥 𝑖 𝑡 2

29. Proof
𝑊 𝑡+1 𝑊 𝑡 =…≤ 𝑖=1 𝐾 𝑝 𝑖 𝑡 − 𝛾 𝐾 1−𝛾 1+ 𝛾 𝐾 𝑥 𝑖 𝑡 + 𝑒−2 𝛾 𝐾 𝑥 𝑖 𝑡 2
Opening the square brackets, we get (just ugly math)
𝑖=1 𝐾 𝑝 𝑖 𝑡 − 𝛾 𝐾 1−𝛾 ⋅1 = 𝑖=1 𝐾 1−𝛾 ⋅ 𝑤 𝑖 𝑡 𝑊 𝑡 1−𝛾 = 𝑖=1 𝐾 𝑤 𝑖 𝑡 𝑊 𝑡 =1
𝑖=1 𝐾 𝑝 𝑖 𝑡 − 𝛾 𝐾 1−𝛾 ⋅ 𝛾 𝐾 𝑥 𝑖 𝑡 = 𝛾 𝐾 1−𝛾 𝑖=1 𝐾 𝑝 𝑖 𝑡 𝑥 𝑖 𝑡 − 𝛾 𝐾 𝑥 𝑖 𝑡 ≤ 𝛾 𝐾 1−𝛾 𝑖=1 𝐾 𝑝 𝑖 𝑡 𝑥 𝑖 𝑡
𝑖=1 𝐾 𝑝 𝑖 𝑡 − 𝛾 𝐾 1−𝛾 ⋅ 𝑒−2 𝛾 𝐾 𝑥 𝑖 𝑡 2 ≤ 𝑒−2 𝛾 𝐾 −𝛾 𝑖=1 𝐾 𝑝 𝑖 𝑡 𝑥 𝑖 𝑡 2

30. Proof
𝑊 𝑡+1 𝑊 𝑡 =…≤ 𝑖=1 𝐾 𝑝 𝑖 𝑡 − 𝛾 𝐾 1−𝛾 1+ 𝛾 𝐾 𝑥 𝑖 𝑡 + 𝑒−2 𝛾 𝐾 𝑥 𝑖 𝑡 2
Combining it all, we have
≤1+ 𝛾 𝐾 1−𝛾 𝑖=1 𝐾 𝑝 𝑖 𝑡 𝑥 𝑖 𝑡 + 𝑒−2 𝛾 𝐾 −𝛾 𝑖=1 𝐾 𝑝 𝑖 𝑡 𝑥 𝑖 𝑡 2
Using fact #3 ( 𝑖=1 𝐾 𝑝 𝑖 𝑡 𝑥 𝑖 𝑡 = 𝑥 𝑖 𝑡 𝑡 ) and Using fact #4 ( 𝑖=1 𝐾 𝑝 𝑖 𝑡 𝑥 𝑖 𝑡 2 ≤ 𝑖=1 𝐾 𝑥 𝑖 𝑡 )
≤1+ 𝛾 𝐾 1−𝛾 𝑥 𝑖 𝑡 𝑡 + 𝑒−2 𝛾 𝐾 −𝛾 𝑖=1 𝐾 𝑥 𝑖 𝑡

31. Proof 𝑊 𝑡+1 𝑊 𝑡 =…≤1+ 𝛾 𝐾 1−𝛾 𝑥 𝑖 𝑡 𝑡 + 𝑒−2 𝛾 𝐾 2 1−𝛾 𝑖=1 𝐾 𝑥 𝑖 𝑡
𝑊 𝑡+1 𝑊 𝑡 =…≤1+ 𝛾 𝐾 1−𝛾 𝑥 𝑖 𝑡 𝑡 + 𝑒−2 𝛾 𝐾 −𝛾 𝑖=1 𝐾 𝑥 𝑖 𝑡
Taking the log and using 1+𝑥≤ 𝑒 𝑥 (specifically ln 1+𝑥 ≤𝑥):
ln 𝑊 𝑡+1 𝑊 𝑡 ≤ 𝛾 𝐾 1−𝛾 𝑥 𝑖 𝑡 𝑡 + 𝑒−2 𝛾 𝐾 −𝛾 𝑖=1 𝐾 𝑥 𝑖 𝑡
Summing over 𝒕 from 𝟏 to 𝑻 we get:
ln 𝑊 𝑇+1 𝑊 1 ≤ 𝛾 𝐾 1−𝛾 𝑡=1 𝑇 𝑥 𝑖 𝑡 𝑡 + 𝑒−2 𝛾 𝐾 −𝛾 𝑡=1 𝑇 𝑖=1 𝐾 𝑥 𝑖 𝑡

32. Proof
ln 𝑊 𝑇+1 𝑊 1 ≤ 𝛾 𝐾 1−𝛾 𝑡=1 𝑇 𝑥 𝑖 𝑡 𝑡 + 𝑒−2 𝛾 𝐾 2 1−𝛾 𝑡=1 𝑇 𝑖=1 𝐾 𝑥 𝑖 𝑡 ln 𝑊 𝑇+1 𝑊 1 ≤ 𝛾 𝐾 1−𝛾 𝐺 Exp3 + 𝑒−2 𝛾 𝐾 2 1−𝛾 𝑡=1 𝑇 𝑖=1 𝐾 𝑥 𝑖 𝑡 ∗ 1−𝛾 𝛾 𝐾 ln 𝑊 𝑇+1 𝑊 1 − 𝑒−2 𝛾 𝐾 𝑡=1 𝑇 𝑖=1 𝐾 𝑥 𝑖 𝑡 ≤ 𝐺 Exp3 We’ll get back to this equation in a second

33. Proof Choosing some action 𝑗 and since 𝑤 𝑗 𝑇+1 ≤ 𝑊 𝑇+1 , note that:
ln 𝑊 𝑇+1 𝑊 1 ≥ ln 𝑤 𝑗 𝑇+1 𝑊 1 = ln 𝑤 𝑗 𝑇+1 − ln 𝑊 1
Recall that 𝑤 𝑗 𝑡+1 ← 𝑤 𝑗 𝑡 ⋅ exp 𝛾 𝑥 𝑗 𝑡 𝐾 and so we can expand the log recursively:
ln 𝑤 𝑗 𝑇+1 = 𝛾 𝐾 ⋅ 𝑥 𝑗 𝑡 + ln 𝑤 𝑗 𝑇 =…= 𝛾 𝐾 𝑡=1 𝑇 𝑥 𝑗 𝑡 + ln 𝑤 𝑗 1
Recall that 𝑤 𝑗 1 =1 and 𝑊 1 =𝐾 (uniform initialization) and so:
$ ln 𝑊 𝑇+1 𝑊 1 ≥ 𝛾 𝐾 𝑡=1 𝑇 𝑥 𝑗 𝑡 − ln 𝐾

34. Proof ∗ 1−𝛾 𝛾 𝐾 ln 𝑊 𝑇+1 𝑊 1 − 𝑒−2 𝛾 𝐾 𝑡=1 𝑇 𝑖=1 𝐾 𝑥 𝑖 𝑡 ≤ 𝐺 Exp3
$ ln 𝑊 𝑇+1 𝑊 1 ≥ 𝛾 𝐾 𝑡=1 𝑇 𝑥 𝑗 𝑡 − ln 𝐾
Combining ∗ and $ we get:
1−𝛾 𝛾 𝐾 ⋅ 𝛾 𝐾 𝑡=1 𝑇 𝑥 𝑗 𝑡 − 1−𝛾 𝛾 𝐾 ⋅ ln 𝐾 − 𝑒−2 𝛾 𝐾 𝑡=1 𝑇 𝑖=1 𝐾 𝑥 𝑖 𝑡 ≤ 𝐺 Exp3
1−𝛾 𝑡=1 𝑇 𝑥 𝑗 𝑡 − 𝐾 𝛾 ln 𝐾 − 𝑒−2 𝛾 𝐾 𝑡=1 𝑇 𝑖=1 𝐾 𝑥 𝑖 𝑡 ≤ 𝐺 Exp3

35. Proof 1−𝛾 𝑡=1 𝑇 𝑥 𝑗 𝑡 − 𝐾 𝛾 ln 𝐾 − 𝑒−2 𝛾 𝐾 𝑡=1 𝑇 𝑖=1 𝐾 𝑥 𝑖 𝑡 ≤ 𝐺 Exp3
Recall we saw that 𝐸 𝑥 𝑖 𝑡 𝑖 1 , 𝑖 2 ,…, 𝑖 𝑡−1 = 𝑥 𝑖 𝑡 . Taking expectation on both sides we get:
1−𝛾 𝑡=1 𝑇 𝑥 𝑗 𝑡 − 𝐾 ln 𝐾 𝛾 − 𝑒−2 𝛾 𝐾 𝑖=1 𝐾 𝑡=1 𝑇 𝑥 𝑖 𝑡 ≤𝐸 𝐺 Exp3
Since 𝑡=1 𝑇 𝑥 𝑐 𝑖 ≤ 𝐺 max for any 𝑐, and since we have a minus sign, we can do:
1−𝛾 𝑡=1 𝑇 𝑥 𝑗 𝑡 − 𝐾 ln 𝐾 𝛾 − 𝑒−2 𝛾 𝐾 𝐾 𝐺 max ≤𝐸 𝐺 Exp3

36. Proof 1−𝛾 𝑡=1 𝑇 𝑥 𝑗 𝑡 − 𝐾 ln 𝐾 𝛾 − 𝑒−2 𝛾 𝐺 max ≤𝐸 𝐺 Exp3
Note this was true for any action 𝑗, which means including the maximal action!
1−𝛾 𝐺 max − 𝐾 ln 𝐾 𝛾 − 𝑒−2 𝛾 𝐺 max ≤𝐸 𝐺 Exp3
𝐺 max −𝐸 𝐺 Exp3 ≤ 𝑒−1 𝛾 𝐺 max + 𝐾 ln 𝐾 𝛾
Q.E.D.(!)

37. Break(?)

38. Is it enough? Maybe? What was our problem?
If it was a clear “yes”, I wouldn’t have asked it :P
What was our problem?

39. Room for improvement We assumed an upper limit 𝒈≥ 𝑮 max is known
We then selected
𝛾= min 1, 𝐾 ln 𝐾 𝑒−1 𝑔
This obtained the bound
𝐺 max −𝐸 𝐺 Exp3 ≤2.63 𝑔𝐾 ln 𝐾
If we don’t know 𝑮 max , we may over-shoot 𝒈
For example, setting 𝑔≔𝑇
This would yield a less tight result :/

40. New goal Maintain the same bound over the weak regret 𝑂 𝐾 𝐺 max ln 𝐾
Don’t assume a known 𝒈≥ 𝑮 max
In fact, do better!
Maintain this bound uniformly throughout the execution*
*A formal definition will follow

41. Notations# (better than ++++)
𝐺 𝑖 𝑡+1 – The return of action 𝑖 till (exclusive) time horizon 𝑡
𝐺 𝑖 𝑡+1 = 𝑠=1 𝑡 𝑥 𝑖 𝑠
𝐺 𝑖 𝑡+1 – The estimated return of action 𝑖 till time horizon 𝑡
𝐺 𝑖 𝑡+1 = 𝑠=1 𝑡 𝑥 𝑖 𝑠
𝐺 max 𝑡+1 – The maximal return of any single action till time horizon 𝑡
𝐺 max 𝑡+1 = max 𝑖 𝐺 𝑖 𝑡+1

42. Re-stating our goal
At every time-step 1≤𝑡≤𝑇, the weak regret till that point should maintain the bound
𝑂 𝐾 𝐺 max 𝑡+1 ln 𝐾
This bound will hold all the way!
This also gives a hint on how to do this:
Previously we guessed 𝑔≥ 𝐺 max
Instead, lets maintain 𝑔≥ 𝐺 max 𝑡+1
Update 𝑔 and 𝛾 as 𝐺 max 𝑡 grows
Effectively – we are searching for the right 𝜸!

43. Exp3.1 (creative names!) Initialize: Set 𝑡←1
For each “epoch” 𝒓=𝟎,𝟏,𝟐,… do:
𝑔 𝑟 ≝ 4 𝑟 ⋅ 𝐾 ln 𝐾 𝑒−1
(Re-)Initialize Exp3 with:
𝛾 𝑟 ≝ min 1, 𝐾 ln 𝐾 𝑒−1 𝑔 𝑟 = min 1,1/2 𝑟
While 𝐦𝐚𝐱 𝒊 𝑮 𝒊 𝒕 ≤ 𝒈 𝒓 −𝑲/ 𝜸 𝒓 do:
Do one step with Exp3
(Update our tracking of 𝐺 𝑖 𝑡+1 for all actions)
𝑡←t+1

44. Bounds (Theorem 4.1) Theorem 4.1:
𝐺 max −𝐸 𝐺 Exp3.1 ≤8 𝑒−1 𝐺 max 𝐾 ln 𝐾 +8 𝑒−1 𝐾+2𝐾 ln 𝐾
Proof outline:
Lemma 4.2: Bound the weak regret per “epoch”
By giving a lower bound on the actual reward
Lemma 4.3: Bound the number of epochs for a finite horizon 𝑇
Combine both to obtain the final bound

45. Epoch notations Let 𝑇 be the overall number of steps
Needed only for the proof
We don’t need to know this in advance
Let 𝑅 be the overall number of epochs
Same here
Let 𝑆 𝑟 and 𝑇 𝑟 be the first and last steps (𝑡) of epoch 𝑟
𝑆 1 =1, 𝑇 𝑅 =𝑇
An epoch may be empty – in that case 𝑆 𝑟 = 𝑇 𝑟 +1> 𝑇 𝑟
𝐺 max ≝ 𝐺 max 𝑇+1

46. Lemma 4.2 For any action 𝑗 and for every epoch 𝑟
𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡 ≥ 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 −2 𝑒−1 𝑔 𝑟 𝐾 ln 𝐾
Holds trivially for empty epochs
We’ll prove for 𝑇 𝑟 ≥ 𝑆 𝑟

47. Proof Some many slides back, we proved the following:
1−𝛾 𝑡=1 𝑇 𝑥 𝑗 𝑡 − 𝐾 ln 𝐾 𝛾 − 𝑒−2 𝛾 𝐾 𝑡=1 𝑇 𝑖=1 𝐾 𝑥 𝑖 𝑡 ≤ 𝐺 Exp3
Translating to our current notations, this becomes
1− 𝛾 𝑟 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 − 𝐾 ln 𝐾 𝛾 𝑟 − 𝑒−2 𝛾 𝑟 𝐾 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑖=1 𝐾 𝑥 𝑖 𝑡 ≤ 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡
𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 − 𝛾 𝑟 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 − 𝐾 ln 𝐾 𝛾 𝑟 − 𝑒−2 𝛾 𝑟 𝐾 𝑖=1 𝐾 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 ≤ 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡

48. Proof
𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 − 𝛾 𝑟 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 − 𝐾 ln 𝐾 𝛾 𝑟 − 𝑒−2 𝛾 𝑟 𝐾 𝑖=1 𝐾 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 ≤ 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡
Merging the estimated reward also with all previous epochs:
𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 − 𝛾 𝑟 𝐺 𝑗 𝑇 𝑟 +1 − 𝐾 ln 𝐾 𝛾 𝑟 − 𝑒−2 𝛾 𝑟 𝐾 𝑖=1 𝐾 𝐺 𝑖 𝑇 𝑟 +1 ≤ 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡
From the termination condition, we know that for any action 𝒊, at time 𝒕, in epoch 𝒓:
𝐺 𝑖 𝑡 ≤ 𝑔 𝑟 −𝐾/ 𝛾 𝑟
From fact #2, we know that
𝑥 𝑖 𝑡 ≤𝐾/ 𝛾 𝑟
Together we get that for any action
𝐺 𝑖 𝑡+1 ≤ 𝑔 𝑟
𝐺 𝑖 𝑇 𝑟 +1 ≤ 𝑔 𝑟
𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 − 𝛾 𝑟 𝑔 𝑟 − 𝐾 ln 𝐾 𝛾 𝑟 − 𝑒−2 𝛾 𝑟 𝐾 𝐾 𝑔 𝑟 ≤ 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡

49. Proof
𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 − 𝛾 𝑟 𝑔 𝑟 − 𝐾 ln 𝐾 𝛾 𝑟 − 𝑒−2 𝛾 𝑟 𝐾 𝐾 𝑔 𝑟 ≤ 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡
𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 − 𝐾 ln 𝐾 𝛾 𝑟 − 𝑒−1 𝑔 𝑟 𝛾 𝑟 ≤ 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡
Recall that
𝛾 𝑟 = min 1, 𝐾 ln 𝐾 𝑒−1 𝑔 𝑟
Substitute and Q.E.D

50. Lemma 4.3 Denote 𝑐≝ 𝐾 ln 𝐾 𝑒−1 , 𝑧≝ 2 𝑅−1
So that
𝑔 𝑟 ≝𝑐⋅ 4 𝑟
Then the number of epochs 𝑅 satisfies the inequality
2 𝑅−1 ≝𝑧≤ 𝐾 𝑐 𝐺 max 𝑐
Holds trivially for 𝑅=0
We’ll prove for 𝑅≥1

51. Proof Regardless of the lemma, we know that:
𝐺 max ≝ 𝐺 max 𝑇+1 ≝ 𝐺 max 𝑇 𝑅 +1 ≥ 𝐺 max 𝑇 𝑅−1 +1
𝑇 𝑅−1 is the termination step, so we know the epoch continuation condition was violated
> 𝑔 𝑅−1 − 𝐾 𝛾 𝑅−1
=𝑐⋅ 4 𝑅−1 − 𝐾 min 1,2 − 𝑅−1
≥𝑐⋅ 4 𝑅−1 −𝐾⋅ 2 𝑅−1
=𝑐 𝑧 2 −𝐾𝑧

52. Proof So, we’ve shown that 𝐺 max >𝑐 𝑧 2 −𝐾𝑧
𝑧= 2 𝑅 is a variable, as we are trying to bound 𝑅
This function is a parabola of 𝑧
The minimum is obtained at 𝑧=𝑘/2𝑐
(Reminder: Extremum of 𝑎 𝑥 2 +𝑏𝑥+𝑐 is at −𝑏/2𝑎)
It increases monotonically for 𝑧>𝑘/2𝑐

53. Proof Now, suppose the claim is false
Reversing the original inequality, we get
𝑧> 𝐾 𝑐 𝐺 max 𝑐 > 𝐾 𝑐 𝐺 max 𝑐
𝑧 and the expression in purple are both larger than 𝑘/2𝑐
This is in the increasing part of 𝑐 𝑧 2 −𝐾𝑧:
𝑐 𝑧 2 −𝐾𝑧>𝑐 𝐾 𝑐 𝐺 max 𝑐 2 −𝐾 𝐾 𝑐 𝐺 max 𝑐
> 𝐾 2 𝑐 +2𝐾 𝐺 max 𝑐 + 𝐺 max − 𝐾 2 𝑐 −𝐾 𝐺 max 𝑐
= 𝐺 max +𝐾 𝐺 max 𝑐 > 𝐺 max

54. Proof Assuming the claim is false, we got 𝑐 𝑧 2 −𝐾𝑧> 𝐺 max
But, before we proved that 𝐺 max >𝑐 𝑧 2 −𝐾𝑧
Q.E.D.

55. Theorem 4.1 Our target was:
𝐺 max −𝐸 𝐺 Exp3.1 ≤8 𝑒−1 𝐺 max 𝐾 ln 𝐾 +8 𝑒−1 𝐾+2𝐾 ln 𝐾
Let’s begin

56. Proof
𝐺 Exp3.1 ≝ 𝑡=1 𝑇 𝑥 𝑖 𝑡 𝑡 ≝ 𝑟=0 𝑅 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡 Lemma 4.2 says that for any action 𝑗 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑖 𝑡 𝑡 ≥ 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 −2 𝑒−1 𝑔 𝑟 𝐾 ln 𝐾 Then this should also be for the maximal action ≥ max 𝑗 𝑟=0 𝑅 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 −2 𝑒−1 𝑔 𝑟 𝐾 ln 𝐾

57. Proof
≥ max 𝑗 𝑟=0 𝑅 𝑡= 𝑆 𝑟 𝑇 𝑟 𝑥 𝑗 𝑡 −2 𝑒−1 𝑔 𝑟 𝐾 ln 𝐾 = max 𝑗 𝐺 𝑗 𝑇+1 − 𝑟=0 𝑅 2 𝑒−1 4 𝑟 ⋅ 𝐾 ln 𝐾 𝑒−1 𝐾 ln 𝐾 = max 𝑗 𝐺 𝑗 𝑇+1 −2𝐾 ln 𝐾 ⋅ 𝑟=0 𝑅 2 𝑟 = 𝐺 max −2𝐾 ln 𝐾 2 𝑅+1 −1

58. Proof
= 𝐺 max −2𝐾 ln 𝐾 2 𝑅+1 −1 Using lemma 4.3 we obtain ≥ 𝐺 max −2𝐾 ln 𝐾 4⋅ 𝑒−1 ln 𝐾 + 𝐺 max ⋅ 𝑒−1 𝐾 ln 𝐾 −1 = 𝐺 max −8𝐾 𝑒−1 −8 𝐺 max ⋅ 𝑒−1 ⋅𝐾 ln 𝐾 −4𝐾 ln 𝐾 +2𝐾 ln 𝐾 = 𝐺 max −2𝐾 ln 𝐾 −8𝐾 𝑒−1 −8 𝑒−1 𝐺 max 𝐾 ln 𝐾

59. Finishing the proof
𝐺 Exp3.1 ≥ 𝐺 max −2𝐾 ln 𝐾 −8𝐾 𝑒−1 −8 𝑒−1 𝐺 max 𝐾 ln 𝐾
𝐺 max − 𝐺 Exp3.1 ≤2𝐾 ln 𝐾 +8𝐾 𝑒−1 +8 𝑒−1 𝐺 max 𝐾 ln 𝐾
Using Jensen’s inequality (where the variable is 𝐺 max ), we complete the proof and take the expectation
Full details in the paper
We don’t have time at this point anyhow :P

60. Questions?