1. Aviv Rosenberg 10/01/18 Seminar on Experts and Bandits
Online Convex Optimization in the Bandit Setting: Gradient Descent without a Gradient
Aviv Rosenberg
10/01/18
Seminar on Experts and Bandits

2. Online Convex Optimization Problem
Convex set 𝑆
In every iteration we choose 𝑥 𝑡 ∈𝑆 and then get a convex cost function 𝑐 𝑡 :𝑆→ −𝐶,𝐶 for some 𝐶>0
We want to minimize the regret 𝑡=1 𝑇 𝑐 𝑡 𝑥 𝑡 − min 𝑥∈𝑆 𝑡=1 𝑇 𝑐 𝑡 (𝑥)

3. Bandit Setting
The Gradient Descent approach: 𝑥 𝑡+1 = 𝑥 𝑡 −𝜂𝛻 𝑐 𝑡 ( 𝑥 𝑡 )
Last week we saw 𝑂( 𝑇 ) regret bound
But now instead of 𝑐 𝑡 we get 𝑐 𝑡 ( 𝑥 𝑡 )
Can’t compute 𝛻 𝑐 𝑡 ( 𝑥 𝑡 )!
But we still want to use Gradient Descent
Solution: estimate gradient using one point
We will show 𝑂( 𝑇 3/4 ) regret bound

4. Notation and Assumptions
𝔹= 𝑥∈ ℝ 𝑑 : 𝑥 ≤1 ; 𝕊= 𝑥∈ ℝ 𝑑 : 𝑥 =1
Expected regret
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑥 𝑡 ) − min 𝑥∈𝑆 𝑡=1 𝑇 𝑐 𝑡 (𝑥)
Projection of point 𝑥 onto convex set 𝑆
𝑃 𝑆 𝑥 = arg min 𝑧∈𝑆 𝑥−𝑧
Assume 𝑆 is a convex set such that 𝑟𝔹⊆𝑆⊆𝑅𝔹
1−𝛼 𝑆= 1−𝛼 𝑥:𝑥∈𝑆 ⊆𝑆 is also convex and 0∈(1−𝛼)𝑆⊆𝑅𝔹
𝑦∈ 1−𝛼 𝑆 ↓
𝑦= 1−𝛼 𝑥=𝛼0+ 1−𝛼 𝑥∈𝑆

5. Part 1 Gradient Estimation

6. Gradient Estimation For a function 𝑐 𝑡 and 𝛿>0 define
𝑐 𝑡 𝑦 = 𝔼 𝑣∈𝔹 𝑐 𝑡 (𝑦+𝛿𝑣)
Lemma:
𝛻 𝑐 𝑡 𝑦 = 𝑑 𝛿 𝔼 𝑢∈𝕊 𝑐 𝑡 𝑦+𝛿𝑢 𝑢
To get and unbiased estimator for ∇ 𝑐 𝑡 𝑦 we can sample a unit vector 𝑢 uniformly and compute 𝑑 𝛿 𝑐 𝑡 𝑦+𝛿𝑢 𝑢

7. Proof
𝑐 𝑡 𝑦 = 𝔼 𝑣∈𝔹 𝑐 𝑡 (𝑦+𝛿𝑣)
For 𝑑=1
𝑐 𝑡 𝑦 = 𝔼 𝑣∈[−1,1] 𝑐 𝑡 (𝑦+𝛿𝑣) = 𝔼 𝑣∈[−𝛿,𝛿] 𝑐 𝑡 (𝑦+𝑣) = 1 2𝛿 −𝛿 𝛿 𝑐 𝑡 𝑦+𝑣 𝑑𝑣
Differentiate using the fundamental theorem of calculus
𝛻 𝑐 𝑡 𝑦 = 𝑐 𝑡 ′ 𝑦 = 𝑐 𝑡 𝑦+𝛿 − 𝑐 𝑡 𝑦−𝛿 2𝛿 = 1 𝛿 𝔼 𝑢∈{−1,1} 𝑐 𝑡 𝑦+𝛿𝑢 𝑢

8. Proof Cont. For 𝑑>1 Stoke’s theorem gives
𝛻 𝛿𝔹 𝑐 𝑡 𝑦+𝑣 𝑑𝑣 = 𝛿𝕊 𝑐 𝑡 𝑦+𝑢 𝑢 𝑢 𝑑𝑢
Vol 𝑑 𝛿𝔹 𝛻 𝛿𝔹 𝑐 𝑡 𝑦+𝑣 𝑑𝑣 Vol 𝑑 𝛿𝔹 = Vol 𝑑−1 𝛿𝕊 𝛿𝕊 𝑐 𝑡 𝑦+𝑢 𝑢 𝑢 𝑑𝑢 Vol 𝑑−1 𝛿𝕊
Vol 𝑑 𝛿𝔹 𝛻 𝔼 𝑣∈𝛿𝔹 𝑐 𝑡 (𝑦+𝑣) = Vol 𝑑−1 𝛿𝕊 𝔼 𝑢∈δ𝕊 𝑐 𝑡 𝑦+𝑢 𝑢 𝑢
Vol 𝑑 𝛿𝔹 𝛻 𝔼 𝑣∈𝔹 𝑐 𝑡 (𝑦+𝛿𝑣) = Vol 𝑑−1 𝛿𝕊 𝔼 𝑢∈𝕊 𝑐 𝑡 𝑦+𝛿𝑢 𝑢

9. Proof Cont. Vol 𝑑 𝛿𝔹 𝛻 𝔼 𝑣∈𝔹 𝑐 𝑡 (𝑦+𝛿𝑣) = Vol 𝑑−1 𝛿𝕊 𝔼 𝑢∈𝕊 𝑐 𝑡 𝑦+𝛿𝑢 𝑢
𝑐 𝑡 𝑦 = 𝔼 𝑣∈𝔹 𝑐 𝑡 (𝑦+𝛿𝑣)
Vol 𝑑 𝛿𝔹 𝛻 𝔼 𝑣∈𝔹 𝑐 𝑡 (𝑦+𝛿𝑣) = Vol 𝑑−1 𝛿𝕊 𝔼 𝑢∈𝕊 𝑐 𝑡 𝑦+𝛿𝑢 𝑢
Vol 𝑑 𝛿𝔹 𝛻 𝑐 𝑡 𝑦 = Vol 𝑑−1 𝛿𝕊 𝔼 𝑢∈𝕊 𝑐 𝑡 𝑦+𝛿𝑢 𝑢
𝛻 𝑐 𝑡 𝑦 = Vol 𝑑−1 𝛿𝕊 Vol 𝑑 𝛿𝔹 𝔼 𝑢∈𝕊 𝑐 𝑡 𝑦+𝛿𝑢 𝑢
The following fact concludes the proof
Vol 𝑑−1 𝛿𝕊 Vol 𝑑 𝛿𝔹 = 𝑑 𝛿
For example in ℝ 2 :
2𝜋𝛿 𝜋 𝛿 2 = 2 𝛿

10. Part 2 Regret Bound for Estimated Gradients

11. Zinkevich’s Theorem
Let ℎ 1 ,…, ℎ 𝑇 :(1−𝛼)𝑆→ℝ be convex, differentiable functions
Let 𝑦 1 ,…, 𝑦 𝑇 ∈(1−𝛼)𝑆 be defined by 𝑦 1 =0 and 𝑦 𝑡+1 = 𝑃 (1−𝛼)𝑆 ( 𝑦 𝑡 −𝜂𝛻 ℎ 𝑡 𝑦 𝑡 )
Let 𝐺= max 𝑡 𝛻 ℎ 𝑡 𝑦 𝑡
Then for 𝜂= 𝑅 𝐺 𝑇 and for every y∈(1−𝛼)𝑆
𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 ℎ 𝑡 (𝑦) ≤𝑅𝐺 𝑇

12. Expected Zinkevich’s Theorem
Let 𝑐 1 ,…, 𝑐 𝑇 :(1−𝛼)𝑆→ℝ be convex, differentiable functions
Let 𝑔 1 ,…, 𝑔 𝑇 be random vectors such that
𝔼 𝑔 𝑡 𝑦 𝑡 =𝛻 𝑐 𝑡 𝑦 𝑡
𝑔 𝑡 ≤𝐺 (also implies ∇ 𝑐 𝑡 𝑦 𝑡 ≤𝐺)
Let 𝑦 1 ,…, 𝑦 𝑇 ∈(1−𝛼)𝑆 be defined by 𝑦 1 =0 and 𝑦 𝑡+1 = 𝑃 (1−𝛼)𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
Then for 𝜂= 𝑅 𝐺 𝑇 and for every y∈(1−𝛼)𝑆
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑦 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑦 ≤𝑅𝐺 𝑇

13. Proof
𝑦 𝑡+1 = 𝑃 (1−𝛼)𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
𝑦 1 =0
Define ℎ 𝑡 :(1−𝛼)𝑆→ℝ by ℎ 𝑡 𝑦 = 𝑐 𝑡 𝑦 + 𝑦 𝑇 𝜉 𝑡 where 𝜉 𝑡 = 𝑔 𝑡 −𝛻 𝑐 𝑡 𝑦 𝑡
Notice that
𝛻 ℎ 𝑡 𝑦 𝑡 =𝛻 𝑐 𝑡 𝑦 𝑡 + 𝜉 𝑡 = 𝑔 𝑡
So our updates are the same as running regular gradient descent on ℎ 𝑡
From Zinkevich’s Theorem
𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 ℎ 𝑡 (𝑦) ≤𝑅𝐺 𝑇 (1)

14. ℎ 𝑡 𝑦 = 𝑐 𝑡 𝑦 + 𝑦 𝑇 𝜉 𝑡
𝜉 𝑡 = 𝑔 𝑡 −𝛻 𝑐 𝑡 𝑦 𝑡
𝔼 𝑔 𝑡 𝑦 𝑡 =𝛻 𝑐 𝑡 𝑦 𝑡
Proof Cont.
Notice that
𝔼 𝜉 𝑡 | 𝑦 𝑡 =𝔼 𝑔 𝑡 −𝛻 𝑐 𝑡 𝑦 𝑡 | 𝑦 𝑡 =𝔼 𝑔 𝑡 | 𝑦 𝑡 −∇ 𝑐 𝑡 𝑦 𝑡 =0
Therefore
𝔼 𝑦 𝑡 𝑇 𝜉 𝑡 =𝔼 𝔼 𝑦 𝑡 𝑇 𝜉 𝑡 | 𝑦 𝑡 =𝔼 𝑦 𝑡 𝑇 𝔼 𝜉 𝑡 | 𝑦 𝑡 =0
𝔼 𝑦 𝑇 𝜉 𝑡 = 𝑦 𝑇 𝔼 𝜉 𝑡 = 𝑦 𝑇 𝔼 𝔼 𝜉 𝑡 | 𝑦 𝑡 =0
We get the following connections
𝔼 ℎ 𝑡 ( 𝑦 𝑡 ) =𝔼 𝑐 𝑡 ( 𝑦 𝑡 ) +𝔼 𝑦 𝑡 𝑇 𝜉 𝑡 =𝔼 𝑐 𝑡 ( 𝑦 𝑡 ) (3)
𝔼 ℎ 𝑡 (𝑦) =𝔼 𝑐 𝑡 (𝑦) +𝔼 𝑦 𝑇 𝜉 𝑡 = 𝑐 𝑡 𝑦 (2)

15. Proof Cont. 1 𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 ℎ 𝑡 (𝑦) ≤𝑅𝐺 𝑇
𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 ℎ 𝑡 (𝑦) ≤𝑅𝐺 𝑇
𝔼 ℎ 𝑡 (𝑦) = 𝑐 𝑡 (𝑦)
𝔼 ℎ 𝑡 ( 𝑦 𝑡 ) =𝔼 𝑐 𝑡 ( 𝑦 𝑡 )
Proof Cont.
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑦 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑦 = (3) 𝑡=1 𝑇 𝔼 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 𝑐 𝑡 𝑦 = (2) 𝔼 𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 𝔼 ℎ 𝑡 𝑦 = 𝔼 𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 ℎ 𝑡 𝑦 ≤ (1) 𝑅𝐺 𝑇

16. Part 3 BGD Algorithm

17. Ideal World Algorithm 𝑦 1 ←0 For 𝑡∈{1,…,𝑇}:
𝑔 𝑡 = 𝑑 𝛿 𝑐 𝑡 𝑦 𝑡 +𝛿 𝑢 𝑡 𝑢 𝑡
𝔼 𝑔 𝑡 𝑦 𝑡 =𝛻 𝑐 𝑡 𝑦 𝑡
Ideal World Algorithm
𝑦 1 ←0
For 𝑡∈{1,…,𝑇}:
Select unit vector 𝑢 𝑡 uniformly at random
Play 𝑦 𝑡 and observe cost 𝑐 𝑡 𝑦 𝑡
Compute 𝑔 𝑡 using 𝑢 𝑡
𝑦 𝑡+1 ← 𝑃 𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
To compute 𝑔 𝑡 we need 𝑐 𝑡 𝑦 𝑡 +𝛿 𝑢 𝑡
So we need to play 𝑥 𝑡 = 𝑦 𝑡 +𝛿 𝑢 𝑡 instead
But now we have problems:
𝑥 𝑡 ∈𝑆??
The regret is for c t ( 𝑥 𝑡 ) although we are doing Estimated Gradient Descent for 𝑐 𝑡 𝑦 𝑡

18. Bandit Gradient Descent Algorithm (BGD)
Parameters: 𝜂>0,𝛿>0,0<𝛼<1
𝑦 1 ←0
For 𝑡∈{1,…,𝑇}:
Select unit vector 𝑢 𝑡 uniformly at random
𝑥 𝑡 ← 𝑦 𝑡 +𝛿 𝑢 𝑡
Play 𝑥 𝑡 and observe cost 𝑐 𝑡 𝑥 𝑡 = 𝑐 𝑡 ( 𝑦 𝑡 +𝛿 𝑢 𝑡 )
𝑔 𝑡 ← 𝑑 𝛿 𝑐 𝑡 𝑥 𝑡 𝑢 𝑡 = 𝑑 𝛿 𝑐 𝑡 𝑦 𝑡 +𝛿 𝑢 𝑡 𝑢 𝑡
𝑦 𝑡+1 ← 𝑃 1−𝛼 𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
𝑦 1 ←0
For 𝑡∈{1,…,𝑇}:
Select random unit vector 𝑢 𝑡
Play 𝑦 𝑡 and observe 𝑐 𝑡 𝑦 𝑡
Compute 𝑔 𝑡 using 𝑢 𝑡
𝑦 𝑡+1 ← 𝑃 𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
𝑥 𝑡 ∈𝑆?
We have low regret for 𝑐 𝑡 ( 𝑦 𝑡 ) in 1−𝛼 𝑆
Need to convert it to low regret for 𝑐 𝑡 𝑥 𝑡 in 𝑆

19. Observation 1 For any 𝑥∈𝑆 𝑡=1 𝑇 𝑐 𝑡 1−𝛼 𝑥 ≤ 𝑡=1 𝑇 𝑐 𝑡 𝑥 +2𝛼𝐶𝑇
𝑡=1 𝑇 𝑐 𝑡 1−𝛼 𝑥 ≤ 𝑡=1 𝑇 𝑐 𝑡 𝑥 +2𝛼𝐶𝑇
Proof. From convexity
𝑐 𝑡 1−𝛼 𝑥 = 𝑐 𝑡 𝛼0+ 1−𝛼 𝑥 ≤
𝛼𝑐 𝑡 −𝛼 𝑐 𝑡 𝑥 = 𝑐 𝑡 𝑥 +𝛼 𝑐 𝑡 0 − 𝑐 𝑡 𝑥 ≤
𝑐 𝑡 𝑥 +2𝛼𝐶

20. Observation 2 For any y∈ 1−𝛼 𝑆 and any x∈𝑆
𝑐 𝑡 𝑥 − 𝑐 𝑡 𝑦 ≤ 2𝐶 𝛼𝑟 | 𝑦−𝑥 |
Proof. Denote Δ=𝑥−𝑦
If Δ ≥𝛼𝑟 the observation follows from 2𝐶≤ 2𝐶 𝛼𝑟 | 𝑦−𝑥 |
Otherwise Δ <𝛼𝑟, let 𝑧=𝑦+𝛼𝑟 Δ | Δ | and 𝑧∈𝑆 from
y∈ 1−𝛼 𝑆
𝛼𝑟 Δ Δ ∈𝛼𝑟𝔹⊆𝛼𝑆 ↓
𝑧∈ 1−𝛼 𝑆+𝛼𝑆⊆𝑆

21. Proof Cont. Notice that x= Δ 𝛼𝑟 𝑧+ 1− Δ 𝛼𝑟 𝑦 So from convexity
𝑐 𝑡 𝑥 = 𝑐 𝑡 Δ 𝛼𝑟 𝑧+ 1− Δ 𝛼𝑟 𝑦 ≤
Δ 𝛼𝑟 𝑐 𝑡 𝑧 + 1− Δ 𝛼𝑟 𝑐 𝑡 𝑦 = 𝑐 𝑡 𝑦 + 𝑐 𝑡 𝑧 − 𝑐 𝑡 (𝑦) 𝛼𝑟 Δ ≤ 𝑐 𝑡 𝑦 + 2𝐶 𝛼𝑟 Δ
Other direction is also true
𝑧=𝑦+𝛼𝑟 Δ Δ
Δ=𝑥−𝑦

22. BGD Regret Theorem
For any 𝑇≥ 3𝑅𝑑 2𝑟 2 and for the following parameters
𝜂= 𝛿𝑅 𝑑𝐶 𝑇 𝛿= 3 𝑟 𝑅 2 𝑑 2 12𝑇 𝛼= 3 3𝑅𝑑 2𝑟 𝑇
For every 𝑥∈𝑆 the BGD achieves regret
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑥 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑥 ≤3𝐶 𝑇 5/6 3 𝑑𝑅 𝑟

23. Proof 𝑔 𝑡 = 𝑑 𝛿 𝑐 𝑡 𝑥 𝑡 𝑢 𝑡 𝑥 𝑡 = 𝑦 𝑡 +𝛿 𝑢 𝑡
𝑔 𝑡 = 𝑑 𝛿 𝑐 𝑡 𝑥 𝑡 𝑢 𝑡
𝑥 𝑡 = 𝑦 𝑡 +𝛿 𝑢 𝑡
𝑦 𝑡+1 = 𝑃 1−𝛼 𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
Proof
First we need to show that 𝑥 𝑡 ∈𝑆
Notice that 1−𝛼 𝑆+𝛼𝑟𝔹⊆ 1−𝛼 𝑆+𝛼𝑆⊆𝑆
Since 𝑦 𝑡 ∈ 1−𝛼 𝑆, we just need to show that 𝛿≤𝛼𝑟
𝛿= 3 𝑟 𝑅 2 𝑑 2 12𝑇
𝛼𝑟= 3 3𝑅𝑑 2𝑟 𝑇 𝑟= 3 3𝑅 𝑟 2 𝑑 2 𝑇
This is true because 𝑇≥ 3𝑅𝑑 2𝑟 2

24. Proof Cont. 𝑐 𝑡 𝑦 𝑡 = 𝔼 𝑣∈𝔹 𝑐 𝑡 ( 𝑦 𝑡 +𝛿𝑣)
𝑐 𝑡 𝑦 𝑡 = 𝔼 𝑣∈𝔹 𝑐 𝑡 ( 𝑦 𝑡 +𝛿𝑣)
𝛻 𝑐 𝑡 𝑦 𝑡 = 𝑑 𝛿 𝔼 𝑢∈𝕊 𝑐 𝑡 𝑦 𝑡 +𝛿𝑢 𝑢
𝑔 𝑡 = 𝑑 𝛿 𝑐 𝑡 𝑥 𝑡 𝑢 𝑡
𝑥 𝑡 = 𝑦 𝑡 +𝛿 𝑢 𝑡
𝑦 𝑡+1 = 𝑃 1−𝛼 𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
Proof Cont.
Now we want to bound the regret
We have 𝔼 𝑔 𝑡 𝑦 𝑡 =𝛻 𝑐 𝑡 𝑦 𝑡
𝑔 𝑡 = 𝑑 𝛿 𝑐 𝑡 𝑥 𝑡 𝑢 𝑡 ≤ 𝑑𝐶 𝛿 =:𝐺
Expected Zinkevich’s Theorem says
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑦 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑦 ≤𝑅𝐺 𝑇 = 𝑅𝑑𝐶 𝑇 𝛿 (1)
Where y∈ 1−𝛼 𝑆 and 𝜂= 𝑅 𝐺 𝑇 = 𝛿𝑅 𝑑𝐶 𝑇

25. Proof Cont. From observation 2 we get
𝑐 𝑡 𝑦 − 𝑐 𝑡 𝑥 ≤ 2𝐶 𝛼𝑟 | 𝑦−𝑥 |
For 𝑦∈ 1−𝛼 𝑆
𝑐 𝑡 𝑦 𝑡 = 𝔼 𝑣∈𝔹 𝑐 𝑡 ( 𝑦 𝑡 +𝛿𝑣)
From observation 2 we get
𝑐 𝑡 𝑦 𝑡 − 𝑐 𝑡 𝑥 𝑡 ≤ 𝑐 𝑡 𝑦 𝑡 − 𝑐 𝑡 𝑦 𝑡 + 𝑐 𝑡 𝑦 𝑡 − 𝑐 𝑡 𝑥 𝑡 ≤2 2𝐶 𝛼𝑟 𝛿
𝑐 𝑡 𝑦 − 𝑐 𝑡 𝑦 ≤ 2𝐶 𝛼𝑟 𝛿
Now we get for 𝑦∈ 1−𝛼 𝑆
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑥 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑦 ≤𝔼 𝑡=1 𝑇 𝑐 𝑡 𝑦 𝑡 +2 2𝐶 𝛼𝑟 𝛿 − 𝑡=1 𝑇 ( 𝑐 𝑡 𝑦 − 2𝐶 𝛼𝑟 𝛿)=

26. 1 𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑦 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑦 ≤ 𝑅𝑑𝐶 𝑇 𝛿
Proof Cont.
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑥 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑦 ≤𝔼 𝑡=1 𝑇 𝑐 𝑡 𝑦 𝑡 +2 2𝐶 𝛼𝑟 𝛿 − 𝑡=1 𝑇 ( 𝑐 𝑡 𝑦 − 2𝐶 𝛼𝑟 𝛿)= 𝔼 𝑡=1 𝑇 𝑐 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 𝑐 𝑡 𝑦 +3𝑇 2𝐶 𝛼𝑟 𝛿≤ 𝑅𝑑𝐶 𝑇 𝛿 +3𝑇 2𝐶 𝛼𝑟 𝛿 (2)

27. Proof Cont. 𝑦= 1−𝛼 𝑥 for some 𝑥∈𝑆 so we can use observation 1
𝑡=1 𝑇 𝑐 𝑡 1−𝛼 𝑥 ≤ 𝑡=1 𝑇 𝑐 𝑡 𝑥 +2𝛼𝐶𝑇
2 𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑥 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑦 ≤ 𝑅𝑑𝐶 𝑇 𝛿 +3𝑇 2𝐶 𝛼𝑟 𝛿
Proof Cont.
𝑦= 1−𝛼 𝑥 for some 𝑥∈𝑆 so we can use observation 1
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑥 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑥 ≤𝔼 𝑡=1 𝑇 𝑐 𝑡 𝑥 𝑡 − 𝑡=1 𝑇 𝑐 𝑡 1−𝛼 𝑥 +2𝛼𝐶𝑇=
𝔼 𝑡=1 𝑇 𝑐 𝑡 𝑥 𝑡 − 𝑡=1 𝑇 𝑐 𝑡 𝑦 +2𝛼𝐶𝑇≤ 𝑅𝑑𝐶 𝑇 𝛿 +𝑇 6𝐶 𝛼𝑟 𝛿+2𝛼𝐶𝑇
Substituting the parameters finishes the proof.
𝛿= 3 𝑟 𝑅 2 𝑑 2 12𝑇 𝛼= 3 3𝑅𝑑 2𝑟 𝑇

28. BGD with Lipschitz Regret Theorem
If all 𝑐 𝑡 are 𝐿−Lipschitz then for 𝑇 sufficiently large and the parameters
𝜂= 𝛿𝑅 𝑑𝐶 𝑇 𝛿= 𝑇 −1/4 𝑅𝑑𝐶𝑟 3(𝐿𝑟+𝐶) 𝛼= 𝛿 𝑟
The BGD achieves regret
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑥 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑥 ≤2 𝑇 3/4 3𝑅𝑑𝐶(𝐿+ 𝐶 𝑟 )

29. Part 4 Reshaping

30. Removing Dependence in 1/𝑟
There are algorithms that for a convex set 𝑟𝔹⊆𝑆⊆𝑅𝔹 find an affine transformation 𝑇 that puts 𝑆 in near-isotropic position and run in time 𝑂 𝑑 4 𝑝𝑜𝑙𝑦𝑙𝑜𝑔(𝑑, 𝑅 𝑟 )
𝑇 𝑆 ⊆ ℝ 𝑑 is in isotropic position if the covariance matrix of a random sample from 𝑇 𝑆 is the identity matrix
This gives us 𝔹⊆𝑇(𝑆)⊆𝑑𝔹
So we have new 𝑅=𝑑 and 𝑟=1
Also if 𝑐 𝑡 is 𝐿− Lipschitz then 𝑐 𝑡 ⃘ 𝑇 −1 is 𝐿𝑅− Lipschitz

31. Removing Dependence in 1/𝑟
So if we first put 𝑆 in near-isotropic position we get the regret bound
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑥 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑥 ≤6 𝑇 𝑑( 𝐶𝐿𝑅 +𝐶)
And without the Lipschitz condition
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑥 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑥 ≤6 𝑇 𝑑𝐶
2 𝑇 3/4 3𝑅𝑑𝐶(𝐿+ 𝐶 𝑟 )
3𝐶 𝑇 5/6 3 𝑑𝑅 𝑟

32. Part 5 Adaptive Adversary

33. Expected Adaptive Zinkevich’s Theorem
Let 𝑐 1 ,…, 𝑐 𝑇 :(1−𝛼)𝑆→ℝ be convex, differentiable functions ( 𝑐 𝑡 depends on 𝑦 1 ,…, 𝑦 𝑡−1 )
Let 𝑔 1 ,…, 𝑔 𝑇 be random vectors such that
𝔼 𝑔 𝑡 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 =𝛻 𝑐 𝑡 𝑦 𝑡
𝑔 𝑡 ≤𝐺 (also implies ∇ 𝑐 𝑡 ( 𝑦 𝑡 ) ≤𝐺)
Let 𝑦 1 ,…, 𝑦 𝑇 ∈(1−𝛼)𝑆 be defined by 𝑦 1 =0 and 𝑦 𝑡+1 = 𝑃 (1−𝛼)𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
Then for 𝜂= 𝑅 𝐺 𝑇 and for every 𝑦∈(1−𝛼)𝑆
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑦 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑦 ≤3𝑅𝐺 𝑇

34. Proof
𝑦 𝑡+1 = 𝑃 (1−𝛼)𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
𝑦 1 =0
Define ℎ 𝑡 :(1−𝛼)𝑆→ℝ by ℎ 𝑡 𝑦 = 𝑐 𝑡 𝑦 + 𝑦 𝑇 𝜉 𝑡 where 𝜉 𝑡 = 𝑔 𝑡 −𝛻 𝑐 𝑡 𝑦 𝑡
Notice that
𝛻 ℎ 𝑡 𝑦 𝑡 =𝛻 𝑐 𝑡 𝑦 𝑡 + 𝜉 𝑡 = 𝑔 𝑡
So our updates are the same as running regular gradient descent on ℎ 𝑡
From Zinkevich’s Theorem
𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 ℎ 𝑡 (𝑦) ≤𝑅𝐺 𝑇 (1)

35. Proof Cont.
ℎ 𝑡 𝑦 = 𝑐 𝑡 𝑦 + 𝑦 𝑇 𝜉 𝑡
𝜉 𝑡 = 𝑔 𝑡 −𝛻 𝑐 𝑡 𝑦 𝑡
𝔼 𝑔 𝑡 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 =𝛻 𝑐 𝑡 𝑦 𝑡
Notice that
𝔼 𝜉 𝑡 | 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 =𝔼 𝑔 𝑡 −𝛻 𝑐 𝑡 𝑦 𝑡 | 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 =
𝔼 𝑔 𝑡 | 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 −𝛻 𝑐 𝑡 𝑦 𝑡 =0
𝔼 𝑦 𝑡 𝑇 𝜉 𝑡 =𝔼 𝔼 𝑦 𝑡 𝑇 𝜉 𝑡 | 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 =𝔼 𝑦 𝑡 𝑇 𝔼 𝜉 𝑡 | 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 =0
We get the following connection between 𝔼 ℎ 𝑡 ( 𝑦 𝑡 ) and 𝔼 𝑐 𝑡 ( 𝑦 𝑡 )
𝔼 ℎ 𝑡 ( 𝑦 𝑡 ) =𝔼 𝑐 𝑡 ( 𝑦 𝑡 ) +𝔼 𝑦 𝑡 𝑇 𝜉 𝑡 =𝔼 𝑐 𝑡 ( 𝑦 𝑡 ) (3)

36. Proof Cont. 𝑦 𝑡+1 = 𝑃 (1−𝛼)𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
𝑦 𝑡+1 = 𝑃 (1−𝛼)𝑆 𝑦 𝑡 −𝜂 𝑔 𝑡
𝔼 𝑔 𝑡 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 =𝛻 𝑐 𝑡 𝑦 𝑡
𝜉 𝑡 = 𝑔 𝑡 −𝛻 𝑐 𝑡 𝑦 𝑡
𝜉 𝑡 ≤ 𝑔 𝑡 + 𝛻 𝑐 𝑡 𝑦 𝑡 ≤2𝐺
For every 1≤𝑠<𝑡≤𝑇 we have 𝔼 𝜉 𝑠 𝑇 𝜉 𝑡 =𝔼 𝔼 𝜉 𝑠 𝑇 𝜉 𝑡 | 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡
Given 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 we know 𝑔 𝑠 (so also 𝜉 𝑠 ) and therefore
𝔼 𝜉 𝑠 𝑇 𝜉 𝑡 =𝔼 𝜉 𝑠 𝑇 𝔼 𝜉 𝑡 | 𝑦 1 , 𝑐 1 ,…,𝑦 𝑡 , 𝑐 𝑡 =0
We use it to get
𝔼 𝑡=1 𝑇 𝜉 𝑡 ≤𝔼 𝑡=1 𝑇 𝜉 𝑡 = 𝑡=1 𝑇 𝔼 𝜉 𝑡 ≤𝑠<𝑡≤𝑇 𝔼 𝜉 𝑠 𝑇 𝜉 𝑡 =
𝑡=1 𝑇 𝔼 𝜉 𝑡 ≤ 𝑡=1 𝑇 𝔼 2𝐺 2 =4𝑇 𝐺 2

37. Proof Cont. Now we connect between 𝔼 ℎ 𝑡 (𝑦) and 𝔼 𝑐 𝑡 (𝑦)
ℎ 𝑡 𝑦 = 𝑐 𝑡 𝑦 + 𝑦 𝑇 𝜉 𝑡
𝔼 𝑡=1 𝑇 𝜉 𝑡 ≤2𝐺 𝑇
𝑆⊆𝑅𝔹
Proof Cont.
Now we connect between 𝔼 ℎ 𝑡 (𝑦) and 𝔼 𝑐 𝑡 (𝑦)
𝔼 𝑡=1 𝑇 ℎ 𝑡 𝑦 −𝔼 𝑡=1 𝑇 𝑐 𝑡 (𝑦) ≤𝔼 𝑡=1 𝑇 ℎ 𝑡 𝑦 − 𝑐 𝑡 (𝑦) =
𝔼 𝑦 𝑇 𝑡=1 𝑇 𝜉 𝑡 ≤𝔼 𝑦 𝑡=1 𝑇 𝜉 𝑡 ≤𝑅𝔼 𝑡=1 𝑇 𝜉 𝑡 ≤2𝑅𝐺 𝑇 (2)

38. Proof Cont. 1 𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 ℎ 𝑡 (𝑦) ≤𝑅𝐺 𝑇
𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 ℎ 𝑡 (𝑦) ≤𝑅𝐺 𝑇
2 𝔼 𝑡=1 𝑇 ℎ 𝑡 𝑦 −𝔼 𝑡=1 𝑇 𝑐 𝑡 𝑦 ≤2𝑅𝐺 𝑇
𝔼 ℎ 𝑡 ( 𝑦 𝑡 ) =𝔼 𝑐 𝑡 ( 𝑦 𝑡 )
Proof Cont.
𝔼 𝑡=1 𝑇 𝑐 𝑡 ( 𝑦 𝑡 ) − 𝑡=1 𝑇 𝑐 𝑡 𝑦 = 𝑡=1 𝑇 𝔼 𝑐 𝑡 𝑦 𝑡 −𝔼 𝑡=1 𝑇 𝑐 𝑡 𝑦 = (3) 𝑡=1 𝑇 𝔼 ℎ 𝑡 𝑦 𝑡 −𝔼 𝑡=1 𝑇 𝑐 𝑡 𝑦 ≤ (2) 𝔼 𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 −𝔼 𝑡=1 𝑇 ℎ 𝑡 𝑦 +2 𝑅𝐺 𝑇 = 𝔼 𝑡=1 𝑇 ℎ 𝑡 𝑦 𝑡 − 𝑡=1 𝑇 ℎ 𝑡 𝑦 +2𝑅𝐺 𝑇 ≤ (1) 3𝑅𝐺 𝑇