1. CSCI B609: “Foundations of Data Science”
Lecture 5: Dimension Reduction,
Separating and Fitting Gaussians
Slides at
Grigory Yaroslavtsev

2. Gaussian Annulus Theorem
Gaussian in 𝒅 dimensions ( 𝑁 𝒅 ( 0 𝒅 , 1)):
Pr 𝒙=( 𝑧 1 ,…, 𝑧 𝒅 ) = 2𝜋 − 𝒅 2 𝑒 − 𝑧 𝑧 2 2 +…+ 𝑧 𝒅 2 2
Nearly all mass in annulus of radius 𝒅 and width 𝑂(1):
Thm. For any 𝜷≤ 𝒅 all but 3 𝑒 −𝒄 𝜷 2 probability mass satisfies 𝒅 −𝜷≤ 𝒙 𝟐 ≤ 𝒅 +𝜷 for constant 𝒄

3. Nearest Neighbors and Random Projections
Given a database 𝑨 of 𝑛 points in ℝ 𝒅
Preprocess 𝑨 into a small data structure 𝑫
Should answer following queries fast:
Given 𝒒∈ ℝ 𝒅 find closest 𝒙∈𝑨: 𝑎𝑟𝑔𝑚𝑖 𝑛 𝑥∈𝑨 𝒒−𝒙 2
Project each 𝒙∈𝑨 onto 𝑓(𝒙), where 𝑓: ℝ 𝒅 → ℝ 𝒌
Pick 𝒌 vectors 𝒖 1 ,…, 𝒖 𝒌 i.i.d: 𝒖 𝑖 ∼ 𝑁 𝒅 0 𝒅 ,1
𝑓 𝒗 =(〈 𝒖 1 ,𝒗〉,…, 〈𝒖 𝒌 ,𝒗〉)
Will show that w.h.p. 𝑓 𝒗 2 ≈ 𝒌 𝒗 2
Return: 𝑎𝑟𝑔𝑚𝑖 𝑛 𝑥∈𝐴 𝑓 𝒒 −𝑓 𝒙 2 =𝑎𝑟𝑔𝑚𝑖 𝑛 𝑥∈𝐴 𝑓 𝒒−𝒙 2 ≈ 𝒌 𝑎𝑟𝑔𝑚𝑖 𝑛 𝑥∈𝐴 𝒒−𝒙 2

4. Random Projection Theorem
Pick 𝒌 vectors 𝒖 1 ,…, 𝒖 𝒌 i.i.d: 𝒖 𝑖 ∼ 𝑁 𝒅 0 𝒅 ,1
𝑓 𝒗 =(〈 𝒖 1 ,𝒗〉,…, 〈𝒖 𝒌 ,𝒗〉)
Will show that w.h.p. 𝑓 𝒗 2 ≈ 𝒌 𝒗 2
Thm. Fix 𝒗∈ ℝ 𝒅 then ∃𝑐>0: for 𝝐∈ 0,1 :
Pr 𝒖 𝑖 ∼ 𝑁 𝒅 0 𝒅 , 𝑓 𝒗 2 − 𝒌 𝒗 2 ≥𝝐 𝒌 𝒗 2 ≤3 𝑒 −𝑐𝒌 𝝐 2
Scaling: 𝒗 𝟐 =1
Key fact: 𝒖 𝑖 ,𝒗 = 𝑗=1 𝒅 𝒖 𝑖𝑗 𝒗 𝑗 ~𝑁 0, 𝒗 ​ =𝑁 0,1
Apply “Gaussian Annulus Theorem” with 𝒌=𝒅

5. Nearest Neighbors and Random Projections
Thm. Fix 𝒗∈ ℝ 𝒅 then ∃𝑐>0: for 𝝐∈ 0,1 :
Pr 𝒖 𝑖 ∼ 𝑁 𝒅 0 𝒅 , 𝑓 𝒗 2 − 𝒌 𝒗 2 ≥𝝐 𝒌 𝒗 2 ≤3 𝑒 −𝑐𝒌 𝝐 2
Return: 𝑎𝑟𝑔𝑚𝑖 𝑛 𝑥∈𝐴 𝑓 𝒒 −𝑓 𝒙 2 ≈ 𝒌 𝑎𝑟𝑔𝑚𝑖 𝑛 𝑥∈𝐴 𝒒−𝒙 2
Fix and let 𝒗= 𝒒− 𝒙 𝑖 for 𝒙 𝑖 ∈𝑨 and let 𝒌=𝑂 𝜸log 𝑛 𝝐 2
1±𝝐 𝒌 𝒒− 𝒙 𝑖 𝟐 ≈ 𝑓 𝒒 −𝑓 𝒙 𝟐 (prob. 1− 𝑛 −𝛾 )
Union bound:
For fixed 𝒒 distances to 𝑨 preserved with prob. 1− 𝑛 −𝛾+1

6. Separating Gaussians One-dimensional mixture of Gaussians:
𝑝 𝑥 = 𝑤 1 𝑝 1 𝑥 + 𝑤 2 𝑝 2 𝑥
E.g. modeling heights of men/women
Parameter estimation problem:
Given samples from a mixture of Gaussians
Q: Estimate means and (co)-variances
Sample origin problem:
Given samples from well-separated Gaussians
Q: Did they come from the same Gaussian?

7. Separating Gaussians Gaussian in 𝒅 dimensions ( 𝑁 𝒅 ( 0 𝒅 , 1)):
Pr 𝒙=( 𝑧 1 ,…, 𝑧 𝒅 ) = 2𝜋 − 𝒅 2 𝑒 − 𝑧 𝑧 2 2 +…+ 𝑧 𝒅 2 2
Nearly all mass in annulus of radius 𝒅 and width 𝑂(1)
Almost all mass in a slab 𝒙 −𝑐≤ 𝑥 1 ≤𝑐} for 𝑐=𝑂(1)
Pick 𝒙 ~Gaussian and rotate coordinates to make it 𝑥 1
Pick 𝒚∼Gaussian, w.h.p. projection of 𝒚 on 𝒙 is ∈[−𝑐,𝑐]
𝒙 −𝒚 2 ≈ 𝒙 𝒚 2 2

8. Separating Gaussians In coordinates: 𝒙=( 𝒅 ±𝑂 1 , 0,0,…, 0)
𝒙=( 𝒅 ±𝑂 1 , 0,0,…, 0)
𝒚=(±𝑂 1 , 𝒅 ±𝑂 1 , 0,…, 0)
W.h.p: 𝒙−𝒚 𝟐 𝟐 =2𝒅±𝑂( 𝒅 )

9. Separating Gaussians
Two spherical unit variance Gaussians centered at 𝒑,𝒒
𝛿= 𝒑−𝒒 𝟐
(𝒙∼𝑁(𝒑,1),𝒚∼𝑁(𝒒,1))
𝒙= 𝒅 ±𝑂 1 , 0,0,0…, 0
𝒚=(±𝑂 1 ,𝛿±𝑂 1 , 𝒅 ±𝑂 1 , 0,…, 0)
𝒙−𝒚 𝟐 𝟐 = 𝛿 2 +2𝒅±𝑂 𝒅

10. Separating Gaussians Same Gaussian: 𝒙−𝒚 𝟐 𝟐 =2𝒅±𝑂( 𝒅 )
𝒙−𝒚 𝟐 𝟐 =2𝒅±𝑂( 𝒅 )
Different Gaussians:
𝒙−𝒚 𝟐 𝟐 = 𝛿 2 +2𝒅±𝑂 𝒅
Separation requires:
2𝒅±𝑂 𝒅 < 𝛿 2 +2𝒅±𝑂 𝒅
𝑂 𝒅 < 𝛿 2
𝜔 𝒅 𝟏/𝟒 < 𝛿

11. Fitting Spherical Gaussian to Data
Given samples 𝒙 𝟏 , 𝒙 𝟐 ,…, 𝒙 𝒏
Q: What are parameters of best fit 𝑁(𝝁,𝜎)?
Pr 𝒙 𝒊 =( 𝑧 1 ,…, 𝑧 𝒅 ) = 2𝜋 − 𝒅 2 𝑒 − ( 𝝁 1 −𝑧 1 ) ( 𝝁 2 −𝑧 2 ) 2 +…+ ( 𝝁 𝒅 −𝑧 𝒅 ) = 2𝜋 − 𝒅 2 𝑒 − | 𝝁−𝒛|
Pr 𝒙 𝟏 = 𝒛 𝟏 , 𝒙 𝟐 = 𝒛 𝟐 ,…, 𝒙 𝒏 = 𝒛 𝒏 = 2𝜋 − 𝒅𝑛 2 𝑒 − | 𝝁− 𝒛 𝟏 | | 𝝁− 𝒛 𝟐 | …+| 𝝁− 𝒛 𝒅 | 𝜎 2

12. Maximum Likelihood Estimator
PDF: 2𝜋 − 𝒅𝑛 2 𝑒 − | 𝝁− 𝒛 𝟏 | | 𝝁− 𝒛 𝟐 | …+| 𝝁− 𝒛 𝒅 | 𝜎 2
MLE for 𝝁 is 𝝁= 1 𝑛 𝒙 𝟏 + 𝒙 𝟐 +…+ 𝒙 𝒏
Take gradient w.r.t 𝝁 and make it =0
𝛻 𝝁 𝝁−𝒙 =2(𝝁−𝒙)
2 𝝁− 𝒙 𝟏 +2 𝝁− 𝒙 𝟐 +…+2 𝝁− 𝒙 𝒏 =0

13. MLE for Variance 𝑎=| 𝝁− 𝒙 𝟏 | 2 2 +| 𝝁− 𝒙 𝟐 | 2 2 +…+| 𝝁− 𝒙 𝒅 | 2 2
𝑎=| 𝝁− 𝒙 𝟏 | | 𝝁− 𝒙 𝟐 | …+| 𝝁− 𝒙 𝒅 | 2 2
𝜈=1/2 𝜎 2
PDF: 𝑒 −𝑎𝜈 𝒙∈ 𝑅 𝒅 𝑒 − 𝜈 𝒙 𝑑𝒙 𝑛
Log(PDF): −𝑎𝜈−𝑛 ln 𝒙∈ 𝑅 𝒅 𝑒 − 𝜈 𝒙 𝑑𝒙
Differentiate w.r.t. 𝜈 and set derivative =0

14. MLE for Variance Log(PDF): −𝑎𝜈−𝑛 ln 𝒙∈ 𝑅 𝒅 𝑒 − 𝜈 𝒙 2 2 𝑑𝒙
−𝑎+𝑛 𝒙∈ 𝑅 𝒅 𝒙 𝑒 − 𝜈 𝒙 𝑑𝒙 𝒙∈ 𝑅 𝒅 𝑒 − 𝜈 𝒙 𝑑𝒙
𝑦= 𝜈𝒙 :
−𝑎+ 𝑛 𝜈 𝒙∈ 𝑅 𝒅 𝑦 2 𝑒 − 𝑦 2 𝑑𝒙 𝒙∈ 𝑅 𝒅 𝑒 − 𝑦 2 𝑑𝒙 =−𝑎+ 𝑛 𝜈 × 𝒅 2 =0
𝜈= 1 2 𝜎 2 ⇒ MLE(𝜎)= 𝑎 𝑛𝒅 = sample standard deviation