1. Testing with Alternative Distances
Gautam “G” Kamath
FOCS 2017 Workshop: Frontiers in Distribution Testing
October 14, 2017
Based on joint works with
Jayadev Acharya
Cornell
Constantinos Daskalakis
MIT
John Wright
MIT

2. The story so far… Test whether 𝑝=𝑞 versus ℓ 1 𝑝,𝑞 ≥𝜀
ℓ 1 𝑝,𝑞 >𝜀
0< ℓ 1 𝑝,𝑞 ≤𝜀
𝑝=𝑞
Test whether 𝑝=𝑞 versus ℓ 1 𝑝,𝑞 ≥𝜀
Domain of [𝑛]
Success probability ≥2/3
Goal: Strongly sublinear sample complexity
𝑂( 𝑛 1−𝛾 ) for some 𝛾>0
Identity testing (samples from 𝑝, known 𝑞)
Θ( 𝑛 / 𝜀 2 ) samples
[BFFKR’01, P’08, VV’14]
Closeness testing (samples from 𝑝, 𝑞)
Θ( max { 𝑛 2/3 / 𝜀 4/3 , 𝑛 / 𝜀 2 } ) samples
[BFRSW’00, V’11, CDVV’14]

3. Generalize: Different Distances
𝑝=𝑞 or ℓ 1 𝑝,𝑞 ≥𝜀?
𝑑 1 𝑝,𝑞 ≤ 𝜀 1 or 𝑑 2 𝑝,𝑞 ≥ 𝜀 2 ?

4. Generalize: Different Distances
𝑑 1 𝑝,𝑞 ≤ 𝜀 1 or 𝑑 2 𝑝,𝑞 ≥ 𝜀 2 ?
Are 𝑝 and 𝑞 𝜀 1 -close in 𝑑 1 (.,.), or 𝜀 2 -far in 𝑑 2 (.,.)?
Distances of interest: ℓ 1 , ℓ 2 , 𝜒 2 , KL, Hellinger
Classic identity testing: 𝜀 1 =0, 𝑑 2 = ℓ 1
Can we characterize sample complexity for each pair of distances?
Which distribution distances are sublinearly testable? [DKW’18]
Wait, but… why?

5. Wait, but… why? Tolerance for model misspecification
Useful as a proxy in classical testing problems
𝑑 1 as 𝜒 2 distance is useful for composite hypothesis testing
Monotonicity, independence, etc. [ADK’15]
Other distances are natural in certain testing settings
𝑑 2 as Hellinger distance is sometimes natural in multivariate settings
Bayes networks, Markov chains [DP’17,DDG’17]
Costis’ talk

6. Wait, but… why? Tolerance for model misspecification
Useful as a proxy in classical testing problems
𝑑 1 as 𝜒 2 distance is useful for composite hypothesis testing
Monotonicity, independence, etc. [ADK’15]
Other distances are natural in certain testing settings
𝑑 2 as Hellinger distance is sometimes natural in multivariate settings
Bayes networks, Markov chains [DP’17,DDG’17]
Costis’ talk

7. Tolerance Is 𝑝 equal to 𝑞, or are they far from each other?
ideal model
Is 𝑝 equal to 𝑞, or are they far from each other?
But why do we know 𝑞 exactly?
Models are inexact
Measurement errors
Imprecisions in nature
𝑝, 𝑞 may be “philosophically” equal, but not literally equal
When can we test 𝑑 1 𝑝,𝑞 ≤ 𝜀 1 versus ℓ 1 𝑝,𝑞 ≥𝜀?
CLT approximations…
Read data point wrong…
observed model

8. Tolerance 𝑑 1 𝑝,𝑞 ≤ 𝜀 1 vs. ℓ 1 𝑝,𝑞 ≥𝜀? ℓ 1 𝑝,𝑞 ≤𝜀/2 vs. ℓ 1 𝑝,𝑞 ≥𝜀?
ℓ 1 𝑝,𝑞 >𝜀
𝜀 2 < ℓ 1 𝑝,𝑞 ≤𝜀
ℓ 1 𝑝,𝑞 ≤ 𝜀 2
ℓ 1 𝑝,𝑞 >𝜀
0< ℓ 1 𝑝,𝑞 ≤𝜀
𝑝=𝑞
ℓ 1 𝑝,𝑞 >𝜀
𝜒 2 𝑝,𝑞 > 𝜀 2 /2
ℓ 1 𝑝,𝑞 ≤𝜀
𝑑 1 𝑝,𝑞 ≤ 𝜀 1 vs. ℓ 1 𝑝,𝑞 ≥𝜀?
What 𝑑 1 ? How about ℓ 1 ?
ℓ 1 𝑝,𝑞 ≤𝜀/2 vs. ℓ 1 𝑝,𝑞 ≥𝜀?
No! Θ(𝑛/ log 𝑛 ) samples [VV’10]
Chill out, relax…
𝜒 2 -distance: 𝜒 2 𝑝,𝑞 = 𝑖∈Σ 𝑝 𝑖 − 𝑞 𝑖 𝑞 𝑖
Cauchy-Schwarz: 𝜒 2 𝑝,𝑞 ≥ ℓ 1 2 (𝑝,𝑞)
𝜒 2 𝑝,𝑞 ≤ 𝜀 2 /4 vs. ℓ 1 𝑝,𝑞 ≥𝜀?
Yes! 𝑂( 𝑛 / 𝜀 2 ) samples [ADK’15]
𝜒 2 𝑝,𝑞 ≤ 𝜀 2 /2

9. Details for a 𝜒 2 -Tolerant Tester
Goal: Distinguish (i) 𝜒 2 𝑝,𝑞 ≤ 𝜀 versus (ii) ℓ 1 2 𝑝,𝑞 ≥ 𝜀 2
Draw 𝑃𝑜𝑖𝑠𝑠𝑜𝑛(𝑚) samples from 𝑝 (“Poissonization”)
𝑁 𝑖 : number of appearances of symbol 𝑖
𝑁 𝑖 ~ 𝑃𝑜𝑖𝑠𝑠𝑜𝑛 𝑚⋅ 𝑝 𝑖
𝑁 𝑖 ’s are now independent!
Statistic: 𝑍= 𝑖∈Σ 𝑁 𝑖 −𝑚⋅ 𝑞 𝑖 2 − 𝑁 𝑖 𝑚⋅ 𝑞 𝑖
Acharya, Daskalakis, K. Optimal Testing for Properties of Distributions. NIPS 2015.

10. Details for a 𝜒 2 -Tolerant Tester
Goal: Distinguish (i) 𝜒 2 𝑝,𝑞 ≤ 𝜀 versus (ii) ℓ 1 2 𝑝,𝑞 ≥ 𝜀 2
Statistic: 𝑍= 𝑖∈[𝑛] 𝑁 𝑖 −𝑚⋅ 𝑞 𝑖 2 − 𝑁 𝑖 𝑚⋅ 𝑞 𝑖
𝑁 𝑖 : # of appearances of 𝑖; 𝑚: # of samples
𝐸 𝑍 =𝑚⋅ 𝜒 2 (𝑝, 𝑞)
(i): 𝐸 𝑍 ≤𝑚⋅ 𝜀 2 2 , (ii): 𝐸 𝑍 ≥𝑚⋅ 𝜀 2
Can bound variance of 𝑍 with some work
Need to avoid low prob. elements of 𝑞
Either ignore 𝑖 such that 𝑞 𝑖 ≤ 𝜀 2 10𝑛 ; or
Mix lightly (𝑂( 𝜀 2 )) with uniform distribution (also in [G’16])
Apply Chebyshev’s inequality
Side-Note:
Pearson’s 𝜒 2 -test uses statistic 𝑖 𝑁 𝑖 −𝑚 ⋅𝑞 𝑖 2 𝑚⋅ 𝑞 𝑖
Subtracting 𝑁 𝑖 in the numerator gives an unbiased estimator and importantly may hugely decrease variance
[Zelterman’87]
[VV’14, CDVV’14, DKN’15]
Acharya, Daskalakis, K. Optimal Testing for Properties of Distributions. NIPS 2015.

11. Tolerant Identity Testing
Harder
(Implicit in [DK’16])
Daskalakis, K., Wright. Which Distribution Distances are Sublinearly Testable? SODA 2018.

12. Tolerant Testing Takeaways
Can handle ℓ 2 or 𝜒 2 tolerance at no additional cost
Θ( 𝑛 / 𝜀 2 ) samples
KL, Hellinger, or ℓ 1 tolerance are expensive
Θ(𝑛/ log 𝑛 ) samples
KL result based off hardness of entropy estimation
Closeness testing (𝑞 unknown): Even 𝜒 2 tolerance is costly!
Only ℓ 2 tolerance is free
Proven via hardness of ℓ 1 -tolerant identity testing
Since 𝑞 is unknown, 𝜒 2 is no longer a polynomial

13. Application: Testing for Structure
Composite hypothesis testing
Test against a class of distributions!
𝑝∈𝐶 versus ℓ 1 𝑝,𝐶 >𝜀
Example: 𝐶 = all monotone distributions
𝑝 𝑖 ’s are monotone non-increasing
Others: unimodality, log-concavity, monotone hazard rate, independence
All can be tested in Θ 𝑛 / 𝜀 2 samples [ADK’15]
Same complexity as vanilla uniformity testing!
min 𝑞∈𝐶 ℓ 1 (𝑝,𝑞)

14. Testing by Learning Goal: Distinguish 𝑝∈𝐶 from ℓ 1 𝑝,𝐶 >𝜀
Learn-then-Test:
Learn hypothesis 𝑞∈𝐶 such that
𝑝∈𝐶⇒ 𝜒 2 𝑝,𝑞 ≤ 𝜀 2 /2 (needs cheap “proper learner” in 𝜒 2 )
ℓ 1 𝑝,𝐶 >𝜀⇒ ℓ 1 𝑝,𝑞 >𝜀 (automatic since 𝑞∈𝐶)
Perform “tolerant testing”
Given sample access to 𝑝 and description of 𝑞, distinguish
𝜒 2 𝑝,𝑞 ≤ 𝜀 2 /2 from ℓ 1 𝑝,𝑞 >𝜀
Tolerant testing (step 2) is 𝑂( 𝑛 / 𝜀 2 )
Naïve approach (using ℓ 1 instead of 𝜒 2 ) would require Ω(𝑛/ log 𝑛 )
Proper learners in 𝜒 2 (step 1)?
Claim: This is cheap
Naïve approach of using L1 tolerant testing would require n/log n samples. This is why different distances are important!

15. Hellinger Testing Change 𝑑 2 instead of 𝑑 1
Hellinger distance: 𝐻 2 𝑝,𝑞 = 1 2 𝑖∈[𝑛] 𝑝 𝑖 − 𝑞 𝑖 2
Between linear and quadratic relationship with ℓ 1
𝐻 2 𝑝,𝑞 ≤ ℓ 1 𝑝,𝑞 ≤𝐻(𝑝,𝑞)
Natural distance when considering a collection of iid samples
Comes up in some multivariate testing problems 2:55)
Testing 𝑝=𝑞 vs. 𝐻 𝑝,𝑞 ≥𝜀?
Trivial results via ℓ 1 testing
Identity: 𝑂( 𝑛 / 𝜀 4 ) samples
Closeness: 𝑂(max { 𝑛 2/3 / 𝜀 8/3 , 𝑛 / 𝜀 4 } ) samples

16. Hellinger Testing But you can do better!
Testing 𝑝=𝑞 vs. 𝐻 𝑝,𝑞 ≥𝜀?
Trivial results via ℓ 1 testing
Identity: 𝑂( 𝑛 / 𝜀 4 ) samples
Closeness: 𝑂(max { 𝑛 2/3 / 𝜀 8/3 , 𝑛 / 𝜀 4 } ) samples
But you can do better!
Identity: Θ( 𝑛 / 𝜀 2 ) samples
No extra cost for ℓ 2 or 𝜒 2 tolerance either!
Closeness: Θ(min { 𝑛 2/3 / 𝜀 8/3 , 𝑛 3/4 / 𝜀 2 } ) samples
LB and previous UB in [DK’16]
Similar chi-squared statistics as [ADK’15] and [CDVV’14]
Some tweaks and more careful analysis to handle Hellinger
Daskalakis, K., Wright. Which Distribution Distances are Sublinearly Testable? SODA 2018.

17. Miscellanea 𝑝=𝑞 vs. 𝐾𝐿 𝑝,𝑞 ≥𝜀?
Trivially impossible, due to ratio between 𝑝 𝑖 and 𝑞 𝑖
𝑝 𝑖 =𝛿, 𝑞 𝑖 =0, 𝛿→0
Upper bounds for Ω(𝑛/ log 𝑛 ) testing problems?
i.e., 𝐾𝐿 𝑝,𝑞 ≤ 𝜀 2 /4 vs. ℓ 1 𝑝,𝑞 ≥𝜀?
Use estimators mentioned in Jiantao’s talk

18. Thanks!