1. Uncertain Compression
&
Graph Coloring
Madhu Sudan
Harvard
Based on joint works with:
(1) Adam Kalai (MSR), Sanjeev Khanna (U.Penn), Brendan Juba (WUStL)
(2) Elad Haramaty (Harvard)
(3) Badih Ghazi (MIT), Elad Haramaty (Harvard), Pritish Kamath (MIT)
August 4, 2017
IITB: Uncertain Compression & Coloring

2. Classical Compression
The Shannon setting
Alice gets 𝑚∈ [𝑁] chosen from distribution 𝑃
Sends compression 𝑦 = 𝐸 𝑃 𝑚 ∈ 0,1 ∗ to Bob.
Bob computes 𝑚 = 𝐷 𝑃 𝑦
(Alice and Bob both know 𝑃).
Require 𝑚 = 𝑚 (whp).
Goal: min 𝐸 𝑃 , 𝐷 𝑃 𝔼xp 𝑚∼𝑃 | 𝐸 𝑃 𝑚 |
Further (technical) requirement: “Prefix-free”
∀𝑚≠𝑚′, 𝐸 𝑃 (𝑚) not prefix of 𝐸 𝑃 𝑚 ′
Ensures we can encode sequence of messages
August 4, 2017
IITB: Uncertain Compression & Coloring

3. IITB: Uncertain Compression & Coloring
Shannon+Huffman
[Kraft]: Prefix-free compression:
Encoding message 𝑖 with length ℓ 𝑖 possible iff
𝑖 2 − ℓ 𝑖 ≤1
[Shannon]:
Assign length − log 2 𝑃(𝑖) with message 𝑖
Expected length 𝔼 𝑖 − log 𝑃(𝑖) ≤𝐻 𝑃 +1
Motivates 𝐻 𝑃 ≝ 𝔼 𝑖 − log 𝑃 𝑖
[Huffman]:
Constructive, explicit, optimal
August 4, 2017
IITB: Uncertain Compression & Coloring

4. IITB: Uncertain Compression & Coloring
Fundamental problem:
gzip, Lempel-Ziv …
Leads to entropy: Fundamental measure.
Fundamental role in “learning”
Learning ≡ Compression
A goal in language too!
Language evolves
Words introduced, others fade. Why?
Intuitive explanation: distributions on messages evolve!
August 4, 2017
IITB: Uncertain Compression & Coloring

5. Compression as proxy for language
Understanding languages: Complex, not all forces well understood. Others hard to analyze.
Compression: Clean mathematical problem. Faces similar issues as language. (Sometimes) easier to analyze.
But to model issues associated with natural language, need to incorporate uncertainty!
People don’t have same priors on messages.
Need to estimate/bound each others priors
Can compression work with uncertain priors?
August 4, 2017
IITB: Uncertain Compression & Coloring

6. IITB: Uncertain Compression & Coloring
Outline
Part 1: Motivation
Part 2: Formalism
Part 3: Randomized Solution
Part 4: Issues with Randomized Solution
Part 5: Deterministic Issues.
August 4, 2017
IITB: Uncertain Compression & Coloring

7. Uncertain Compression
Design encoding/decoding schemes (𝐸/𝐷) so that
Sender has distribution 𝑃 on [𝑁]
Receiver has distribution 𝑄 on [𝑁]
Sender gets 𝑚∈[𝑁]
Sends 𝐸(𝑃,𝑚) to receiver.
Receiver receives 𝑦 = 𝐸(𝑃,𝑚)
Decodes to 𝑚 =𝐷(𝑄,𝑦)
Want: 𝑚= 𝑚 (provided 𝑃,𝑄 close),
While minimizing 𝔼 xp 𝑚∼𝑃 |𝐸(𝑃,𝑚)|
August 4, 2017
IITB: Uncertain Compression & Coloring

8. Proximity of Distributions
Many alternatives.
Our goal: Find anything non-trivial that allows compression.
Eventual choice:
Δ 𝑃,𝑄 = max 𝑚∈[𝑁] max log 𝑃 𝑚 𝑄 𝑚 , log 𝑄 𝑚 𝑃 𝑚
“Symmetrized worst-case KL divergence”
KL Divergence: 𝐷 𝑃,𝑄 =𝔼 xp 𝑚∼𝑃 log 𝑃 𝑚 𝑄(𝑚)
(So trivially: 𝐷 𝑃,𝑄 ≤Δ(𝑃,𝑄) )
Question: Can message be compressed to within 𝑓 𝐻 𝑃 ,Δ ? Or is it 𝑓(𝐻 𝑃 ,Δ,𝑁)?
August 4, 2017
IITB: Uncertain Compression & Coloring

9. Solution 1: Assuming Randomness
Assume sender+receiver share 𝑟∼ Unif( 0,1 𝑡 )
In particular 𝑟 independent of 𝑃,𝑄,𝑚
Compression scheme:
Let 𝑟=( 𝑟 1 ,…, 𝑟 𝑁 ) with 𝑟 𝑖 ∈ 0,1 𝑡 𝑁
Sender sends prefix 𝑧 of 𝑟 𝑚 ; long enough so that ∀𝑚′ s.t. 𝑧 is a prefix of 𝑟 𝑚 ′ : 𝑃 𝑚 ′ < 𝑃 𝑚 4 Δ
𝔼 xp 𝑟 𝑟 𝑚 ≈− log 𝑃 𝑚 +2Δ ⇒
Thm: Expected compression length ≤𝐻 𝑃 +2Δ
Deterministic?
August 4, 2017
IITB: Uncertain Compression & Coloring

10. Combinatorial Reinterpretation
Can fix 𝑃(𝑚) (adds − log 𝑃 𝑚 to compression).
Define: 𝐴 0 = 𝑚 , 𝐴 1 = 𝑚 ′ |𝑃 𝑚 ′ ≥ 4 −Δ ⋅𝑃(𝑚) ,
𝐴 𝑖 = 𝑚 ′ |𝑃 𝑚 ′ ≥ 4 −𝑖⋅Δ ⋅𝑃(𝑚)
Similarly 𝐵 1 = 𝑚 ′ | 𝑄 𝑚 ′ ≥ 2 −Δ ⋅𝑃(𝑚) ,
𝐵 𝑖 = 𝑚 ′ | 𝑄 𝑚 ′ ≥ 2 − 2𝑖−1 ⋅Δ ⋅𝑃(𝑚)
Nesting: 𝐴 0 ⊆ 𝐵 1 ⊆ 𝐴 1 ⊆ 𝐵 2 ⊆ 𝐴 2 ⋯
Sizes: 𝐴 𝑖 ≤𝐾⋅ 𝐶 2𝑖 , 𝐵 𝑖 ≤𝐾⋅ 𝐶 2𝑖−1 for
𝐾= 1 𝑃 𝑚 ; 𝐶= 2 Δ
Question: Given 𝐾,𝐶 can 𝑚 be distinguished from 𝑚 ′ ∈ 𝐵 1 with 𝑂 𝐾,𝐶 1 bits?
August 4, 2017
IITB: Uncertain Compression & Coloring

11. Compression ≡ Coloring
Weak Uncertainty graph 𝑊 𝑁,𝐾,𝐶
Vertices = 𝐴 0 , 𝐴 1 ,…, 𝐴 ℓ :
Nested, 𝐴 0 =1, 𝐴 𝑖 ≤𝐾⋅ 𝐶 2𝑖 , 𝐴 ℓ =[𝑁]
Edges: 𝐴 0 , 𝐴 1 ,…, 𝐴 ℓ ↔ 𝐴 0 ′ , 𝐴 1 ′ ,…, 𝐴 ℓ ′ ′ iff
𝐴 0 ≠ 𝐴 0 ′
𝐴 𝑖 ⊆ 𝐴 𝑖+1 ′ ; 𝐴 𝑖 ′ ⊆ 𝐴 𝑖+1
Claim: Compression length =𝑓 𝐻 𝑃 ,Δ iff
∀𝐾,𝐶, 𝑁 𝜒 𝑊 𝑁,𝐾,𝐶 = 𝑂 𝐾,𝐶 (1)
𝜒 𝑊 𝑁,𝐾,𝐶 = open!
Did we reduce to a harder problem? 
August 4, 2017
IITB: Uncertain Compression & Coloring

12. Bounding chromatic number
Upper bounds:
Easy! Just give the coloring!
… not always. E.g., “Shift Graph” 𝑆 𝑛,𝑘
Vertices: 𝑛 𝑘 ≝ Sequences of 𝑘 distinct elements of [𝑛]
Edges 𝑖 1 , 𝑖 2 ,…, 𝑖 𝑘 ↔( 𝑖 2 , 𝑖 3 ,…, 𝑖 𝑘 , 𝑖 𝑘+1 )
Thm: [Cole-Vishkin, Linial] If 𝑘≥ log ∗ 𝑛 , then 𝜒 𝑆 𝑛,𝑘 =3. More generally
𝜒 𝑆 𝑛,𝑘 =max 3, log 𝑘+Θ 1 𝑛
Lower bounds … much more challenging!
August 4, 2017
IITB: Uncertain Compression & Coloring

13. IITB: Uncertain Compression & Coloring
Some Results
[Haramaty+S.] 𝜒 𝑊 𝑁,𝐾,𝐶 ≤ exp 𝐾. 𝐶 ℓ ⋅ log ℓ 𝑁 ∀ℓ
[Golowich] 𝜒 𝑊 𝑁,𝐾,𝐶 ≤ exp exp 𝐾⋅ 𝐶 ℓ ⋅ log 2ℓ 𝑁 ∀ℓ
[Trivial] 𝜒 𝑊 𝑁,𝐾,𝐶 ≥𝐾⋅ 𝐶 2
For further understanding define 𝑊 𝑁,𝐾,𝐶 ℓ
Like 𝑊 𝑁,𝐾,𝐶 but without size restriction on 𝐴 ℓ . (so wlog 𝐴 ℓ =[𝑁])
Upper bounds hold even for 𝑊 𝑁,𝐾,𝐶 ℓ
[Haramaty+S]: 𝜒 𝑊 𝑁,𝐾,𝐶 ℓ = log Ω ℓ 𝑁
Need slow growth for long to get 𝑁-independence
August 4, 2017
IITB: Uncertain Compression & Coloring

14. Upper Bounds – 1 [Cole-Vishkin/Linial]
Coloring Shift graph:
Given coloring 𝜒: 𝑆 𝑛,𝑘−1 → 0,1 𝑐 , construct coloring 𝜒 ′ : 𝑆 𝑛,𝑘 → 𝑐 ×{0,1} as follows:
To color 𝑖 1 ,…, 𝑖 𝑘 :
Let (𝑎 1 ,…, 𝑎 𝑐 )=𝜒( 𝑖 1 ,…, 𝑖 𝑘−1 )
And 𝑏 1 ,…, 𝑏 𝑐 =𝜒 𝑖 2 ,…, 𝑖 𝑘
Let 𝑗 be least index s.t. 𝑎 𝑗 ≠ 𝑏 𝑗 (exists!)
𝜒 𝑖 1 ,…, 𝑖 𝑘 = 𝑗, 𝑎 𝑗
Valid? 𝜒 𝑖 2 , …, 𝑖 𝑘+1 = 𝑗, 𝑏 𝑗 or 𝑗 ′ ,𝑥 for 𝑗 ′ ≠𝑗 !
August 4, 2017
IITB: Uncertain Compression & Coloring

15. Generalizing: Homorphisms
𝐺 homomorphic to 𝐻 (𝐺→𝐻) if
∃ 𝜙:𝑉 𝐺 →𝑉 𝐻 s.t. 𝑢 ↔ 𝐺 𝑣⇒𝜙 𝑢 ↔ 𝐻 𝜙 𝑣
Homorphisms?
𝐺 is 𝑘-colorable ⇔ 𝐺→ 𝐾 𝑘
𝐺→𝐻 and 𝐻→𝐿⇒𝐺→𝐿
Homomorphisms and Shift/Uncertainty graphs.
𝑆 𝑛,𝑘 → 𝑆 𝑛,𝑘−1 → 𝑆 𝑛,𝑘−2 →⋯
𝑊 𝑁,𝐾,𝐶 = 𝑊 𝑁,𝐾,𝐶 𝑁 → 𝑊 𝑁,𝐾,𝐶 𝑁−1 →⋯→ 𝑊 𝑁,𝐾,𝐶 ℓ →⋯
Suffices to upper bound 𝜒 𝑊 𝑁,𝐾,𝐶 ℓ
August 4, 2017
IITB: Uncertain Compression & Coloring

16. Degree of Homomorphisms
Say 𝜙:𝐺→𝐻 𝐺
𝑑 𝜙 𝑢 ≝ 𝜙 𝑣 𝑣 ↔ 𝐺 𝑢 |
𝑑 𝜙 ≝max 𝑢 { 𝑑 𝜙 𝑢 } 𝐻
Lemma [HS]:
𝜒 𝐺 ≤𝑂( 𝑑 𝜙 2 log 𝜒 𝐻 )
Lemma [Golowich]:
𝜒 𝐺 ≤𝑂( exp 𝑑 𝜙 log log 𝜒 𝐻 )
For 𝜙: 𝑊 𝑁,𝐾,𝐶 ℓ → 𝑊 𝑁,𝐾,𝐶 ℓ−1
𝑑 𝜙 = exp 𝐾⋅ 𝐶 ℓ
August 4, 2017
IITB: Uncertain Compression & Coloring

17. Proof: (of 𝜒 𝐺 ≤𝑂( 𝑑 𝜙 2 log 𝜒 𝐻 ) )
Denote 𝜒 𝐻 =𝑐; 𝑑 𝜙 =𝑑
Let 𝑀=𝑂 𝑑⋅ log 𝑐 ; 𝑡=2𝑑
Claim: ∃ ℎ 1 ,…, ℎ 𝑀 , ℎ 𝑖 : 𝑐 → 𝑡 s.t.
∀𝑖∉𝑆⊆ 𝑐 , 𝑆 ≤𝑑, ∃𝑗 𝑠.𝑡. ℎ 𝑗 𝑖 ∉ ℎ 𝑗 𝑆
Proof: Pick ℎ 𝑗 ’s at random
Claim: ∃𝑡⋅𝑀 coloring of 𝐺
Proof: Given 𝜒:𝐻→[𝑐], let 𝜒 ′ :𝐺→ 𝑀 ×[𝑡] be:
Let 𝑆 𝑢 = 𝜒 𝜙 𝑣 ) 𝑣 ↔ 𝐺 𝑢 ; 𝑖=𝜒 𝜙 𝑢
Let 𝑗 be s.t. ℎ 𝑗 𝑖 ∉ ℎ 𝑗 𝑆
𝜒 ′ 𝑢 = 𝑗, ℎ 𝑗 𝑖
August 4, 2017
IITB: Uncertain Compression & Coloring

18. Lower bounds: 𝜒 𝑊 𝑁,𝐾,𝐶 ℓ ≥ log (2ℓ) 𝑁
Claim: 𝑆 𝑁,2ℓ subgraph of 𝑊 𝑁,𝐾,𝐶 ℓ
Proof: by inspection …
Linial’s Proof: 𝜒 𝑆 𝑁,ℓ ≥ log 𝜒 𝑆 𝑁,ℓ−1
⇐𝜒 𝑆 𝑁,ℓ−1 ≤ 2 𝜒 𝑆 𝑁,ℓ (lower bounds by upper bounds!)
Given 𝜒: 𝑆 𝑁,ℓ →[𝑐], let 𝜒 ′ : 𝑆 𝑁,ℓ−1 → 2 [𝑐] be:
𝜒 ′ 𝑖 1 ,…, 𝑖 ℓ−1 = 𝜒 𝑖 1 ,… 𝑖 ℓ | 𝑖 ℓ
Claim: 𝜒 ′ 𝑖 1 ,…, 𝑖 ℓ−1 ≠ 𝜒 ′ 𝑖 2 ,…, 𝑖 ℓ
Proof: 𝜒 𝑖 1 ,…, 𝑖 ℓ ∈ 𝜒 ′ 𝑖 1 ,…, 𝑖 ℓ −1
But 𝜒 𝑖 1 ,…, 𝑖 ℓ ≠𝜒 𝑖 2 ,…, 𝑖 ℓ+1 ⇒𝜒 𝑖 1 ,…, 𝑖 ℓ ∉ 𝜒 ′ 𝑖 2 ,…, 𝑖 ℓ
August 4, 2017
IITB: Uncertain Compression & Coloring

19. IITB: Uncertain Compression & Coloring
Conclusion
Compression (Uncertain) ≡ Graph Coloring
Unfortunately – latter is hard!
(not only to solve optimally, but also to understand analytically)
Intriguing Special Case:
𝑊 𝑁 ∗ : 𝐴 𝑖 ≤3𝑖 (linear, not exponential, growth)
Is 𝜒 𝑊 𝑁 ∗ =𝑂 1 ?
Fundamental underlying question: Is entropy the correct measure of natural compressibility
August 4, 2017
IITB: Uncertain Compression & Coloring

20. IITB: Uncertain Compression & Coloring
Thank You
August 4, 2017
IITB: Uncertain Compression & Coloring