1. Information Complexity Lower Bounds
Rotem Oshman, Princeton CCI
Based on:
Bar-Yossef,Jayram,Kumar,Srinivasan’04
Braverman,Barak,Chen,Rao’10

2. Communication Complexity
𝑓 𝑋,𝑌 = ? 𝑋 𝑌
Yao ‘79, “Some complexity questions related to distributive computing”

3. Communication Complexity
Applications:
Circuit complexity
Streaming algorithms
Data structures
Distributed computing
Property testing …

4. Deterministic Protocols
A protocol Π specifies, at each point:
Which player speaks next
What should the player say
When to halt and what to output
Formally, Π: 0,1 ∗ → 𝐴,𝐵,⊥ × 0,1
what we’ve said so far
who speaks next:
𝐴= Alice,
𝐵= Bob,
⊥ = halt
what to say/output

5. Randomized Protocols Can use randomness to decide what to say
Private randomness: each player has a separate source of random bits
Public randomness: both players can use the same random bits
Goal: for any 𝑋,𝑌 compute 𝑓 𝑋,𝑌 correctly with probability ≥1 −𝜖
Communication complexity: worst-case length of transcript in any execution

6. Randomness Can Help a Lot
Example: Equality 𝑋,𝑌
Input: 𝑋,𝑌∈ 0,1 𝑛
Output: is 𝑋=𝑌 ?
Trivial protocol: Alice sends 𝑋 to Bob
For deterministic protocols, this is optimal!

7. Equality Lower Bound
0 𝑛
1 𝑛 … 1
#rectangles ≤ 2 transcript

8. Randomized Protocol Protocol with public randomness:
Select random 𝑍∈ 0,1 𝑛
Alice sends 𝑋,𝑍 = 𝑖=1 𝑛 𝑋 𝑖 𝑍 𝑖 mod 2
Bob accepts iff 𝑌,𝑍 = 𝑋,𝑍
If 𝑋=𝑌: always accept
If 𝑋≠𝑌:
𝑋,𝑍 + 𝑌,𝑍 mod 2= 𝑋+𝑌,𝑍 mod 2
Reject with probability 1/2
non-zero vector

9. Set Disjointness Input: 𝑋,𝑌⊆ 1,…,𝑛 Output: 𝑋∩𝑌=∅ ?
Theorem [Kalyanasundaran, Schnitger ‘92, Razborov ‘92]: randomized CC = Ω 𝑛
Easy to see for deterministic protocols
Today we’ll see a proof by Bar-Yossef, Jayram, Kumar, Srinivasan ‘04

10. Application: Streaming Lower Bounds
Streaming algorithm:
Example: how many distinct items in the data?
Reduction from Disjointness [Alon, Matias, Szegedy ’99]
How much space
is required to approximate f(data)?
algorithm
data

11. Reduction from Disjointness:
Fix a streaming algorithm for Distinct Elements with space 𝑆, universe size 𝑛
Construct a protocol for Disj. with 𝑛 elements:
𝑋={ 𝑥 1 ,…, 𝑥 𝑘 }
𝑌={ 𝑦 1 ,…, 𝑦 ℓ }
algorithm
𝑋∩𝑌=∅ ⇔ #distinct elements in 𝑋∪𝑌 is 𝑋 + 𝑌
State of the algorithm
and 𝑋
(#bits = 𝑆+log 𝑛)

12. Application 2: KW Games Circuit depth lower bounds:
How deep does the circuit need to be? ∧ ∨
𝑥 1 …
𝑥 𝑛
𝑓(𝑥 1 ,…, 𝑥 𝑛 )

13. Application 2: KW Games
Karchmer-Wigderson’93,Karchmer-Raz-Wigderson’94:
find 𝑖 such that 𝑋 𝑖 ≠ 𝑌 𝑖
𝑋 :𝑓 𝑋 =0
𝑌:𝑓 𝑌 =1

14. Application 2: KW Games
Claim: if 𝐾 𝑊 𝑓 has deterministic CC ≥𝑑, then 𝑓 requires circuit depth ≥𝑑.
Circuit with depth 𝑑 ⇒ protocol with length 𝑑 1 ∧ ∨
𝑥 1 …
𝑥 𝑛 1 1 1
𝑋 :𝑓 𝑋 =0
𝑌:𝑓 𝑌 =1

15. Information-Theoretic Lower Bound on Set Disjointness

16. Some Basic Concepts from Info Theory
Entropy of a random variable:
𝐻 𝑋 =− 𝑥 Pr 𝑋=𝑥 log Pr 𝑋=𝑥
Important properties:
𝐻 𝑋 ≥0
𝐻 𝑋 =0 ⇒ 𝑋 is deterministic
𝐻(𝑋) = expected # bits needed to encode 𝑋

17. Some Basic Concepts from Info Theory
Conditional entropy: 𝐻 𝑋 𝑌 = 𝐸 𝑦 𝐻( 𝑋 |𝑌=𝑦 )
Important properties:
𝐻 𝑋|𝑌 ≤𝐻(𝑋)
𝐻 𝑋|𝑌 =𝐻 𝑋 ⇒ 𝑋,𝑌 are independent
Example:
𝑋,𝑍∼Bernoulli 𝐻 𝑋 =− 1 2 log − 1 2 log =1
If 𝑍=0 then 𝑌=𝑋, if 𝑍=1 then 𝑌=1−𝑋
𝐻 𝑋 𝑌,𝑍 = 1 2 𝐻 𝑋|𝑋 𝐻 𝑋|1−𝑋 =0
𝐻 𝑋 𝑌 =1

18. Some Basic Concepts from Info Theory
Mutual information:
𝐼 𝑋;𝑌 =𝐻 𝑋 −𝐻 𝑋 𝑌 =𝐻 𝑌 −𝐻(𝑌|𝑋)
Conditional mutual information:
𝐼 𝑋;𝑌|𝑍 =𝐻 𝑋|𝑍 −𝐻 𝑋 𝑌,𝑍 =𝐻 𝑌|𝑍 −𝐻(𝑌|𝑋,𝑍)
Important properties:
𝐼 𝑋;𝑌 ≥0
𝐼 𝑋;𝑌 =0 ⇒ 𝑋,𝑌 are independent

19. Some Basic Concepts from Info Theory
Chain rule for mutual information:
𝐼 𝑋 1 , 𝑋 2 ;𝑌 =𝐼 𝑋 1 ;𝑌 +𝐼 𝑋 2 ;𝑌 𝑋 1
More generally,
𝐼 𝑋 1 ,…, 𝑋 𝑘 ;𝑌 = 𝑖=1 𝑘 𝐼 𝑋 𝑖 ;𝑌 𝑋 1 ,…, 𝑋 𝑖−1

20. Information Cost of Protocols
Fix an input distribution 𝜇 on 𝑋,𝑌
Given a protocol Π, let Π also denote the distribution of Π’s transcript
Information cost of Π:
𝐼𝐶 Π =𝐼 Π;𝑌 𝑋 +𝐼 Π;𝑋 𝑌
Information cost of a function 𝑓:
𝐼 𝐶 𝜖 𝑓 = inf Π solves 𝑓 𝑤/error≤𝜖 𝐼𝐶 Π

21. Information Cost of Protocols
Important property: 𝐼𝐶 Π ≤|Π|
Proof: by induction. Let Π= Π 1 … Π 𝑡 .
∀𝑟≤𝑡 : 𝐼 Π ≤𝑟 ;𝑌 𝑋 +𝐼 Π ≤𝑟 ;𝑋 𝑌 ≤𝑟.
𝐼 Π ≤𝑟 ;𝑌 𝑋 +𝐼 Π ≤𝑟 ;𝑋 𝑌 what we know after r rounds
=𝐼 Π <𝑟 ;𝑌 𝑋 +𝐼 Π <𝑟 ;𝑋 𝑌 what we knew after r-1 rounds
+ 𝐼 Π 𝑟 ;Y X, Π <𝑟 +𝐼 Π 𝑟 ;X Y, Π <𝑟
what we learn in round r, given what we already know

22. Information vs. Communication
Want: 𝐼 Π 𝑟 ;Y X, Π <𝑟 +𝐼 Π 𝑟 ;X Y, Π <𝑟 ≤1.
Suppose Π 𝑟 is sent by Alice.
What does Alice learn?
Π 𝑟 is a function of Π <𝑟 and 𝑋, so 𝐼 Π 𝑟 ;Y X, Π <𝑟 =0.
What does Bob learn?
𝐼 Π 𝑟 ;Y X, Π <𝑟 ≤ Π 𝑟 =1.

23. Information vs. Communication
Important property: 𝐼𝐶 Π ≤|Π|
Lower bound on information cost ⇒ lower bound on communication complexity
In fact, IC lower bounds are the most powerful technique we know

24. Information Complexity of Disj.
Disjointness: is 𝑋∩𝑌=∅ ?
Disj 𝑋,𝑌 = 𝑖=1 𝑛 𝑋 𝑖 ∧ 𝑌 𝑖
Strategy: for some “hard distribution” 𝜇,
Direct sum: 𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And)
Prove that 𝐼 𝐶 𝜇 And ≥Ω(1).

25. Hard Distribution for Disjointness
For each coordinate 𝑖∈ 𝑛 :
𝑋 𝑖 =0
𝑋 𝑖 =1
𝑌 𝑖 =0
1/3
1/3
𝑌 𝑖 =1
1/3

26. 𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Let Π be a protocol for Disj on 𝑋,𝑌∈ 0,1 𝑛
Construct Π ′ for And as follows:
Alice and Bob get inputs 𝑈,𝑉∈ 0,1
Choose a random coordinate 𝑖∈ 𝑛 , set 𝑋 𝑖 =𝑈, 𝑌 𝑖 =𝑉
Sample 𝑋 −𝑖 , 𝑌 −𝑖 and run Π
For each 𝑗≠𝑖, 𝑋 𝑗 ∧ 𝑌 𝑗 =0 ⇒ Disj 𝑋,𝑌 =And 𝑋 𝑖 , 𝑌 𝑖 𝑋 𝑌 𝑈 𝑉

27. 𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Let Π be a protocol for Disj on 𝑋,𝑌∈ 0,1 𝑛
Construct Π ′ for And as follows:
Alice and Bob get inputs 𝑈,𝑉∈ 0,1
Choose a random coordinate 𝑖∈ 𝑛 , set 𝑋 𝑖 =𝑈, 𝑌 𝑖 =𝑉
Bad idea: publicly sample 𝑋 −𝑖 , 𝑌 −𝑖 𝑋
Suppose in Π, Alice sends 𝑋 1 ⊕…⊕ 𝑋 𝑛 .
In Π, Bob learns one bit ⇒ in Π ′ he should learn 1/𝑛 bit
But if 𝑋 −𝑖 is public Bob learns 1 bit about 𝑈! 𝑈 𝑌 𝑉

28. 𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Let Π be a protocol for Disj on 𝑋,𝑌∈ 0,1 𝑛
Construct Π ′ for And as follows:
Alice and Bob get inputs 𝑈,𝑉∈ 0,1
Choose a random coordinate 𝑖∈ 𝑛 , set 𝑋 𝑖 =𝑈, 𝑌 𝑖 =𝑉
Another bad idea: publicly sample 𝑋 −𝑖 , Bob privately samples 𝑌 −𝑖 given 𝑋 −𝑖
But the players can’t sample 𝑋 −𝑖 , 𝑌 −𝑖 independently…

29. 𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Let Π be a protocol for Disj on 𝑋,𝑌∈ 0,1 𝑛
Construct Π ′ for And as follows:
Alice and Bob get inputs 𝑈,𝑉∈ 0,1
Choose a random coordinate 𝑖∈ 𝑛 , set 𝑋 𝑖 =𝑈, 𝑌 𝑖 =𝑉
Publicly sample 𝑋 1 ,…, 𝑋 𝑖−1
Privately sample 𝑋 (𝑖+1) ,…, 𝑋 𝑛 𝑋 𝑈
Privately sample 𝑌 1 ,…, 𝑌 𝑖−1
Publicly sample 𝑌 𝑖+1 ,…, 𝑌 𝑛 𝑌 𝑉

30. Direct Sum Theorem Transcript of Π ′ =𝑖, 𝑋 <𝑖 , 𝑌 >𝑖 ,Π
Need to show:
𝐼 𝜇 Π ′ ;𝑉 𝑈 + 𝐼 𝜇 Π ′ ;𝑈 𝑉 ≤ 𝐼 𝜇 𝑛 Π;𝑌 𝑋 + 𝐼 𝜇 𝑛 Π;𝑋 𝑌 /𝑛
𝐼 𝜇 Π ′ ;𝑉 𝑈 = 𝐼 𝜇 𝑛 𝑖, 𝑋 <𝑖 , 𝑌 >𝑖 ,Π; 𝑌 𝑖 𝑋 𝑖 = 𝐼 𝜇 𝑛 Π; 𝑌 𝑖 𝑋 ≤𝑖 , 𝑌 >𝑖 ,𝑖 + 𝐼 𝜇 𝑛 𝑖, 𝑋 <𝑖 , 𝑌 >𝑖 ; 𝑌 𝑖 𝑋 𝑖 ≤𝐼 Π, 𝑋 >𝑖 ; 𝑌 𝑖 𝑋 ≤𝑖 , 𝑌 >𝑖 ,𝑖 =𝐼 𝑋 >𝑖 ; 𝑌 𝑖 𝑋 ≤𝑖 , 𝑌 >𝑖 ,𝑖 +𝐼 Π; 𝑌 𝑖 𝑋, 𝑌 >𝑖 ,𝑖 =(1/𝑛) 𝑖=1 𝑛 𝐼 Π; 𝑌 𝑖 𝑋, 𝑌 >𝑖 =𝐼(Π;𝑌│𝑋)/𝑛.

31. Information Complexity of Disj.
Disjointness: is 𝑋∩𝑌=∅ ?
Disj 𝑋,𝑌 = 𝑖=1 𝑛 𝑋 𝑖 ∧ 𝑌 𝑖
Strategy: for some “hard distribution” 𝜇,
Direct sum: 𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And)
Prove that 𝐼 𝐶 𝜇 And ≥Ω(1). 

32. Hardness of And 𝐼𝐶 And =𝐼 Π;𝑌 𝑋 + 𝐼 Π;𝑋 𝑌 ≥Ω 1
2 3 𝐼 Π;𝑌 𝑋= 𝐼 Π;𝑌 𝑋=1
+ 2 3 𝐼 Π;𝑋 𝑌= 𝐼 Π;𝑋 𝑌=1 11 01 00 10
1/3
transcript on 11 should be
“very different” =0 =0

33. Hellinger Distance ℎ 2 𝑃,𝑄 =1− 𝜔 𝑃 𝜔 𝑄 𝜔 Examples:
ℎ 2 𝑃,𝑄 =1− 𝜔 𝑃 𝜔 𝑄 𝜔
Examples:
ℎ 2 𝑃,𝑃 =1− 𝜔 𝑃 𝜔 2 =1− 𝜔 𝑃 𝜔 =0
If 𝑃,𝑄 have disjoint support, ℎ 𝑃,𝑄 =1

34. Hellinger Distance Hellinger distance is a metric
ℎ 𝑃,𝑄 ≥0, with equality iff 𝑃=𝑄
ℎ 𝑃,𝑄 =ℎ 𝑄,𝑃
Triangle inequality: ℎ 𝑃,𝑄 ≤ℎ 𝑃,𝑅 +ℎ 𝑅,𝑄 𝑃 𝑄 𝑅

35. Hellinger Distance If for some 𝜔 we have 𝑃 𝜔 −𝑄 𝜔 =𝛿, then
ℎ 2 𝑃,𝑄 ≥ 𝛿 2 11 01 00 10 ℎ≥

36. Hellinger Distance vs. Mutual Info
Let 𝑃 0 , 𝑃 1 be two distributions
Select 𝑍 by choosing 𝐽∼Bernoulli , then drawing 𝑍∼ 𝑃 𝐽
Then 𝐼 𝑍;𝐽 ≥ ℎ 2 𝑃 0 , 𝑃 1 11 01 00 10
1/3
𝐼 Π;𝑌 𝑋=0 ≥ ℎ 2 Π 00 , Π 01
𝐼 Π;𝑋 𝑌=0 ≥ ℎ 2 Π 00 , Π 10

37. Hardness of And 01 11 00 10 1/3 1/3 1/3 Same for Bob until
Alice acts differently
1/3 01 ℎ≥ 11
Same for Alice until
Bob acts differently
1/3
1/3 00 10

38. “Cut-n-Paste Lemma” ℎ Π 00 , Π 11 =ℎ Π 01 , Π 10
ℎ Π 00 , Π 11 =ℎ Π 01 , Π 10
Recall: ℎ 2 Π 𝑋𝑌 , Π 𝑋 ′ 𝑌 ′ =1− 𝑡 Π 𝑋𝑌 𝑡 Π 𝑋 ′ 𝑌 ′ 𝑡
Enough to show: we can write
Π 𝑋𝑌 𝑡 = 𝑞 𝐴 𝑡,𝑋 ⋅ 𝑞 𝐵 𝑡,𝑌
ℎ 2 Π 00 , Π 11 =1− 𝑡 𝑞 𝐴 𝑡,0 𝑞 𝐵 𝑡,0 𝑞 𝐴 𝑡,1 𝑞 𝐵 𝑡,1 =1− 𝑡 𝑞 𝐴 𝑡,0 𝑞 𝐵 𝑡,1 𝑞 𝐴 𝑡,0 𝑞 𝐵 𝑡,1 = ℎ 2 Π 01 , Π 11

39. “Cut-n-Paste Lemma” We can write Π 𝑋𝑌 𝑡 = 𝑞 𝐴 𝑡,𝑋 ⋅ 𝑞 𝐵 𝑡,𝑌 Proof:
Π 𝑋𝑌 𝑡 = 𝑞 𝐴 𝑡,𝑋 ⋅ 𝑞 𝐵 𝑡,𝑌
Proof:
Π induces a distribution on “partial transcripts” of each length 𝑘:
Π 𝑋𝑌 𝑘 𝑡 = probability that first 𝑘 bits are 𝑡
By induction: Π 𝑋𝑌 𝑘 𝑡 = 𝑞 𝐴 𝑘 𝑡,𝑋 ⋅ 𝑞 𝐵 𝑘 𝑡,𝑌
Base case: Π 𝑋𝑌 𝑘 𝜖 =1
Set 𝑞 𝐴 𝑘 𝜖,𝑋 = 𝑞 𝐵 𝑘 𝜖,𝑌 =1

40. “Cut-n-Paste Lemma”
Step: Π 𝑋𝑌 𝑘+1 𝑡 = Π 𝑋𝑌 𝑘 𝑡 ≤𝑘 ⋅ Pr next bit= 𝑡 𝑘+1
Suppose after 𝑡 ≤𝑘 it is Alice’s turn to speak
What Alice says depends on:
Her input
Her private randomness
The transcript so far, 𝑡 ≤𝑘
So Pr next bit= 𝑡 𝑘+1 =𝑓 𝑡 ≤𝑘 ,𝑋, 𝑡 𝑘+1 =𝑓 𝑡,𝑋
Set 𝑞 𝐴 𝑘+1 𝑡,𝑋 = 𝑞 𝐴 𝑘 𝑡 ≤𝑘 ,𝑋 ⋅𝑓 𝑡,𝑋 ,
𝑞 𝐵 𝑘+1 𝑡,𝑋 = 𝑞 𝐵 𝑘 𝑡 ≤𝑘 ,𝑋

41. Hardness of And
1/3
𝐼𝐶 And =𝐼 Π;𝑌 𝑋 + 𝐼 Π;𝑋 𝑌 = 𝐼 Π;𝑌 𝑋= 𝐼 Π;𝑋 𝑌=0
≥const⋅ ℎ 2 Π 00 , Π ℎ 2 Π 00 , Π 10
≥const′⋅ ℎ Π 00 , Π 01 +ℎ Π 00 , Π
≥const′⋅ ℎ 2 Π 01 , Π 10 =const′⋅ ℎ 2 Π 00 , Π 11 ≥Ω 1 01 ℎ≥ 11
1/3
1/3 00 10

42. Multi-Player Communication Complexity

43. The Coordinator Model
𝑓 𝑋 1 ,…, 𝑋 𝑘 = ?
𝑘 sites
𝑛 bits
𝑋 1
𝑋 2
𝑋 𝑘 …

44. Multi-Party Set Disjointness
Input: 𝑋 1 ,…, 𝑋 𝑘 ⊆ 𝑛
Output: is ⋂ 𝑋 𝑖 =∅?
Braverman,Ellen,O.,Pitassi,Vaikuntanathan’13:
lower bound of Ω 𝑛𝑘 bits

45. Reduction from Disj to graph connectivity
Given 𝑋 1 ,…, 𝑋 𝑘 , we want to
Choose vertices 𝑉
Design inputs 𝐸 1 ,…, 𝐸 𝑘 such that 𝐺 𝑉, 𝐸 1 ∪…∪ 𝐸 𝑘 𝑉, 𝐸 1 ∪…∪ 𝐸 𝑘 is connected iff ⋂ 𝑋 𝑖 =∅

46. Reduction from Disj to graph connectivity
𝑝 1
𝑝 2
𝑝 𝑘
(Players)
(Elements) 1 2 3 4 5 6
𝑋 𝑖
𝑛 ∖ ⋃ 𝑋 𝑖
input graph connected ⇔
⋃ 𝑋 𝑖 ≠ 𝑛
⋂ 𝑋 𝑖 ≠∅

47. Other Stuff
Distributed computing

48. Other Stuff Compressing down to information cost
Number-on-forehead lower bounds
Open questions in communication complexity