1. On the Capacity of Information Networks Nick Harvey Collaborators: Micah Adler (UMass), Kamal Jain (Microsoft), Bobby Kleinberg (MIT/Berkeley/Cornell), and April Lehman (MIT/Google)

2. What is the capacity of a network?

3. s1s1 s2s2 Send items from s 1  t 1 and s 2  t 2 Problem: no disjoint paths bottleneck edge What is the capacity of a network? t2t2 t1t1

4. b1⊕b2b1⊕b2 An Information Network b1b1 b2b2 s1s1 s2s2 t2t2 t1t1 If sending information, we can do better Send xor b 1 ⊕ b 2 on bottleneck edge

5. Moral of Butterfly Network Flow Capacity ≠ Information Flow Capacity

6. Network Coding New approach for information flow problems  Blend of combinatorial optimization, information theory  Multicast, k-Pairs k-Pairs problems: Network coding when each commodity has one sink  Analogous to multicommodity flow Definitions for cyclic networks are subtle

7. Multicommodity Flow Efficient algorithms for computing maximum concurrent (fractional) flow. Connected with metric embeddings via LP duality. Approximate max-flow min-cut theorems. Network Coding Computing the max concurrent coding rate may be:  Undecidable  Decidable in poly-time No adequate duality theory. No cut-based parameter is known to give sublinear approximation in digraphs. Directed and undirected problems behave quite differently

8. Coding rate can be much larger than flow rate! Butterfly:  Coding rate = 1  Flow rate = ½ Thm [HKL’04,LL’04]:  graphs G(V,E) where Coding Rate = Ω( flow rate ∙ |V| ) Directed k-pairs s1s1 s2s2 t2t2 t1t1 Thm:  graphs G(V,E) where Coding Rate = Ω( flow rate ∙ |E| )  And this is optimal  Recurse on butterfly construction

9. Coding rate can be much larger than flow rate! …and much larger than the sparsity (same example) Directed k-pairs Flow Rate  Sparsity < Coding Rate in some graphs

10. No known undirected instance where coding rate ≠ max flow rate! (The undirected k-pairs conjecture) Undirected k-pairs Flow Rate  Coding Rate  Sparsity Pigeonhole principle argument Gap can be Ω(log n) when G is an expander

11. Undirected k-Pairs Conjecture Flow Rate Sparsity Coding Rate <= ?? =< Undirected k-pairs conjecture Unknown until this work

12. Okamura-Seymour Graph s1s1 t1t1 s2s2 t2t2 s3s3 t3t3 s4s4 t4t4 Every cut has enough capacity to carry all commodities separated by the cut Cut

13. Okamura-Seymour Max-Flow s1s1 t1t1 s2s2 t2t2 s3s3 t3t3 s4s4 t4t4 Flow Rate = 3/4 s i is 2 hops from t i. At flow rate r, each commodity consumes  2r units of bandwidth in a graph with only 6 units of capacity.

14. The trouble with information flow… If an edge codes multiple commodities, how to charge for “consuming bandwidth”? We work around this obstacle and bound coding rate by 3/4. s1s1 t1t1 s2s2 t2t2 s3s3 t3t3 s4s4 t4t4 At flow rate r, each commodity consumes at least 2r units of bandwidth in a graph with only 6 units of capacity.

15. Definition: A e if for every coding solution, the messages sent on edges of A uniquely determine the message sent on e. Given A and e, how hard is it to determine whether A e? Is it even decidable? Theorem: There is an algorithm to compute whether A e in time O(k²m).  Based on a combinatorial characterization of informational dominance Informational Dominance i  i  i

16. What can we prove? s1 t3s1 t3 s2 t1s2 t1 s3 t2s3 t2 s4s4 t4t4 Combine Informational Dominance with Shannon inequalities for Entropy Flow rate = coding rate for “Special Bipartite Graphs”:  Bipartite  Every source is 2 hops away from its sink  Dual of flow LP is optimized by assigning length 1 to all edges Next: show that proving conjecture for all graphs is quite hard

17. k-pairs conjecture & I/O complexity I/O complexity model [AV’88] :  A large, slow external memory consisting of pages each containing p records  A fast internal memory that holds 2 pages  Basic I/O operation: read in two pages from external memory, write out one page

18. I/O Complexity of Matrix Transposition Matrix transposition: Given a p×p matrix of records in row-major order, write it out in column-major order. Obvious algorithm requires O(p²) ops. A better algorithm uses O(p log p) ops.

19. I/O Complexity of Matrix Transposition Matrix transposition: Given a p×p matrix of records in row-major order, write it out in column-major order. Obvious algorithm requires O(p²) ops. A better algorithm uses O(p log p) ops. s1s1 s2s2

20. I/O Complexity of Matrix Transposition Matrix transposition: Given a p x p matrix of records in row-major order, write it out in column-major order. Obvious algorithm requires O(p²) ops. A better algorithm uses O(p log p) ops. s1s1 s2s2 s3s3 s4s4

21. I/O Complexity of Matrix Transposition Matrix transposition: Given a p x p matrix of records in row-major order, write it out in column-major order. Obvious algorithm requires O(p²) ops. A better algorithm uses O(p log p) ops. s1s1 s2s2 s3s3 s4s4 t1t1 t3t3

22. I/O Complexity of Matrix Transposition Matrix transposition: Given a p x p matrix of records in row-major order, write it out in column-major order. Obvious algorithm requires O(p²) ops. A better algorithm uses O(p log p) ops. s1s1 s2s2 s3s3 s4s4 t1t1 t3t3 t2t2 t4t4

23. Matching Lower Bound Theorem: (Floyd ’72, AV’88) A matrix transposition algorithm using only read and write operations (no arithmetic on values) must perform Ω(p log p) I/O operations. s1s1 s2s2 s3s3 s4s4 t1t1 t3t3 t2t2 t4t4

24. Ω(p log p) Lower Bound Proof: Let N ij denote the number of ops in which record (i,j) is written. For all j, Σ i N ij ≥ p log p. Hence Σ ij N ij ≥ p² log p. Each I/O writes only p records. QED. s1s1 s2s2 s3s3 s4s4 t1t1 t3t3 t2t2 t4t4

25. The k-pairs conjecture and I/O complexity Definition: An oblivious algorithm is one whose pattern of read/write operations does not depend on the input. Theorem: If there is an oblivious algorithm for matrix transposition using o(p log p) I/O ops, the undirected k-pairs conjecture is false. s1s1 s2s2 s3s3 s4s4 t1t1 t3t3 t2t2 t4t4

26. The k-pairs conjecture and I/O complexity Proof:  Represent the algorithm with a diagram as before.  Assume WLOG that each node has only two outgoing edges. s1s1 s2s2 s3s3 s4s4 t1t1 t3t3 t2t2 t4t4

27. The k-pairs conjecture and I/O complexity Proof:  Represent the algorithm with a diagram as before.  Assume WLOG that each node has only two outgoing edges.  Make all edges undirected, capacity p.  Create a commodity for each matrix entry. s1s1 s2s2 s3s3 s4s4 t1t1 t3t3 t2t2 t4t4

28. The k-pairs conjecture and I/O complexity Proof:  The algorithm itself is a network code of rate 1.  Assuming the k-pairs conjecture, there is a flow of rate 1.  Σ i,j d(s i,t j ) ≤ p |E(G)|.  Arguing as before, LHS is Ω(p² log p).  Hence |E(G)|=Ω(p log p). s1s1 s2s2 s3s3 s4s4 t1t1 t3t3 t2t2 t4t4

29. Other consequences for complexity The undirected k-pairs conjecture implies:  A Ω(p log p) lower bound for matrix transposition in the cell-probe model. [Same proof.]  A Ω(p² log p) lower bound for the running time of oblivious matrix transposition algorithms on a multi-tape Turing machine. [I/O model can emulate multi-tape Turing machines with a factor p speedup.]

30. Distance arguments Rate-1 flow solution implies Σ i d(s i,t i ) ≤ |E|  LP duality; directed or undirected Does rate-1 coding solution imply Σ i d(s i,t i ) ≤ |E|?  Undirected graphs: this is essentially the k-pairs conjecture!  Directed graphs: this is completely false

31. k commodities (s i,t i ) Distance d(s i,t i ) = O(log k)  i O(k) edges! Recursive construction s(1) s(2)s(3)s(4)s(5)s(6)s(7)s(8) t(1) t(2)t(3)t(4)t(5)t(6)t(7)t(8)

32. Recursive Construction s1s1 s2s2 t1t1 t2t2 G (1): Equivalent to: s1s1 s2s2 t1t1 t2t2 Edge capacity = 1 2 commodities 7 edges Distance = 3

33. Recursive Construction s1s1 s2s2 s3s3 s4s4 t1t1 t2t2 t3t3 t4t4 G (2): Start with two copies of G (1)

34. Recursive Construction s1s1 s2s2 s3s3 s4s4 t1t1 t2t2 t3t3 t4t4 G (2): Replace middle edges with copy of G (1)

35. Recursive Construction s1s1 s2s2 s3s3 s4s4 G (1) t1t1 t2t2 t3t3 t4t4 G (2): 4 commodities, 19 edges, Distance = 5

36. Recursive Construction G (n-1) G (n): # commodities = 2 n, |V| = O(2 n ), |E| = O(2 n ) Distance = 2n+1 s1s1 s2s2 t1t1 t2t2 s3s3 s4s4 t3t3 t4t4 s 2 n -1 s2ns2n t 2 n -1 t2nt2n

37. Summary Directed instances:  Coding rate >> flow rate Undirected instances:  Conjecture: Flow rate = Coding rate  Proof for special bip graphs  Tool: Informational Dominance  Proving conjecture solves Matrix Transposition Problem

38. Open Problems Computing the network coding rate in DAGs:  Recursively decidable?  How do you compute a o(n)-factor approximation? Undirected k-pairs conjecture:  Stronger complexity consequences?  Prove a Ω(log n) gap between sparsest cut and coding rate for some graphs  …or, find a fast matrix transposition algorithm.

39. Backup Slides

40. Optimality The graph G (n) proves: Thm [HKL’05]:  graphs G(V,E) where NCR = Ω( flow rate ∙ |E| ) G (n) is optimal: Thm [HKL’05]:  graph G(V,E), NCR/flow rate = O(min {|V|,|E|,k})

41. s1s1 s2s2 t2t2 t1t1 A does not dominate B Informational Dominance Def: A dominates B if information in A determines information in B in every network coding solution.

42. Informational Dominance Def: A dominates B if information in A determines information in B in every network coding solution. s1s1 s2s2 t2t2 t1t1 A dominates B Sufficient Condition: If no path from any source  B then A dominates B

43. Informational Dominance Example s1s1 s2s2 t1t1 t2t2 “Obviously” flow rate = NCR = 1  How to prove it? Markovicity?  No two edges disconnect t 1 and t 2 from both sources!

44. Informational Dominance Example s1s1 s2s2 t1t1 t2t2 Cut A Sufficient Condition: If no path from any source  B then A dominates B

45. Informational Dominance Example s1s1 s2s2 t1t1 t2t2 Our characterization implies that A dominates {t 1,t 2 }  H(A)  H(t 1,t 2 ) Cut A

46. Rate ¾ for Okamura-Seymour s1 t3s1 t3 s2 t1s2 t1 s3 t2s3 t2 s4s4 t4t4 s1s1 i s1s1 t3t3 s3s3

47. s1 t3s1 t3 s2 t1s2 t1 s3 t2s3 t2 s4s4 t4t4 i i i ++ ≥ ++

48. s1 t3s1 t3 s2 t1s2 t1 s3 t2s3 t2 s4s4 t4t4 i i i ++ ≥ ++

49. s1 t3s1 t3 s2 t1s2 t1 s3 t2s3 t2 s4s4 t4t4 i i i ++ ≥ ++ i

50. s1 t3s1 t3 s2 t1s2 t1 s3 t2s3 t2 s4s4 t4t4 ++ ≥ + ii ≥

51. s1 t3s1 t3 s2 t1s2 t1 s3 t2s3 t2 s4s4 t4t4 ++ ≥ + ≥ 3 H(source) + 6 H(undirected edge) ≥ 11 H(source)6 H(undirected edge) ≥ 8 H(source) ¾ ≥ RATE