1. of 31 09/19/2011UIUC: Communication & Computation1 Communication & Computation A need for a new unifying theory Madhu Sudan Microsoft, New England

2. of 31 09/19/2011UIUC: Communication & Computation2 Theory of Computing Turing architecture Turing architecture FiniteState Control Control R/W UniversalMachine Encodings of other machines One machine to rule them all! von Neumann architecture von Neumann architecture CPU RAM

3. of 31 09/19/2011UIUC: Communication & Computation3 Theory of Communication Shannons architecture for communication over noisy channel Shannons architecture for communication over noisy channel Yields reliable communication Yields reliable communication (and storage (= communication across time)). (and storage (= communication across time)). Noisy Channel Encoder Decoder Y Ŷ m = E(m) D(Ŷ) = m?

4. of 31 09/19/2011UIUC: Communication & Computation4 Turing Shannon Turing Turing Assumes perfect storage Assumes perfect storage and perfect communication and perfect communication To get computation To get computation Shannon Shannon Assumes computation Assumes computation To get reliable storage + communication To get reliable storage + communication Chicken vs. Egg? Chicken vs. Egg? Fortunately both realized! Fortunately both realized! Encoder Decoder

5. of 31 09/19/2011UIUC: Communication & Computation5 1940s – 2000: Theories developed mostly independently. Theories developed mostly independently. Shannon abstraction (separating information theoretic properties of encoder/decoder from computational issues) – mostly successful. Shannon abstraction (separating information theoretic properties of encoder/decoder from computational issues) – mostly successful. Turing assumption (reliable storage/communication) – mostly realistic. Turing assumption (reliable storage/communication) – mostly realistic.

6. of 31 09/19/2011UIUC: Communication & Computation6 Modern Theory (of Comm. & Comp.) Network (society?) of communicating computers Network (society?) of communicating computers Diversity of Diversity of Capability Capability Protocols Protocols Objectives Objectives Concerns Concerns Alice Bob Charlie Dick Fred Eve

7. of 31 09/19/2011UIUC: Communication & Computation7 Modern Challenges (to communication) Nature of communication is more complex. Nature of communication is more complex. Channels are more complex (composed of many smaller, potentially clever sub-channels) Channels are more complex (composed of many smaller, potentially clever sub-channels) Alters nature of errors Alters nature of errors Scale of information being stored/communicated is much larger. Scale of information being stored/communicated is much larger. Does scaling enhance reliability or decrease it? Does scaling enhance reliability or decrease it? The Meaning of Information The Meaning of Information Entities constantly evolving. Can they preserve meaning of information? Entities constantly evolving. Can they preserve meaning of information?

8. of 31 09/19/2011UIUC: Communication & Computation8 Part I: Modeling errors

9. of 31 09/19/2011UIUC: Communication & Computation9 Shannon (1948) vs. Hamming (1950)

10. of 31 09/19/2011UIUC: Communication & Computation10 Which is the right model? 60 years of wisdom … 60 years of wisdom … Error model can be fine-tuned … Error model can be fine-tuned … Fresh combinatorics, algorithms, probabilistic models can be built … Fresh combinatorics, algorithms, probabilistic models can be built … … to fit Shannon Model. … to fit Shannon Model. An alternative – List-Decoding [Elias 56]! An alternative – List-Decoding [Elias 56]! allowed to produce list {m 1,…,m l } allowed to produce list {m 1,…,m l } Successful if {m 1,…,m l } contains m. Successful if {m 1,…,m l } contains m. 60 years of wisdom this is good enough! 60 years of wisdom this is good enough! [70s]: Corrects as many adversarial errors as random ones! [70s]: Corrects as many adversarial errors as random ones! Decoder

11. of 31 09/19/2011UIUC: Communication & Computation11 Challenges in List-decoding! Algorithms? Algorithms? Correcting a few errors is already challenging! Correcting a few errors is already challenging! Can we really correct 70% errors? 99% errors? Can we really correct 70% errors? 99% errors? When an adversary injects them? When an adversary injects them? Note: More errors than data! Note: More errors than data! Till 1988 … no list-decoding algorithms. Till 1988 … no list-decoding algorithms. [Goldreich-Levin 88] – Raised question [Goldreich-Levin 88] – Raised question Gave non-trivial algorithm (for weak code). Gave non-trivial algorithm (for weak code). Gave cryptographic applications. Gave cryptographic applications.

12. of 31 09/19/2011UIUC: Communication & Computation12 Algorithms for List-decoding [S. 96], [Guruswami + S. 98]: [S. 96], [Guruswami + S. 98]: List-decoding of Reed-Solomon codes. List-decoding of Reed-Solomon codes. Corrected p-fraction error with linear rate. Corrected p-fraction error with linear rate. [98 – 06] Many algorithmic innovations … [98 – 06] Many algorithmic innovations … [ Guruswami, Shokrollahi, Koetter-Vardy, Indyk ] [ Guruswami, Shokrollahi, Koetter-Vardy, Indyk ] [Parvaresh-Vardy 05 + Guruswami-Rudra 06] [Parvaresh-Vardy 05 + Guruswami-Rudra 06] List-decoding of new variant of Reed-Solomon codes. List-decoding of new variant of Reed-Solomon codes. Correct p-fraction error with optimal rate. Correct p-fraction error with optimal rate.

13. of 31 09/19/2011UIUC: Communication & Computation13 Reed-Solomon List-Decoding Problem Given: Given: Parameters: n,k,t Parameters: n,k,t Points: (x 1,y 1 ),…,(x n,y n ) in the plane Points: (x 1,y 1 ),…,(x n,y n ) in the plane (over finite fields, actually) Find: Find: All degree k polynomials that pass through t of the n points. All degree k polynomials that pass through t of the n points. i.e., f such that deg(f) k deg(f) k |{i s.t. f(x i ) = y i }| t |{i s.t. f(x i ) = y i }| t

14. of 31 09/19/2011UIUC: Communication & Computation14 Decoding by Example + Picture [S. 96] n=14;k=1;t=5 Algorithm Idea: Find algebraic explanation Find algebraic explanation of all points. of all points. Stare at it! Stare at it! Factor the polynomial!

15. of 31 09/19/2011UIUC: Communication & Computation15 Decoding Algorithm Fact: There is always a degree 2n polynomial thru n points Fact: There is always a degree 2n polynomial thru n points Can be found in polynomial time (solving linear system). Can be found in polynomial time (solving linear system). [80s]: Polynomials can be factored in polynomial time [Grigoriev, Kaltofen, Lenstra] [80s]: Polynomials can be factored in polynomial time [Grigoriev, Kaltofen, Lenstra] Leads to (simple, efficient) list-decoding correcting p fraction errors for p 1 Leads to (simple, efficient) list-decoding correcting p fraction errors for p 1

16. of 31 09/19/2011UIUC: Communication & Computation16 Conclusion More errors (than data!) can be dealt with … More errors (than data!) can be dealt with … More computational power leads to better error-correction. More computational power leads to better error-correction. Theoretical Challenge: List-decoding on binary channel (with optimal (Shannon) rates). Theoretical Challenge: List-decoding on binary channel (with optimal (Shannon) rates). Important to clarify the right model. Important to clarify the right model.

17. of 31 09/19/2011UIUC: Communication & Computation17 Part II: Massive Data; Local Algorithms

18. of 31 09/19/2011UIUC: Communication & Computation18 Reliability vs. Size of Data Q: How reliably can one store data as the amount of data increases? Q: How reliably can one store data as the amount of data increases? [Shannon]: Can store information at close to optimal rate, and prob. decoding error drops exponentially with length of data. [Shannon]: Can store information at close to optimal rate, and prob. decoding error drops exponentially with length of data. Surprising at the time? Surprising at the time? Decoding time grows with length of data Decoding time grows with length of data Exponentially in Shannon Exponentially in Shannon Subsequently polynomial, even linear. Subsequently polynomial, even linear. Is the bad news necessary? Is the bad news necessary?

19. of 31 09/19/2011UIUC: Communication & Computation19 Sublinear time algorithmics Algorithms dont always need to run in linear time (!), provided … Algorithms dont always need to run in linear time (!), provided … They have random access to input, They have random access to input, Output is short (relative to input), Output is short (relative to input), Answers dont have usual, exact, guarantee! Answers dont have usual, exact, guarantee! Applies, in particular, to Applies, in particular, to Given CD, test to see if it has (too many) errors? [Locally Testable Codes] Given CD, test to see if it has (too many) errors? [Locally Testable Codes] Given CD, recover particular block. [Locally Decodable Codes] Given CD, recover particular block. [Locally Decodable Codes] Decoder

20. of 31 09/19/2011UIUC: Communication & Computation20 Progress [1990-2008]

21. of 31 09/19/2011UIUC: Communication & Computation21 Challenges ahead Technical challenges Technical challenges Linear rate testability? Linear rate testability? Polynomial rate decodability? Polynomial rate decodability? Logarithmic time decodability with linear rate? Logarithmic time decodability with linear rate? Bigger Challenge Bigger Challenge What is the model for the future storage of information? What is the model for the future storage of information? How are we going to cope with increasing drive to digital information? How are we going to cope with increasing drive to digital information?

22. of 31 09/19/2011UIUC: Communication & Computation22 Part III: The Meaning of Information

23. of 31 09/19/2011UIUC: Communication & Computation23 The Meaning of Bits Is this perfect communication? Is this perfect communication? What if Alice is trying to send instructions? What if Alice is trying to send instructions? In other words … an algorithm In other words … an algorithm Does Bob understand the correct algorithm? Does Bob understand the correct algorithm? What if Alice and Bob speak in different (programming) languages? What if Alice and Bob speak in different (programming) languages? Channel Alice Bob Bob 01001011 01001011 Bob Freeze!

24. of 31 09/19/2011UIUC: Communication & Computation24 Motivation: Better Computing Networked computers use common languages: Networked computers use common languages: Interaction between computers (getting your computer onto internet). Interaction between computers (getting your computer onto internet). Interaction between pieces of software. Interaction between pieces of software. Interaction between software, data and devices. Interaction between software, data and devices. Getting two computing environments to talk to each other is getting problematic: Getting two computing environments to talk to each other is getting problematic: time consuming, unreliable, insecure. time consuming, unreliable, insecure. Can we communicate more like humans do? Can we communicate more like humans do?

25. of 31 09/19/2011UIUC: Communication & Computation25 Some modelling Say, Alice and Bob know different programming languages. Alice wishes to send an algorithm A to Bob. Say, Alice and Bob know different programming languages. Alice wishes to send an algorithm A to Bob. Bad News: Cant be done Bad News: Cant be done For every Bob, there exist algorithms A and A, and Alices, Alice and Alice, such that Alice sending A is indistinguishable (to Bob) from Alice sending A For every Bob, there exist algorithms A and A, and Alices, Alice and Alice, such that Alice sending A is indistinguishable (to Bob) from Alice sending A Good News: Need not be done. Good News: Need not be done. From Bobs perspective, if A and A are indistinguishable, then they are equally useful to him. From Bobs perspective, if A and A are indistinguishable, then they are equally useful to him. Question: What should be communicated? Why? Question: What should be communicated? Why?

26. of 31 09/19/2011UIUC: Communication & Computation26 Progress Report I: Computational Goal Bob (weak computer) communicating with Alice (strong computer) to solve hard problem. Bob (weak computer) communicating with Alice (strong computer) to solve hard problem. Alice Helpful if she can help some (weak) Bob solve the problem. Alice Helpful if she can help some (weak) Bob solve the problem. Theorem [Juba & S., STOC 08]: Bob can use Alices help to solve his problem iff problem is verifiable (for every Helpful Alice). Theorem [Juba & S., STOC 08]: Bob can use Alices help to solve his problem iff problem is verifiable (for every Helpful Alice). Misunderstanding = Mistrust Misunderstanding = Mistrust

27. of 31 Progress Report II: General Goals [Goldreich,Juba,S. – ECCC 2010] [Goldreich,Juba,S. – ECCC 2010] Not every goal is computational. Does the [JS] result extend to other settings? Not every goal is computational. Does the [JS] result extend to other settings? First: What do general goals look like? First: What do general goals look like? Non-trivial to define (in language- independent form). Non-trivial to define (in language- independent form). But can be done. But can be done. Second: Results extend provided goals are verifiable, and players are helpful. Second: Results extend provided goals are verifiable, and players are helpful. Definitions can be extended. Definitions can be extended. 09/19/2011UIUC: Communication & Computation27

28. of 31 Progress Report III: Efficiency? One of the main contributions of [JS08] was a measure of efficiency of achieving understanding. One of the main contributions of [JS08] was a measure of efficiency of achieving understanding. Unfortunately protocol in [JS08] could be inefficient. Unfortunately protocol in [JS08] could be inefficient. [JS08] proves such inefficiency is inherent. [JS08] proves such inefficiency is inherent. [JS – ICS 2011]: [JS – ICS 2011]: New measure of efficiency: New measure of efficiency: Takes into account compatibility of user with server; and broadmindedness of server and shows understanding can be achieved efficiently if these parameters are small. Takes into account compatibility of user with server; and broadmindedness of server and shows understanding can be achieved efficiently if these parameters are small. 09/19/2011UIUC: Communication & Computation28

29. of 31 Main Contribution: A new model Classical Shannon Model March 1, 2011Semantic Communication @ UCLA29 A B Channel B2B2B2B2 AkAkAkAk A3A3A3A3 A2A2A2A2 A1A1A1A1 B1B1B1B1 B3B3B3B3 BjBjBjBj Semantic Communication Model New Class of Problems New challenges Needs more attention! [Kalai,Khanna,J.,S. – ICS 2011] Compression in this setting: Leads to ambiguous, redundant compression

30. of 31 09/19/2011UIUC: Communication & Computation30 Concluding More, complex, errors can be dealt with, thanks to improved computational abilities. More, complex, errors can be dealt with, thanks to improved computational abilities. Need to build/study tradeoffs between global reliability and local computation. Need to build/study tradeoffs between global reliability and local computation. Meaning of information needs to be preserved! Meaning of information needs to be preserved! Need to merge computation and communication more tightly! Need to merge computation and communication more tightly!

31. of 31 09/19/2011UIUC: Communication & Computation31 Thank You!

32. of 31 09/19/2011UIUC: Communication & Computation32 Ongoing Work [Juba & S.] Assertion/Assumption: Communication happens when communicators have (explicit) goals. Assertion/Assumption: Communication happens when communicators have (explicit) goals. Goals: Goals: (Remote) Control: (Remote) Control: Actuating some change in environment Actuating some change in environment Example Example Printing on printer Printing on printer Buying from Amazon Buying from Amazon Intellectual: Intellectual: Learn something from (about?) environment Learn something from (about?) environment Example Example This lecture (whats in it for you? For me?) This lecture (whats in it for you? For me?)

33. of 31 09/19/2011UIUC: Communication & Computation33 Example Problems Bob wishes to … Bob wishes to … … solve undecidable problem (virus-detection) … solve undecidable problem (virus-detection) Not verifiable; so solves problems incorrectly for some Alices. Not verifiable; so solves problems incorrectly for some Alices. Hence does not learn her language. Hence does not learn her language. … break cryptosystem … break cryptosystem Verifiable; so Bob can use her help. Verifiable; so Bob can use her help. Must be learning her language! Must be learning her language! … Sort integers … Sort integers Verifiable; so Bob does solve her problem. Verifiable; so Bob does solve her problem. Trivial: Might still not be learning her language. Trivial: Might still not be learning her language.

34. of 31 09/19/2011UIUC: Communication & Computation34 Generalizing Generic Goals Generic Goals Typical goals: Wishful Typical goals: Wishful Is Alice a human? or computer? Is Alice a human? or computer? Does she understand me? Does she understand me? Will she listen to me (and do what I say)? Will she listen to me (and do what I say)? Achievable goals: Verifiable Achievable goals: Verifiable Bob should be able to test achievement by looking at his input/output exchanges with Alice. Bob should be able to test achievement by looking at his input/output exchanges with Alice. Question: Which wishful goals are verifiable? Question: Which wishful goals are verifiable?