1. of 27 August 6, 2015KAIST: Reliable Meaningful Communication1 Reliable Meaningful Communication Madhu Sudan Microsoft Research

2. of 27 This Talk Part I: Reliable Communication Part I: Reliable Communication Problem and History (briefly) Problem and History (briefly) Part II: Recovering when errors overwhelm Part II: Recovering when errors overwhelm Sample of my work in the area Sample of my work in the area Part III: Modern challenges Part III: Modern challenges Communicating amid uncertainty Communicating amid uncertainty August 6, 2015KAIST: Reliable Meaningful Communication2

3. of 27 Part I: Reliable Communication August 6, 2015KAIST: Reliable Meaningful Communication3

4. of 27 Reliable Communication? Problem from the 1940s: Advent of digital age. Problem from the 1940s: Advent of digital age. Communication media are always noisy Communication media are always noisy But digital information less tolerant to noise! But digital information less tolerant to noise! August 6, 2015KAIST: Reliable Meaningful Communication4 AliceAlice BobBob We are not ready We are now ready

5. of 27 Reliability by Repetition August 6, 2015KAIST: Reliable Meaningful Communication5

6. of 27 Shannon’s Theory [1948] August 6, 2015KAIST: Reliable Meaningful Communication6 AliceAlice BobBob Encoder Decoder

7. of 27 Shannon’s Theorem August 6, 2015KAIST: Reliable Meaningful Communication7

8. of 27 Shannon’s contributions Far-reaching architecture: Far-reaching architecture: Profound analysis: Profound analysis: First (?) use of probabilistic method. First (?) use of probabilistic method. Deep Mathematical Discoveries: Deep Mathematical Discoveries: Entropy, Information, Bit? Entropy, Information, Bit? August 6, 2015KAIST: Reliable Meaningful Communication8 AliceAlice BobBob Encoder Decoder

9. of 27 Challenges post-Shannon August 6, 2015KAIST: Reliable Meaningful Communication9

10. of 27 Progress 1950-2010 Profound contributions to the theory: Profound contributions to the theory: New coding schemes, decoding algorithms, analysis techniques … New coding schemes, decoding algorithms, analysis techniques … Major fields of research: Major fields of research: Communication theory, Coding Theory, Information Theory. Communication theory, Coding Theory, Information Theory. Sustained Digital Revolution: Sustained Digital Revolution: Widespread conversion of everything to “bits” Widespread conversion of everything to “bits” Every storage and communication technology relies/builds on the theory. Every storage and communication technology relies/builds on the theory. “Marriage made in heaven” [Jim Massey] “Marriage made in heaven” [Jim Massey] August 6, 2015KAIST: Reliable Meaningful Communication10

11. of 27 Part II: Overwhelming #errors August 6, 2015KAIST: Reliable Meaningful Communication11

12. of 27 Explicit Codes: Reed-Solomon Code August 6, 2015KAIST: Reliable Meaningful Communication12

13. of 27 Overwhelming Errors? List Decoding August 6, 2015KAIST: Reliable Meaningful Communication13

14. of 27 Reed-Solomon List-Decoding Problem August 6, 2015KAIST: Reliable Meaningful Communication14

15. of 27 Decoding by example + picture [S’96] Algorithm idea: Find algebraic explanation Find algebraic explanation of all points. of all points. Stare at the solution Stare at the solution (factor the polynomial) (factor the polynomial) Ssss Ssss August 6, 2015KAIST: Reliable Meaningful Communication15

16. of 27 Decoding by example + picture [S’96] Algorithm idea: Find algebraic explanation Find algebraic explanation of all points. of all points. Stare at the solution Stare at the solution (factor the polynomial) (factor the polynomial) Ssss Ssss August 6, 2015KAIST: Reliable Meaningful Communication16

17. of 27 August 6, 2015KAIST: Reliable Meaningful Communication17 Decoding Algorithm

18. of 27 Part III: Modern Challenges Communication Amid Uncertainty? August 6, 2015KAIST: Reliable Meaningful Communication18

19. of 27 New Kind of Uncertainty Uncertainty always has been a central problem: Uncertainty always has been a central problem: But usually focusses on uncertainty introduced by the channel But usually focusses on uncertainty introduced by the channel Rest of the talk: Uncertainty at the endpoints (Alice/Bob) Rest of the talk: Uncertainty at the endpoints (Alice/Bob) Modern complication: Modern complication: Alice+Bob communicating using computers Alice+Bob communicating using computers Huge diversity of computers/computing environments Huge diversity of computers/computing environments Computers as diverse as humans; likely to misinterpret communication. Computers as diverse as humans; likely to misinterpret communication. Alice: How should I “explain” to Bob? Alice: How should I “explain” to Bob? Bob: What did Alice mean to say? Bob: What did Alice mean to say? August 6, 2015KAIST: Reliable Meaningful Communication19

20. of 27 New Era, New Challenges: Interacting entities not jointly designed. Interacting entities not jointly designed. Can’t design encoder+decoder jointly. Can’t design encoder+decoder jointly. Can they be build independently? Can they be build independently? Can we have a theory about such? Can we have a theory about such? Where we prove that they will work? Where we prove that they will work? Hopefully: Hopefully: YES YES And the world of practice will adopt principles. And the world of practice will adopt principles. August 6, 2015KAIST: Reliable Meaningful Communication20

21. of 27 Example Problem Archiving data Archiving data Physical libraries have survived for 100s of years. Physical libraries have survived for 100s of years. Digital books have survived for five years. Digital books have survived for five years. Can we be sure they will survive for the next five hundred? Can we be sure they will survive for the next five hundred? Problem: Uncertainty of the future. Problem: Uncertainty of the future. What formats/systems will prevail? What formats/systems will prevail? Why aren’t software systems ever constant? Why aren’t software systems ever constant? Problem: Problem: When designing one system, it is uncertain what the other’s design is (or will be in the future)! When designing one system, it is uncertain what the other’s design is (or will be in the future)! August 6, 2015KAIST: Reliable Meaningful Communication21

22. of 27 Challenge: If Decoder does not know the Encoder, how should it try to guess what it meant? If Decoder does not know the Encoder, how should it try to guess what it meant? Similar example: Similar example: Learning to speak a foreign language Learning to speak a foreign language Humans do … (?) Humans do … (?) Can we understand how/why? Can we understand how/why? Will we be restricted to talking to humans only? Will we be restricted to talking to humans only? Can we learn to talk to “aliens”? Whales? Can we learn to talk to “aliens”? Whales? Claim: Claim: Questions can be formulated mathematically. Questions can be formulated mathematically. Solutions still being explored. Solutions still being explored. August 6, 2015KAIST: Reliable Meaningful Communication22

23. of 27 Modelling uncertainty Classical Shannon Model August 6, 2015KAIST: Reliable Meaningful Communication23 A B Channel B2B2B2B2 AkAkAkAk A3A3A3A3 A2A2A2A2 A1A1A1A1 B1B1B1B1 B3B3B3B3 BjBjBjBj Uncertain Communication Model New Class of Problems New challenges Needs more attention!

24. of 27 Modern questions/answers Communicating players share large context. Communicating players share large context. Knowledge of English, grammar, socio-political context Knowledge of English, grammar, socio-political context Or … Operating system, communication protocols, apps, compression schemes. Or … Operating system, communication protocols, apps, compression schemes. But sharing is not perfect. But sharing is not perfect. Can we retain some of the benefit of the large shared context, when sharing is imperfect? Can we retain some of the benefit of the large shared context, when sharing is imperfect? Answer: Yes … in many cases … [ ongoing work ] Answer: Yes … in many cases … [ ongoing work ] New understanding of human mechanisms New understanding of human mechanisms New reliability mechanisms coping with uncertainty! New reliability mechanisms coping with uncertainty! August 6, 2015KAIST: Reliable Meaningful Communication24

25. of 27 Language as compression August 6, 2015KAIST: Reliable Meaningful Communication25 1 2

26. of 27 Summary Reliability in Communication Reliability in Communication Key Engineering problem of the past century Key Engineering problem of the past century Led to novel mathematics Led to novel mathematics Remarkable solutions Remarkable solutions Hugely successful in theory and practice Hugely successful in theory and practice New Era has New Challenges New Era has New Challenges Hopefully new solutions, incorporating ideas from … Hopefully new solutions, incorporating ideas from … Information theory, computability/complexity, game theory, learning, evolution, linguistics … Information theory, computability/complexity, game theory, learning, evolution, linguistics … … Further enriching mathematics … Further enriching mathematics August 6, 2015KAIST: Reliable Meaningful Communication26

27. of 27 Thank You! August 6, 2015KAIST: Reliable Meaningful Communication27

28. of 27 A challenging special case August 6, 2015KAIST: Reliable Meaningful Communication28

29. of 27 Meaning of Meaning? Difference between meaning and words Difference between meaning and words Exemplified in Exemplified in Turing machine vs. universal encoding Turing machine vs. universal encoding Algorithm vs. computer program Algorithm vs. computer program Can we learn to communicate former? Can we learn to communicate former? Many universal TMs, programming languages Many universal TMs, programming languages [Juba,S.’08], [Goldreich,Juba,S.’12]: [Juba,S.’08], [Goldreich,Juba,S.’12]: Not generically … Not generically … Must have a goal: what will we get from the bits? Must have a goal: what will we get from the bits? Must be able to sense progress towards goal. Must be able to sense progress towards goal. Can use sensing to detect errors in understanding, and to learn correct meaning. Can use sensing to detect errors in understanding, and to learn correct meaning. [Leshno,S’13]: [Leshno,S’13]: Game theoretic interpretation Game theoretic interpretation August 6, 2015KAIST: Reliable Meaningful Communication29

30. of 27 Communication as Coordination Game [Leshno,S.’13] Two players playing series of coordination games Two players playing series of coordination games Coordination? Coordination? Two players simultaneously choose 0/1 actions. Two players simultaneously choose 0/1 actions. “Win” if both agree : “Win” if both agree : Alice’s payoff: not less if they agree Alice’s payoff: not less if they agree Bob’s payoff: strictly higher if they agree. Bob’s payoff: strictly higher if they agree. How should Bob play? How should Bob play? Doesn’t know what Alice will do. But can hope to learn. Doesn’t know what Alice will do. But can hope to learn. Can he hope to eventually learn her behavior and (after finite # of miscoordinations) always coordinate? Can he hope to eventually learn her behavior and (after finite # of miscoordinations) always coordinate? Theorem: Theorem: Not Deterministically (under mild “general” assumptions) Not Deterministically (under mild “general” assumptions) Yes with randomness (under mild restrictions) Yes with randomness (under mild restrictions) August 6, 2015KAIST: Reliable Meaningful Communication30