of 23 09/24/2013HLF: Reliable Meaningful Communication1 Reliable Meaningful Communication Madhu Sudan Microsoft, Cambridge, USA

of 23 Reliable Communication? Problem from the 1940s: Advent of digital age. Problem from the 1940s: Advent of digital age. Communication media are always noisy Communication media are always noisy But digital information less tolerant to noise! But digital information less tolerant to noise! 09/24/2013HLF: Reliable Meaningful Communication2 AliceAlice BobBob We are not ready We are now ready

of 23 Theory of Communication [Shannon, 1948] [Shannon, 1948] Model for noisy communication channels Model for noisy communication channels Architecture for reliable communication Architecture for reliable communication Analysis of some communication schemes Analysis of some communication schemes Limits to any communication scheme Limits to any communication scheme 09/24/2013HLF: Reliable Meaningful Communication3

of 23 Channel = Probabilistic Map from Input to Output Channel = Probabilistic Map from Input to Output Example: Binary Symmetric Channel (BSC(p)) Example: Binary Symmetric Channel (BSC(p)) Modelling Noisy Channels 09/24/2013HLF: Reliable Meaningful Communication p p p 1-p Input Output ChannelChannel

of 23 Some limiting values p=0 p=0 Channel is perfectly reliable. Channel is perfectly reliable. No need to do anything to get 100% utilization No need to do anything to get 100% utilization (1 bit of information received/bit sent) p=½ p=½ Channel output independent of senders signal. Channel output independent of senders signal. No way to get any information through. No way to get any information through. (0 bits of information received/bit sent) 09/24/2013HLF: Reliable Meaningful Communication5

of 23 Lessons from Repetition 09/24/2013HLF: Reliable Meaningful Communication6

of 23 Shannons Architecture 09/24/2013HLF: Reliable Meaningful Communication7 AliceAlice BobBob Encoder Decoder

of 23 Shannons Analysis 09/24/2013HLF: Reliable Meaningful Communication8

of 23 Limit theorems 09/24/2013HLF: Reliable Meaningful Communication9

of 23 An aside: Data Compression 09/24/2013HLF: Reliable Meaningful Communication10

of ? [Hamming 1950]: Error-correcting codes [Hamming 1950]: Error-correcting codes More constructive look at encoding/decoding functions. More constructive look at encoding/decoding functions. Many new codes/encoding functions: Many new codes/encoding functions: Based on Algebra, Graph-Theory, Probability. Based on Algebra, Graph-Theory, Probability. Many novel algorithms: Many novel algorithms: Make encoding/decoding efficient. Make encoding/decoding efficient. Result: Result: Most channels can be exploited. Most channels can be exploited. Even if error is not probabilistic. Even if error is not probabilistic. Profound influence on practice. Profound influence on practice. 09/24/2013HLF: Reliable Meaningful Communication11

of 23 Modern Challenges 09/24/2013HLF: Reliable Meaningful Communication12

of 23 New Kind of Uncertainty Uncertainty always has been a central problem: Uncertainty always has been a central problem: But usually focusses on uncertainty introduced by the channel But usually focusses on uncertainty introduced by the channel Rest of the talk: Uncertainty at the endpoints (Alice/Bob) Rest of the talk: Uncertainty at the endpoints (Alice/Bob) Modern complication: Modern complication: Alice+Bob communicating using computers Alice+Bob communicating using computers Both know how to program. Both know how to program. May end up changing encoder/decoder (unintentionally/unilaterally). May end up changing encoder/decoder (unintentionally/unilaterally). Alice: How should I explain to Bob? Alice: How should I explain to Bob? Bob: What did Alice mean to say? Bob: What did Alice mean to say? 09/24/2013HLF: Reliable Meaningful Communication13

of 23 New Era, New Challenges: Interacting entities not jointly designed. Interacting entities not jointly designed. Cant design encoder+decoder jointly. Cant design encoder+decoder jointly. Can they be build independently? Can they be build independently? Can we have a theory about such? Can we have a theory about such? Where we prove that they will work? Where we prove that they will work? Hopefully: Hopefully: YES YES And the world of practice will adopt principles. And the world of practice will adopt principles. 09/24/2013HLF: Reliable Meaningful Communication14

of 23 Example Problem Archiving data Archiving data Physical libraries have survived for 100s of years. Physical libraries have survived for 100s of years. Digital books have survived for five years. Digital books have survived for five years. Can we be sure they will survive for the next five hundred? Can we be sure they will survive for the next five hundred? Problem: Uncertainty of the future. Problem: Uncertainty of the future. What formats/systems will prevail? What formats/systems will prevail? Why arent software systems ever constant? Why arent software systems ever constant? Problem: Problem: When designing one system, it is uncertain what the others design is (or will be in the future)! When designing one system, it is uncertain what the others design is (or will be in the future)! 09/24/2013HLF: Reliable Meaningful Communication15

of 23 Challenge: If Decoder does not know the Encoder, how should it try to guess what it meant? If Decoder does not know the Encoder, how should it try to guess what it meant? Similar example: Similar example: Learning to speak a foreign language Learning to speak a foreign language Humans do … (?) Humans do … (?) Can we understand how/why? Can we understand how/why? Will we be restricted to talking to humans only? Will we be restricted to talking to humans only? Can we learn to talk to aliens? Whales? Can we learn to talk to aliens? Whales? Claim: Claim: Questions can be formulated mathematically. Questions can be formulated mathematically. Solutions still being explored. Solutions still being explored. 09/24/2013HLF: Reliable Meaningful Communication16

of 23 Modelling uncertainty Classical Shannon Model 09/24/2013HLF: Reliable Meaningful Communication17 A B Channel B2B2B2B2 AkAkAkAk A3A3A3A3 A2A2A2A2 A1A1A1A1 B1B1B1B1 B3B3B3B3 BjBjBjBj Uncertain Communication Model New Class of Problems New challenges Needs more attention!

of 23 Language as compression 09/24/2013HLF: Reliable Meaningful Communication18 1 2

of 23 A challenging special case 09/24/2013HLF: Reliable Meaningful Communication19

of 23 Meaning of Meaning? Difference between meaning and words Difference between meaning and words Exemplified in Exemplified in Turing machine vs. universal encoding Turing machine vs. universal encoding Algorithm vs. computer program Algorithm vs. computer program Can we learn to communicate former? Can we learn to communicate former? Many universal TMs, programming languages Many universal TMs, programming languages [Juba,S.08], [Goldreich,Juba,S.12]: [Juba,S.08], [Goldreich,Juba,S.12]: Not generically … Not generically … Must have a goal: what will we get from the bits? Must have a goal: what will we get from the bits? Must be able to sense progress towards goal. Must be able to sense progress towards goal. Can use sensing to detect errors in understanding, and to learn correct meaning. Can use sensing to detect errors in understanding, and to learn correct meaning. [Leshno,S13]: [Leshno,S13]: Game theoretic interpretation Game theoretic interpretation 09/24/2013HLF: Reliable Meaningful Communication20

of 23 Communication as Coordination Game [Leshno,S.13] Two players playing series of coordination games Two players playing series of coordination games Coordination? Coordination? Two players simultaneously choose 0/1 actions. Two players simultaneously choose 0/1 actions. Win if both agree : Win if both agree : Alices payoff: not less if they agree Alices payoff: not less if they agree Bobs payoff: strictly higher if they agree. Bobs payoff: strictly higher if they agree. How should Bob play? How should Bob play? Doesnt know what Alice will do. But can hope to learn. Doesnt know what Alice will do. But can hope to learn. Can he hope to eventually learn her behavior and (after finite # of miscoordinations) always coordinate? Can he hope to eventually learn her behavior and (after finite # of miscoordinations) always coordinate? Theorem: Theorem: Not Deterministically (under mild general assumptions) Not Deterministically (under mild general assumptions) Yes with randomness (under mild restrictions) Yes with randomness (under mild restrictions) 09/24/2013HLF: Reliable Meaningful Communication21

of 23 Summary Understanding how to communicate meaning is challenging: Understanding how to communicate meaning is challenging: Randomness remains key resource! Randomness remains key resource! Much still to be explored. Much still to be explored. Needs to incorporate ideas from many facets Needs to incorporate ideas from many facets Information theory Information theory Computability/Complexity Computability/Complexity Game theory Game theory Learning, Evolution … Learning, Evolution … But Mathematics has no boundaries … But Mathematics has no boundaries … 09/24/2013HLF: Reliable Meaningful Communication22

of 23 Thank You! 09/24/2013HLF: Reliable Meaningful Communication23