1. In the beginning was the Word... 情報理論：日本語，英語で隔年開講 今年度は日本語で授業を行うが，スライドは英語のものを 使用 Information Theory: English and Japanese, alternate years the course will be taught in Japanese in this year video-recorded English classes  Lecture Archives 2011 Slides are in English this slide can be found at http://apal.naist.jp/~kaji/lecture/http://apal.naist.jp/~kaji/lecture/ test questions are given in both of Japanese and English 1

2. Information Theory Information Theory （情報理論） is founded by C. E. Shannon in 1948 focuses on mathematical theory of communication gave essential impacts on today’s digital technology wired/wireless communication/broadcasting CD/DVD/HDD data compression cryptography, linguistics, bioinformatics, games,... In this class, we learn basic subjects of information theory. (half undergraduate level + half graduate school level) 2 Claude E. Shannon 1916-2001

3. class plan This class consists of four chapters (+ this introduction): chapter 0: the summary and the schedule of this course (today) chapter 1: measuring information chapter 2: compact representation of information chapter 3: coding for noisy communication chapter 4: cryptography 3

4. what’s the problem? To understand our problem, date back to 1940s... Teletype ( 電信 ) was widely used for communication. Morse code: dots ( ∙ ) and dashes ( − ) 4 They already had “digital communication”. 10111000111000000010101010001110111011100011101110001 dot = 1 unit long, dash = 3 units long 1 unit silence between marks 3 units silence between letters, etc.

5. machinery for information processing No computers yet, but there were “machines”... 5 They could do something complicated. The transmission/recording of messages were... inefficient...messages should be as short as possible unreliable...messages are often disturbed by noises The efficiency and the reliability were two major problems. Teletype model 14-KTR, 1940 http://www.baudot.net/teletype/M14.htm Enigma machine http://enigma.wikispaces.com/

6. the model of communication A communication system can be modeled as; 6 C.E. Shannon, A Mathematical Theory of Communication, The Bell System Technical Journal, 27, pp. 379–423, 623–656, 1948. encoder, modulator, codec, etc... channel, storage medium, etc...

7. what is the “efficiency”? A communication is efficient if the size of B is small. subject to A = D, or A ≈ D with, or without noise (B ≠ C, or B = C) 7 ABCD

8. problem one: efficiency Example: You need to record the weather of Tokyo everyday. weather = {sunny, cloudy, rainy} You can use “0” and “1”, but you cannot use blank spaces. 8 weather sunny cloudy rainy codeword 00 01 10 2-bit record everyday 200 bits for 100 days 0100011000 Can we shorten the representation?

9. better code? The code B gives shorter representation than the code A. Can we decode the code B correctly? Yes, as far as the sequence is processed from the beginning. Is there a code which is more compact than this code B? No, and Yes(→ next slide). 9 weather sunny cloudy rainy code A 00 01 10 code B 00 01 1 code A...0100011000 code B...010001100

10. think average Sometimes, events are not equally likely... 10 with the code A, 2.0 bit / event (always) with the code B, 2  0.5 + 2  0.3 + 1  0.2 = 1.8 bit / event in average with the code C, 1  0.5 + 2  0.3 + 2  0.2 = 1.5 bit / event in average weather sunny cloudy rainy probability 0.5 0.3 0.2 code A 00 01 10 code B 00 01 1 code C 1 01 00

11. the best code? Can we represent information with 0.00000000001 bit per event?...No, maybe. It is likely that there is a “limit” which we cannot get over. Shannon investigated the limit mathematically. → For this event set, we need 1.485 or more bit per event. 11 weather sunny cloudy rainy probability 0.5 0.3 0.2 This is the amount of information which must be carried by the code.

12. class plan in April chapter 0: the summary and the schedule of this course chapter 1: measuring information We establish a mathematical means to measure information in a quantitative manner. chapter 2: compact representation of information We learn several coding techniques which give compact representation of information. chapter 3: coding for noisy communication chapter 4: cryptography 12

13. what is the “reliability”? A communication is reliable if A = D or A ≈ D. the existence of noise is essential (B ≠ C) How small can we make the size of B? 13 ABCD

14. problem two: reliability Communication is not always reliable. transmitted information ≠ received information 14 Errors of this kind are unavoidable in real communication. In the usual conversation, we sometimes use phonetic codes. ABCABC ABCADC ABCAlpha, Bravo, Charlie ABC Alpha, Bravo, Charlie あさひの「あ」 いろはの「い」

15. phonetic code A phonetic code adds redundant information. The redundant part helps correcting possible errors. →use this mechanism over 0-1 data, and we can correct errors! 15 the real information redundant ( 冗長な ) information for correcting possible errors

16. redundancy Q. Can we add “redundancy” to binary data? A. Yes, use parity bits. A parity bit is... a binary digit which is to make the number of 1’s in data even. 00101 → 001010 (two 1’s → two 1’s) 11010 → 110101 (three 1’s → four 1’s) One parity bit may tell you that there are odd numbers of errors, but not more than that. 16

17. to correct error(s) basic idea: use several parity bits to correct errors Example: Add five parity bits to four-bits data (a 0, a 1, a 2, a 3 ). 17 This code corrects one-bit error, but it is too straightforward. a0a0 a2a2 a1a1 a3a3 p0p0 p1p1 q0q0 q1q1 r codeword = (a 0, a 1, a 2, a 3, p 0, p 1, q 0, q 1, r)

18. class plan in May chapter 0: the summary and the schedule of this course chapter 1: measuring information chapter 2: compact representation of information chapter 3: coding for noisy communication We study practical coding techniques for finding and correcting errors. chapter 4: cryptography We review techniques for protecting information from intensive attacks. 18

19. schedule April 19 Tue 10 17 24 Thu 12 19 26 May01 08 15 22 29 03 10 17 24 31 June × × test: questions given in English/Japanese report (quiz): will be assigned by the end of April (Mon) 0405 × statistics in 2011: A... 51 / B... 20 / C... 18 / did not pass... 13

20. chapter 1: measuring information 20

21. motivation “To tell plenty of things, we need more words.”...maybe true, but can you give the proof of this statement? We will need to... 1. measure information quantitatively ( 定量的に測る ) 2. observe the relation between the amount of information and its representation. Chapter 1 focuses on the first step above. 21

22. the uncertainty ( 不確実さ ) Information tells what has happened at the information source. Before you receive information, there is much uncertainty. After you receive information, the uncertainty becomes small. the difference of uncertainty  the amount of information FIRST, we need to measure the uncertainty of information source. 22 much uncertainty Before small uncertainty After this difference indicates the amount of information

23. the definition of uncertainty The uncertainty is defined according to the statistics ( 統計量 ), BUT, we do not have enough time today.... In the rest of today’s talk, we study two typical information sources. memoryless & stationary information source Markov information source 23

24. assumption In this class, we assume that... an information source produces one symbol per unit time (discrete-time information source) the set of possible symbols is finite and countable ( 有限可算 ) (digital information source) Note however that, in the real world, there are continuous-time and/or analogue information sources. cf. sampling & quantization 24

25. Preliminary ( 準備 ) Assume a discrete-time digital information source S: M = {a 1,..., a k }... the set of symbols of S (S is said to be a k-ary information source.) X t...the symbol which S produces at time t The sequence X 1,..., X n is called a message produced by S. Example: S = fair dice 25 if the message is, then

26. memoryless & stationary information source 26 trial 1 trial 2 trial 3 : 123456... ajcgea... gajkfh... wasdas... : the same probability distribution memoryless = 無記憶 stationary = 定常

27. memoryless & stationary information source 27

28. Markov information source a simple model of information source with memory The choice of the next symbol depends on at most m previous symbols (m-th order Markov source) 28 Andrey Markov 1856-1922 m = 0  memoryless source m = 1  simple Markov source

29. Example of (simple) Markov source S... memoryless & stationary source with P(0) = q, P(1) = 1 – q 29 S R 1-bit register XtXt if X t–1 = 0, then R = 0: S = 0  X t = 0... P Xt|Xt–1 (0 | 0) = q S = 1  X t = 1... P Xt|Xt–1 (1 | 0) = 1 – q if X t–1 = 1, then R = 1: S = 0  X t = 1... P Xt|Xt–1 (1 | 1) = q S = 1  X t = 0... P Xt|Xt–1 (0 | 1) = 1 – q

30. 01 1 / 1–q 0 / 1–q 0 / q1 / q Markov source as a finite state machine m-th order k-ary Markov source: The next symbols depends on previous m symbols. The model is having one of k m internal states. The state changes when a new symbol is generated.  finite state machine 30 S R 1-bit register XtXt generated symbol probability

31. 31 two important properties irreducible ( 既約 ) Markov source: We can move to any state from any state. A BC this example is NOT irreducible aperiodic ( 非周期的 ) Markov source: We have no periodical behavior (strict discussion needed...). AB this example is NOT aperiodic irreducible + aperiodic = regular

32. 32 example of the regular Markov source AB 0/0.9 1/0.1 0/0.8 1/0.2 converge ( 収束する ) to the same probabilities stationary probabilities time P (state=A) P (state=B) 1 1.0 0.0 2 0.9 0.1 3 0.89 0.11 4 0.889 0.111... start from the state 0 time P (state=A) P (state=B) 1 0.0 1.0 2 0.8 0.2 3 0.88 0.12 4 0.888 0.112... start from the state 1

33. 33 computation of the stationary probabilities  t : P(state = A) at time t  t : P(state = B) at time t  t+1 = 0.9  t + 0.8  t  t+1 = 0.1  t + 0.2  t  t+1 +  t+1 = 1 AB 0/0.9 1/0.1 0/0.81/0.2 If  t and  t converge to  and , respectively, then we can put  t+1 =  t =  and  t+1 =  t = .  = 0.9  + 0.8   = 0.1  + 0.2   +  = 1  =8/9,  =1/9

34. 34 Markov source as a stationary source After enough time has elapsed... a regular Markov source can be regarded as a stationary source AB 0/0.9 1/0.1 0/0.81/0.2  =8/9,  =1/9 0 will be produced with probability P(0) = 0.9  + 0.8  = 0.889 1 will be produced with probability P(1) = 0.1  + 0.2  = 0.111

35. 35 summary of today’s class overview of this course motivation four chapters typical information sources memoryless & stationary source Markov source

36. 36 exercise Determine the stationary probabilities. Compute the probability that 010 is produced. A BC 0/0.4 0/0.5 1/0.6 0/0.81/0.5 1/0.2 This is to check your understanding. This is not a report assignment.