1. Information theory in the Modern Information Society A.J. Han Vinck University of Duisburg/Essen January 2003 Vinck@exp-math.uni-essen.de

2. content What is Information theory ? Why is it important ? Where do we find it ?

3. Claude Elwood Shannon: April 30, 1916 - February 24, 2001 Shannon (1948), Information theory, The Mathematical theory of Communication

4. What is Information theory about ? Information: knowledge that can be used Communication: exchange of Information Our goal: efficient; reliable; secure

5. Express everything in 0 and 1 Discrete ensemble: a,b,c,d  00, 01, 10, 11 in general: k binary digits specify 2 k messages Analogue signal: 1) sample and 2) represent sample value binary 11 10 01 00 t v Output 00, 10, 01, 01, 11

6. Shannon‘s contributions Modeling: Modeling: how to go from analogue to digital fundamental communication models Bounds: Bounds: how far can we go? achievability impossibility Constructions: Constructions: constructive communication methods with optimum performance and many more!!! 1011 R P

7. efficient: efficient: general problem statement remove redundancy exact, no errors !! remove irrelevance distortion !! Topics: how ? how good ? how fast ?how complex ? + + +

8. efficient: efficient: text represent every symbol with 8 bit  1 book: 8 * (500 pages) * 1000 symbols = 4 Mbit  1 book  compression possible to 1 Mbit (1:4)

9. efficient: efficient: speech sampling speed 8000 samples/sec; accuracy 8 bits/sample; speed 64 kBit/s;  45 minutes lecture = 45*60*64k =180Mbit  45 books  compression possible to 4.8 kBit/s (1:10)

10. efficient: efficient: CD music sampling speed 44.1 k samples/sec; accuracy 16 bits/sample  storage capacity for one hour stereo: 5 Gbit  1250 books  compression possible to 4 bits/sample ( 1:4 )

11. efficient: efficient: digital pictures 300 x 400 pixels x 3 colors x 8 bit/sample  2.9 Mbit/picture; for 25 images/second we need 75 Mb/s 2 hour pictures need 540 Gbit  130.000 books  compression needed (1:100)

12. efficient: summary text:  1 book storage: = 4 Mbit  1 book speech:  45 minutes lecture = 45*60*64k =180Mbit  45 books CD music:  storage capacity for one hour stereo: 5 Gbit  1250 books digital pictures:  2 hour pictures need 540 Gbit  130.000 books

13. efficient: general idea represent likely symbols with short length binary words where likely is derived from -prediction of next symbol in source output - context between the source symbols words sounds context in pictures qquq-ue, q-ua, q-ui, q-uo

14. Morse

15. efficient: applications  Text: Zip; GIF etc.  Music: MP3  Pictures: JPEG, MPEG Contributors in data reduction/compression: Information theorists: A. Lempel and Jacob Ziv : Huffman a.m.m.

16. efficient: example JPEG 15.817MB4.566MB3.267MB2.351MB

17. Secure: Secure: example 1 Problem: Is B the owner of the open lock?

18. Secure: Secure: classical Problem: Is the key present at B?

19. Secure: Secure: example 2

20. Reliable: Transmit 0 or 1 Receive 0 or 1 0 0 correct 01 in - correct 11 correct 1 0 in - correct What can we do about it ?

21. Reliable: 2 examples Transmit A: = 0 0 B: = 1 1 Receive 0 0 or 1 1 OK 0 1 or 1 0 NOK 1 error detected! A: = 0 0 0 B: = 1 1 1  000, 001, 010, 100  A  111, 110, 101, 011  B 1 error corrected!

22. Error Sensitivity: Illustration Error sensitivity: 0.0001=0.01%Error sensitivity: 0.0005=0.05%

23. Optical Storage DVD's seven-fold increase in data capacity over the CD has been largely achieved by tightening up the tolerances throughout the predecessor system The data structure was made more efficient by using a better, more efficient error correction code system.

24. Errors in networking

25. no- comment

26. a meshed structure 3 links down partial Fundamental Fundamental problems to consider fast re-routing of information how to include redundancy ? how much redundancy? Cost versus reliability

27. The success story Qualcomm CDMA Founding information theorists: Irwin Jacobs and Andrew Viterbi

28. narrow-band and broad-band noise SOLUTION SOLUTION: FREQUENCY and TIME DIVISION 123123 time frequency

29. PPM Code Example 6 code words: 123 231 312 132 213 321 distance: = 2 123123

30. Code Division Multiple Access (CDMA) 6 users: 123 231 312 132 213 321 – 1 2 3 4 5 6 2 transmit at the same time 123123 2/3 5/6 5 1/2 2/4 4/5 2/5 2 2

31. Why IT at this university? It is fundamental. The theory is well established Based on –Discrete Mathematics; algorithms –Physics Applications: –Communications; networking; Computer science –Multi-media; medical imaging; biology; languages –Information retrieval; information control

32. Other application: powerline communications

33. Information Theory In 1948, Bell Labs scientist Claude Shannon developed Information Theory, and the world of communications technology has never been the same.