Introduction to Information Theory- Entropy Subject: Information Theory and Coding By Professor Dr. Ayad Ghany Ismaeel for Dept. of computer Technical Eng./ Computer Comm. network . 6/25/2018

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 Topics Model of a Digital Communication System Communication Channel Shannon’s Definition of Communication Shannon want to maximizing the speed of ADSL at your home “Information Theory” or “The Shannon Theory” In terms of Information Theory Terminology Measurement of Information Entropy Discrete Continues References 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Model of a Digital Communication System Father of Digital Communication Message e.g. English symbols Encoder e.g. English to 0,1 sequence Information Source Coding Communication Channel Destination Decoding Can have noise or distortion Decoder e.g. 0,1 sequence to English 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Communication Channel Includes 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 And even this… 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Shannon’s Definition of Communication “The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.” “Frequently the messages have meaning” “... [which is] irrelevant to the engineering problem.” 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 Shannon Wants to… Shannon wants to find a way for “reliably” transmitting data throughout the channel at “maximal” possible rate. Information Source Coding Communication Channel Destination Decoding For example, maximizing the speed of ADSL @ your home 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

And he thought about this problem for a while… He later on found a solution and published in this 1948 paper. 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 In his 1948 paper he build a rich theory to the problem of reliable communication, now called “Information Theory” or “The Shannon Theory” in honor of him. 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 Shannon’s Vision Data Source Encoding Channel Encoding Channel User Source Decoding Channel Decoding 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 Example: Disk Storage Data Zip Add CRC Channel User Unzip Verify CRC 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

In terms of Information Theory Terminology Zip Source Encoding = Data Compression Unzip Source Decoding = Data Decompression Add CRC Channel Encoding Error Protection/ cyclic redundancy check CRC error: indicates that some data in your Zip file (.zip or .zipx) is damaged. = Verify CRC Channel Decoding = Error Correction 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 Shannon Theory The original 1948 Shannon Theory contains: Measurement of Information Source Coding Theory Channel Coding Theory 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Measurement of Information Shannon’s first question is “How to measure information in terms of bits?” = ? bits = ? bits 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

All events are probabilistic! Using Probability Theory, Shannon showed that there is only one way to measure information in terms of number of bits: called the entropy function: is the expected value (average) of the information contained in each message. 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 For example Tossing a dice: Outcomes are 1,2,3,4,5,6 Each occurs at probability 1/6 Information provided by tossing a dice is 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Wait! It is nonsense! The number 2.585-bits is not an integer!! What does you mean? 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 For example, in the experience of throwing a piece normal money. The possibility of the emergence of writing is equal to the possibility of the appearance of the logo. And therefore (we'll use a logarithm base 2): p1 = 0.5, h1 = i(p1) = – log 0.5 = 1 p2 = 0.5, h2 = i(p2) = -log 0.5 = 1 And thus the average amount of information and entropy is: H =( p1) (i1) + (p2)( i2) = 0.5(1) + 0.5(1) = 1 In general, the average amount of information can be the expense of the relationship: Shannon proposed the one to decide (Bits) If we use the logarithm for the basis of 2 per account Entropy. Bit here means the amount of information that we get from this bit. 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 Example Suppose you have: p1 = 0.5 Chance of Rain Tomorrow p2 = 0.001 possibility that impair dinner on my account Does this reflect continued logarithm idea of the amount of information? Yes, fortunately. It achieves the following: log ( 1 ) = 0 log (0) = - infinity And thus it can be used as a measure of the value of information if multiplied by (-1) which gives zero when the event does not contain any surprise (we know it is falling pieces): i(p) = – log(p) To see how it behaves in the previous example, we'll use temporarily the natural logarithm: h1 = i(p1) = – ln(0.5) = 0.693 h2 = i(p2) = -ln(0.001) = 6.9 h = h1 + h2 = 0.693 + 6.9 = 7.593 h’ = i(p) = -ln(p1.p2) = -ln(p1) – ln(p2) = h1 + h2 = h 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 Continuous Entropy 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Continuous Entropy (Cont.) An Example The Uniform Distribution, allow (f ) to be the uniform distribution on [a, b]. That is: Informally, the continuous entropy of the uniform distribution is equal to the log of the width of the interval. note that h(X) can be negative! For example: if X is uniformly distributed in [0; 1/2 ], then h(X) = log ( ½- 0) = log 1/2 = -1. If entropy can be negative. 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02

Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02 References Richard W. Hamming, “Coding and Information Theory” , 2nd Edition, Prentice-Hall, New Jersey-USA, 1986. Mauro Barni and Benedetta Tondi, “Lecture notes on: Information Theory and Coding”, Universita degli Studi di Siena Facolta di Ingegneria, 2012. 6/25/2018 Prof. Dr. Ayad Barznji Information Theory-Entropy/ L01+02