Information Theory

Information Suppose that we have the source alphabet of q symbols s 1, s 2,.., s q, each with its probability p(s i )=p i. How much information do we get when we receive one of these symbols ? Suppose that we have the source alphabet of q symbols s 1, s 2,.., s q, each with its probability p(s i )=p i. How much information do we get when we receive one of these symbols ? For example, if p 1 =1, then there is no “ surprise ”, no information since you know what the message must be For example, if p 1 =1, then there is no “ surprise ”, no information since you know what the message must be

On the other hand, if the probabilities are all very different, then when a symbol with a low probability arrives, you feel more surprised, get more information, than when a symbol with a higher probability arrives On the other hand, if the probabilities are all very different, then when a symbol with a low probability arrives, you feel more surprised, get more information, than when a symbol with a higher probability arrives I(p) : A function which measures the amount of information-surprise ( uncertainty in the occurrence of an event of probability p ). I(p) : A function which measures the amount of information-surprise ( uncertainty in the occurrence of an event of probability p ). Define I(p) = log1/p = – log p Define I(p) = log1/p = – log p If we use the base 2 logs, the resulting unit of information is call a bit ( binary digit ). If we use the base 2 logs, the resulting unit of information is call a bit ( binary digit ). Information

Properties of I(p) Properties of I(p) I(p)  0 (a real nonnegative measure) I(p)  0 (a real nonnegative measure) I(p 1 p 2 ) = I(p 1 )+I(p 2 ) for independent event I(p 1 p 2 ) = I(p 1 )+I(p 2 ) for independent event I(p) is a continuous function of p I(p) is a continuous function of p If we get I(s i ) units of information when we receive the symbol s i how much do we get on the average ? If we get I(s i ) units of information when we receive the symbol s i how much do we get on the average ? Since p i is the probability of getting the information I(s i ), then on the average we get for each symbol s i, Since p i is the probability of getting the information I(s i ), then on the average we get for each symbol s i, p i I(s i ) = p i log 2 (1/ p i ) Entropy

Thus, on the average, over the whole alphabet of symbols s i, we will get Thus, on the average, over the whole alphabet of symbols s i, we will get Conventionally, we write Conventionally, we write as the entropy of the signaling system S having symbols s i and probability p i. Entropy

Entropy S={s 1, s 2, s 3, s 4 }, p 1 =1/2, p 2 =1/4, p 3 = p 4 =1/8 Then H 2 (S) = 1/2 log /4 log /8log /8log 2 8 = (1/2)+(1/4)2+(1/8)3+(1/8)3 = 1(3/4) bits of information In this example, L avg of Huffman code can reach H r (S)

Entropy and Coding Instantaneous Codes : A code is said to be instantaneous if when a complete symbol is received, the receiver immediately knows this, and do not have to look further before deciding what message symbol is received Instantaneous Codes : A code is said to be instantaneous if when a complete symbol is received, the receiver immediately knows this, and do not have to look further before deciding what message symbol is received No encoded symbol of this code is a prefix of any other symbol No encoded symbol of this code is a prefix of any other symbol

Entropy and Coding Example : s 1 =0 s 1 =0 s 2 =10 s 2 =01 s 3 =110 s 3 =011 s 4 =111 s 4 =111 instantaneous non-instantaneous Example : s 1 =0 s 1 =0 s 2 =10 s 2 =01 s 3 =110 s 3 =011 s 4 =111 s 4 =111 instantaneous non-instantaneous s 2 s 3 s 1 s s 3 s 4 s s 2 s 3 s 1 s s 3 s 4 s 4 It is necessary and sufficient that an instantaneous code have no code word s i which is a prefix of another code word, s j. It is necessary and sufficient that an instantaneous code have no code word s i which is a prefix of another code word, s j.

Shannon-Fano Coding Shannon-Fano coding is less efficient than is Huffman coding, but has the advantage that you can go directly from the probability p i to the code word length l i Shannon-Fano coding is less efficient than is Huffman coding, but has the advantage that you can go directly from the probability p i to the code word length l i

Shannon-Fano Coding Example : p 1 =p 2 = 1/4, p 3 =p 4 =p 5 =p 6 =1/8 Example : p 1 =p 2 = 1/4, p 3 =p 4 =p 5 =p 6 =1/8 Shannon-Fano lengths: Shannon-Fano lengths: l 1 = l 2 = 2, l 3 = l 4 = l 5 = l 6 =3 we then assign s 1 =00 s 3 =100 s 2 =01 s 4 =101 s 5 =110 s 5 =110 s 6 =111 s 6 =111

Extensions of a Code Example: Example: s 1 : p 1 = 2/3 s 1 0 s 2 : p 2 = 1/3 s 2 1 Lavg = 1 The second extension takes two symbols at a time s 1 s 1 : p 11 = 4/9 s 1 s 1 1 s 1 s 2 : p 12 = 2/9 s 1 s 2 01 s 2 s 1 : p 21 = 2/9 s 2 s s 2 s 2 : p 22 = 1/9 s 2 s Lavg = 17/18 = Lavg for the third extension : 76/87 = Lavg for the fourth extension :