(C) 2000, The University of Michigan 1 Language and Information Handout #2 September 21, 2000

(C) 2000, The University of Michigan 2 Course Information Instructor: Dragomir R. Radev Office: 305A, West Hall Phone: (734) Office hours: TTh 3-4 Course page: Class meets on Thursdays, 5-8 PM in 311 West Hall

(C) 2000, The University of Michigan 3 Readings Textbook: –Oakes, Chapter 2, pages 53 – 76 Additional readings –M&S, Chapter 7, pages (minus Section 7.4) –M&S, Chapter 8, pages (minus Sections 8.3-4)

(C) 2000, The University of Michigan 4 Information Theory

(C) 2000, The University of Michigan 5 Entropy Let p(x) be the probability mass function of a random variable X, over a discrete set of symbols (or alphabet) X: p(x) = P(X=x), x  X Example: throwing two coins and counting heads and tails Entropy (self-information): is the average uncertainty of a single random variable:

(C) 2000, The University of Michigan 6 Information theoretic measures Claude Shannon (information theory): “information = unexpectedness” Series of events (messages) with associated probabilities: p i (i = 1.. n) Goal: to measure the information content, H(p 1, …, p n ) of a particular message Simplest case: the messages are words When p i is low, the word is less informative

(C) 2000, The University of Michigan 7 Properties of information content H is a continuous function of the p i If all p are equal (p i = 1/n), then H is a monotone increasing function of n if a message is broken into two successive messages, the original H is a weighted sum of the resulting values of H

(C) 2000, The University of Michigan 8 Example Only function satisfying all three properties is the entropy function: p 1 = 1/2, p 2 = 1/3, p 3 = 1/6 H = -  p i log 2 p i

(C) 2000, The University of Michigan 9 Example (cont’d) H = - (1/2 log 2 1/2 + 1/3 log 2 1/3 + 1/6 log 2 1/6) = 1/2 log /3 log /6 log 2 6 = 1/ / /6 = 1.46 H =  p i log 2 (1/p i ) Alternative formula for H:

(C) 2000, The University of Michigan 10 Another example Example: –No tickets left: P = 1/2 –Matinee shows only: P = 1/4 –Eve. show, undesirable seats: P = 1/8 –Eve. Show, orchestra seats: P = 1/8

(C) 2000, The University of Michigan 11 Example (cont’d) H = - (1/2 log 1/2 + 1/4 log 1/4 + 1/8 log 1/8 + 1/8 log 1/8) H = - (1/2 x -1) + (1/4 x -2) + (1/8 x -3) + (1/8 x -3) H = 1.75 (bits per symbol)

(C) 2000, The University of Michigan 12 Characteristics of Entropy When one of the messages has a probability approaching 1, then entropy decreases. When all messages have the same probability, entropy increases. Maximum entropy: when P = 1/n (H = ??) Relative entropy: ratio of actual entropy to maximum entropy Redundancy: 1 - relative entropy

(C) 2000, The University of Michigan 13 Entropy examples Letter frequencies in Simplified Polynesian: P(1/8), T(1/4), K(1/8), A(1/4), I (1/8), U (1/8) What is H(P)? What is the shortest code that can be designed to describe simplified Polynesian? What is the entropy of a weighted coin? Draw a diagram.

(C) 2000, The University of Michigan 14 Joint entropy and conditional entropy The joint entropy of a pair of discrete random variables X, Y  p(x,y) is the amount of information needed on average to specify both their values H (X,Y) = -  x  y p(x,y) log 2 p(X,Y) The conditional entropy of a discrete random variable Y given another X, for X, Y  p(x,y) expresses how much extra information is need to communicate Y given that the other party knows X H (Y|X) = -  x  y p(x,y) log 2 p(y|x)

(C) 2000, The University of Michigan 15 Connection between joint and conditional entropies There is a chain rule for entropy (note that the products in the chain rules for probabilities have become sums because of the log): H (X,Y) = H(X) + H(Y|X) H (X 1,…,X n ) = H(X 1 ) + H(X 2 |X 1 ) + … + H(X n |X 1,…,X n-1 )

(C) 2000, The University of Michigan 16 Simplified Polynesian revisited

(C) 2000, The University of Michigan 17 Mutual information Mutual information: reduction in uncertainty of one random variable due to knowing about another, or the amount of information one random variable contains about another. H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y) H(X) – H(X|Y) = H(Y) – H(Y|X) = I(X;Y)

(C) 2000, The University of Michigan 18 Mutual information and entropy H(X|Y) I(X;Y) H(Y|X) H(X|Y) H(X,Y) I(X;Y) is 0 iff two variables are independent For two dependent variables, mutual information grows not only with the degree of dependence, but also according to the entropy of the variables

(C) 2000, The University of Michigan 19 Formulas for I(X;Y) I(X;Y) = H(X) – H(X|Y) = H(X) + H(Y) – H(X,Y) I(X;Y) =  xy p(x,y) log 2 p(x)p(y) p(x,y) Since H(X|X) = 0, note that H(X) = H(X)-H(X|X) = I(X;X) I(x;y) = log 2 p(x)p(y) p(x,y) : pointwise mutual information

(C) 2000, The University of Michigan 20 The noisy channel model Encoder Channel p(y|x) Decoder Message from a finite alphabet XYŴ W Input to channel Output from channel Attempt to reconstruct message based on output p p Binary symmetric channel

(C) 2000, The University of Michigan 21 Statistical NLP as decoding problems

(C) 2000, The University of Michigan 22 Coding

(C) 2000, The University of Michigan 23 Compression Huffman coding (prefix property) Ziv-Lempel codes (better) arithmetic codes (better for images - why?)

(C) 2000, The University of Michigan 24 Huffman coding Developed by David Huffman (1952) Average of 5 bits per character Based on frequency distributions of symbols Algorithm: iteratively build a tree of symbols starting with the two least frequent symbols

(C) 2000, The University of Michigan 25

(C) 2000, The University of Michigan c bd f g ij he a

(C) 2000, The University of Michigan 27

(C) 2000, The University of Michigan 28 Exercise Consider the bit string: Use the Huffman code from the example to decode it. Try inserting, deleting, and switching some bits at random locations and try decoding.

(C) 2000, The University of Michigan 29 Ziv-Lempel coding Two types - one is known as LZ77 (used in GZIP) Code: set of triples a: how far back in the decoded text to look for the upcoming text segment b: how many characters to copy c: new character to add to complete segment

(C) 2000, The University of Michigan 30 p pe pet peter peter_ peter_pi peter_piper peter_piper_pic peter_piper_pick peter_piper_picked peter_piper_picked_a peter_piper_picked_a_pe peter_piper_picked_a_peck_ peter_piper_picked_a_peck_o peter_piper_picked_a_peck_of peter_piper_picked_a_peck_of_pickl peter_piper_picked_a_peck_of_pickled peter_piper_picked_a_peck_of_pickled_pep peter_piper_picked_a_peck_of_pickled_pepper peter_piper_picked_a_peck_of_pickled_peppers

(C) 2000, The University of Michigan 31

(C) 2000, The University of Michigan 32 Arithmetic coding Uses probabilities Achieves about 2.5 bits per character

(C) 2000, The University of Michigan 33

(C) 2000, The University of Michigan 34 Exercise Assuming the alphabet consists of a, b, and c, develop arithmetic encoding for the following strings: aaaaab ababaa abccab cbabac