1. Mutual Information, Joint Entropy & Conditional Entropy

2. Contents Entropy Joint entropy & conditional entropy
Mutual information

3. Entropy(1/2) Entropy(self-information)
the amount of information in a random variable
average uncertainty of a random variable
the average length of the message needed to transmit an outcome of that variable
the size of the search space consisting of the possible values of a random variable and its associated probabilities
Properties
( : providing no new information)
increases with message length

4. Entropy(2/2) - Example Simplified Polynesian letter frequencies
per-letter entropy
coding
p t k a i u

5. Joint Entropy & Conditional Entropy(1/4)
the amount of information needed on average to specify both their values
Conditional Entropy
how much extra information you still need to supply on average to communicate Y given that the other party knows X

6. Joint Entropy & Conditional Entropy(2/4)
Chain Rules for Entropy

7. Joint Entropy & Conditional Entropy(3/4) - Example
Simplified Polynesian Revisited
syllable structure
all words consist of sequences of CV syllables.
C: consonant, V: vowel

8. Joint Entropy & Conditional Entropy(4/4)
Entropy Rate(per-word/per-letter entropy)
Entropy of a Language

9. Mutual Information(1/2)
the reduction in uncertainty of one random variable due to knowing about another
the amount of information one random variable contains about another
measure of independence
: two variables are independent
grows according to ...
the degree of dependence
the entropy of the variables

10. Mutual Information(2/2)
Conditional Mutual Information
Chain Rule
Pointwise Mutual Information
between two particular points

11. The Noisy Channel Model(1/2)
Assumption
the output of the channel depends probabilistically on the input
Channel capacity
Channel
p(y|x)
Encoder
Decoder
Message from a finite alphabet
Input to channel
Attempt to reconstruct message based on output
Output from channel

12. The Noisy Channel Model(2/2)
The Noisy Channel Model in Linguistics
decode the output to give the most likely input
applications
MT, POS tagging, OCR, Speech recognition, ...
Noisy Channel
Decoder
: language model, : channel probability

13. Derivation of Mutual Information