Presented by Minkoo Seo March, 2006 Entropy Presented by Minkoo Seo March, 2006

Self information (Surprisal) Suppose that almost all children like ice cream. You ask her, “Do you like ice cream?” Her answer is always yes. There’s no surprisal and no information.

Self information (Surprisal) (cont) If a child answers, “No”, then you are surprised. You expect more information from surprising answers. In other words,

Self information (Surprisal) (cont) Now, suppose that there are two events. We want to get ‘self information’ value from them. The result would be

Self information (Surprisal) (cont) Multiplication is neither easy nor intuitive. We want information can be added like other measures. So, information Probability of a message

Entropy Suppose that you are tossing a coin. Your observation was TTHTHTTT. Then, how can we measure the average information per symbol?

Entropy (cont) We define Shannon’s entropy, or decrease in uncertainty, as Expected self information All messages

Joint Entropy Joint entropy of two discrete random variable is defined as

Conditional Entropy Conditional entropy of a discrete random variable X given Y=y is defined as Basic property of the conditional entropy Information of X when we know Y Information of X,Y Given Y

Mutual Information A quantity that measures the mutual dependence of the two variables. How much information can be obtained by observing the other.

Mutual Information (cont) Definition Amount of uncertainty of B(or A) reduced when A(or B) is known Information of B gained by observing A Information of B Information of B which is remained when we know A

Applications Coding theory Clustering Bits needed for describing a data Clustering Make a cluster from two variables whose mutual information is large Large mutual information implies that those two variables conveys the almost same information

References Wikipedia Ian Korf et al, BLAST, O’Reilly, 2003.