1. Information Theory Kenneth D. Harris 18/3/2015

2. Information theory is… 1.Information theory is a branch of applied mathematics, electrical engineering, and computer science involving the quantification of information. (Wikipedia) 2.Information theory is probability theory where you take logs to base 2.

3. Morse code Code words are shortest for the most common letters This means that messages are, on average, sent more quickly.

4. What is the optimal code? X is a random variable Alice wants to tell Bob the value of X (repeatedly) What is the best binary code to use? How many bits does it take (on average) to transmit the value of X?

5. Optimal code lengths Value of XProbabilityCode word A½0 B¼10 C¼11

6. Entropy

7. Connection to physics

8. Conditional entropy

9. Mutual information

10. Properties of mutual information

11. Data processing inequality

12. Kullback-Leibler divergence Length of codewordLength of optimal codeword

13. Continuous variables Decimal placesEntropy 13.3219 2 6.6439 39.9658 413.2877 516.6096 Infinity

14. K-L divergence for continuous variables

15. Mutual information of continuous variables

16. Differential entropy

17. Maximum entropy distributions

18. Examples of maximum entropy distributions Data typeStatisticDistribution ContinuousMean and varianceGaussian Non-negative continuousMeanExponential ContinuousMeanUndefined AngularCircular mean and vector strength Von Mises Non-negative IntegerMeanGeometric Continuous stationary process Autocovariance functionGaussian process Point processFiring ratePoisson process

19. In neuroscience… We often want to compute the mutual information between a neural activity pattern, and a sensory variable. If I want to tell you the sensory variable and we both know the activity pattern, how many bits can we save? If I want to tell you the activity pattern, and we both know the sensory variable, how many bits can we save?

20. Estimating mutual information

21. Naïve estimate X=0X=1 Y=001 Y=110