Chapter 20 Information Theory

Slides:



Advertisements
Similar presentations
The microcanonical ensemble Finding the probability distribution We consider an isolated system in the sense that the energy is a constant of motion. We.
Advertisements

Estimation of Means and Proportions
Heat capacity The electronic heat capacity C e can be found by taking the derivative of Equation (19.18): For temperatures that are small compared with.
Entropy. Optimal Value Example The decimal number 563 costs 10  3 = 30 units. The binary number costs 2  10 = 20 units.  Same value as decimal.
Lecture 2: Basic Information Theory Thinh Nguyen Oregon State University.
The Statistical Interpretation of Entropy The aim of this lecture is to show that entropy can be interpreted in terms of the degree of randomness as originally.
Copyright © Cengage Learning. All rights reserved. 6 Point Estimation.
1 Seventh Lecture Error Analysis Instrumentation and Product Testing.
Information Theory and Security
Inferential Statistics
Some basic concepts of Information Theory and Entropy
STATISTIC & INFORMATION THEORY (CSNB134)
Copyright © Cengage Learning. All rights reserved. CHAPTER 11 ANALYSIS OF ALGORITHM EFFICIENCY ANALYSIS OF ALGORITHM EFFICIENCY.
Unit 4: Mathematics Introduce the laws of Logarithms. Aims Objectives
1. Entropy as an Information Measure - Discrete variable definition Relationship to Code Length - Continuous Variable Differential Entropy 2. Maximum Entropy.
McGraw-Hill/IrwinCopyright © 2009 by The McGraw-Hill Companies, Inc. All Rights Reserved. Chapter 9 Hypothesis Testing.
Probability Rules!. ● Probability relates short-term results to long-term results ● An example  A short term result – what is the chance of getting a.
Chemical Reactions in Ideal Gases. Non-reacting ideal gas mixture Consider a binary mixture of molecules of types A and B. The canonical partition function.
Information theory (part II) From the form g(R) + g(S) = g(RS) one can expect that the function g( ) shall be a logarithm function. In a general format,
Chapter 2. Thermodynamics, Statistical Mechanics, and Metropolis Algorithm (2.1~2.5) Minsu Lee Adaptive Cooperative Systems, Martin Beckerman.
Chapter 6: Random Errors in Chemical Analysis. 6A The nature of random errors Random, or indeterminate, errors can never be totally eliminated and are.
ENTROPY Entropy measures the uncertainty in a random experiment. Let X be a discrete random variable with range S X = { 1,2,3,... k} and pmf p k = P X.
KHARKOV NATIONAL MEDICAL UNIVERSITY MEDICAL INFORMATICS МЕДИЧНА ІНФОРМАТИКА MEDICAL INFORMATICS.
APPLICATIONS OF DIFFERENTIATION Antiderivatives In this section, we will learn about: Antiderivatives and how they are useful in solving certain.
Theoretical distributions: the Normal distribution.
5.1 Exponential Functions
Adding Probabilities 12-5
Virtual University of Pakistan
Chapter 13 Classical & Quantum Statistics
Copyright © Cengage Learning. All rights reserved.
Chapter 12 Chi-Square Tests and Nonparametric Tests
Chapter 7. Classification and Prediction
Chapter-2 Maxwell-Boltzmann Statistics.
Decisions Under Risk and Uncertainty
Confidence Interval Estimation
What does science have to do with math???
ECO 173 Chapter 10: Introduction to Estimation Lecture 5a
Digital Logic & Design Dr. Waseem Ikram Lecture 02.
Classification of Research
The Statistical Imagination
Ch9: Decision Trees 9.1 Introduction A decision tree:
Distribution of the Sample Means
Chapter 5 Sampling Distributions
Hiroki Sayama NECSI Summer School 2008 Week 3: Methods for the Study of Complex Systems Information Theory p I(p)
Chapter 5 Sampling Distributions
COT 5611 Operating Systems Design Principles Spring 2012
COT 5611 Operating Systems Design Principles Spring 2014
ECO 173 Chapter 10: Introduction to Estimation Lecture 5a
Chapter 4 – Part 3.
Introduction to Instrumentation Engineering
Chapter 9 Hypothesis Testing.
Precalculus Mathematics for Calculus Fifth Edition
More about Posterior Distributions
Chapter 5 Sampling Distributions
Digital Logic & Design Lecture 02.
Virtual University of Pakistan
Chapter 1: Statistical Basis of Thermodynamics
Numerical Analysis Lecture 2.
Homework Solution Draw out all 15 possible microstates for a system of three molecules with equally-spaced (1 quantum) energy levels and four quanta of.
Information Theory PHYS 4315 R. S. Rubins, Fall 2009.
Statistical analysis and its application
CHAPTER – 1.1 UNCERTAINTIES IN MEASUREMENTS.
Statistical Mechanics and Canonical Ensemble
Lecture 2: Basic Information Theory
Information Theoretical Analysis of Digital Watermarking
Ticket in the Door GA Milestone Practice Test
Chapter 8 Section 6 Solving Exponential & Logarithmic Equations
Objectives 6.1 Estimating with confidence Statistical confidence
Grand Canonical Ensemble and Criteria for Equilibrium
Continuous Random Variables: Basics
Presentation transcript:

Chapter 20 Information Theory

20.1 Introduction Statistical thermodynamics provides the tool of calculating entropy. Entropy is a measure of the degree of randomness or disorder of a system. Disorder implies a lack of information regarding the exact state of the system. A disordered system is one about which we lack complete information.

20.2 Uncertainty and Information Claude Shannon laid down the foundations of the information theory. Further developed by Leon Brillouin. Applied to Statistical thermodynamics by E. T. Jaynes. The cornerstone of the Shannon theory is the observation that information is a combination of the certain and the uncertain, of the expected and the unexpected.

The degree of surprise generated by a certain event – one that has already occurred – is zero. If a less probable event is reported, the information conveyed is greater. The information should increase as the probability decreases.

For a given experiment, consider a set of possible outcomes whose probabilities are p1, p2, … pn. It is possible to find a quantity H(p1 . . . pn) that measures in a unique way the amount of uncertainty represented by the given set of probabilities. Only three conditions are needed to specify the function H(p1 . . . pn) to within a constant factor. They are: 1. H is a continuous function of the p. 2. If all the pi’s are equal, pi = 1/n; then H(1/n,…, 1/n) is a monotonic increasing function of n. 3. If the possible outcomes of a particular experiment depend on the possible outcomes of n subsidiary experiments, then H is the sum of the uncertainties of the subsidiary experiments.

The above discussion leads to g(R) + g(S) = g(RS), where one can expect that the function g( ) shall be a logarithm function. In a general format, the function can be written as g(x) = A ln(x) + C, where A and C are constants. From the earlier transformation g(x) = x f(1/x), one gets that the uncertainty quantity, H, shall be (1/p)f(p) = A ln(p) + C, where p = 1/n (n is the total number of event). Therefore, f(p) = A*p*ln(p) + C Given that if the probability is 1, the uncertainty H must be zero, the constant C should be equal to ZERO. Thus, f(p) = A*p*ln(p). Since p is smaller than 1, ln(p) shall be minus and thus the constant A is inherently negative

Following conventional notion, we write f(p) = -K. p Following conventional notion, we write f(p) = -K*p*ln(p), where K is a positive coefficient. The uncertainty quantity H(p1, p2, …pn) = Σ f(pi) Thus, H(p1, p2, …pn) = Σ –K*pi*ln(pi) = -K Σpi*ln(pi) Example: H(1/2, 1/3, 1/6) = -K*[1/2ln(1/2) + 1/3*ln(1/3) + 1/6*ln(1/6)] = -K*(-0.346 – 0.377 – 0.299) = 1.01K from the decomposed procedure, H(1/2,1/2) + 1/2H(2/3, 1/3) = -K*[1/2ln(1/2) + 1/2ln(1/2)] -1/2*K*[2/3*ln(2/3) + 1/3ln(1/3)] = -K(-0.346-0.346) – K/2*(-0.27 – 0.366) = 1.01K For equal probable events, pi = 1/n, H = K*ln(n)

In a binary case, where two possible outcomes of an experiment with probabilities, p1 and p2 with p1 + p2 = 1 H = - K*[p1ln(p1) + p2ln(p2)] To determine H value when p1 is 0 or 1, one need L’Hopital’s rule lim[u(x)/v(x)] as x approaches 0 equals lim[u’(x)/v’(x)] Therefore, as p1 approaches 0, lim[p1ln(p1)] = lim[(1/x)/(-1/x2)] = 0 The uncertainty is therefore 0 when either p1 or p2 is zero! Under what value of p1 while H reaches the maximum? differentiate eq - K*[p1ln(p1) + p2ln(p2)] against p1 and set the derivative equal 0 dH/dp1 = -K*[ln(p1) + p1/(p1) - ln(1- p1) – (1- p1)/ (1- p1)] = 0 which leads to p1 = 1/2

20.3 Unit of Information Choosing 2 as the basis of the logarithm and take K = 1, one gets H = 1 We call the unit of information a bit for binary event. Decimal digit, H = log2(10) = 3.32, thus a decimal digit contains about 3 and 1/3 bits of information.

Linguistics A more refined analysis works in terms of component syllables. One can test what is significant in a syllable in speech by swapping syllables and seeing if meaning or tense is changed or lost. The table gives some examples of the application of this statistical approach to some works of literature.

Linguistics The type of interesting results that arise from such studies include: (a) English has the lowest entropy of any major language, and (b) Shakespeare’s work has the lowest entropy of any author studied. These ideas are now progressing beyond the scientific level and are impinging on new ideas of criticism. Here as in biology, the thermodynamic notions can be helpful though they must be applied with caution because concepts such as ‘quality’ cannot be measured as they are purely subjective

Maximum entropy The amount of uncertainty: Examples on the connection between entropy and uncertainty (gases in a partitioned container) The determination of the probability that has maximum entropy.

Suppose one knows the mean value of some particular variable x, Where the unknown probabilities satisfy the condition: In general there will be a large number of probability distributions consistent with the above information. We will determine the one distribution which yields the largest uncertainty (i.e. information). We need Lagrange multiplier to carry out the analysis.

Where and Then, Solving for ln(pi):

Determine the new Lagrange multipliers λ and u So that We define the partition function Then

Determine multiplier μ We have Therefore,

20.5 The connection to statistical thermodynamics The entropy is define as Then

A disordered system is likely to be in any number of different quantum states. If Nj = 1 for N different states and Nj= 0 for all other available states, The above function is positive and increases with increasing N. Associating Nj/N with the probability pj The expected amount of information we would gain is a measure of our lack of knowledge of the state of the system. Negative entropy (negentropy)

The Boltzmann distribution for non-degenerate energy state Where

Summary Information theory is an extension of thermodynamics and probability theory. Much of the subject is associated with the names of Brillouin and Shannon. It was originally concerned with passing messages on telecom munication systems and with assessing the efficiency of codes. Today it is applied to a wide range of problems, ranging from the analysis of language to the design of computers. In this theory the word ‘information’ is used in a special sense. Sup pose that we are initially faced with a problem about which we have no ‘information’ and that there are P possible answers. When we are given some ‘information’ this has the effect of reducing the number of possible answers and if we are given enough ‘information’ we may get to a unique answer. The effect of increased information is thus to reduce the uncer tainty about a situation. In a sense, therefore, informatiouis the antithesis of entropy since entropy is a measure of the randomness or disorder of a system. This contrast led to the coining of the word rtegentropy to describe information. The basic unit of information theory is the bit—a shortened form of ‘binary digit’.

For example, if one is given a playing card face down without any information, it could be any one of 52; if one is then told that it is an ace, it could be any one of 4; if told that it is also a spade, one knows for certain which card one has. As we are given more information, The situation becomes more certain. In general, to determine which of the P possible outcomes is realized, the required information I is defined as H = K ln P