INFORMATION THEORY BYK.SWARAJA ASSOCIATE PROFESSOR MREC

Introduction Communication : The purpose of communication system is to carry the information bearing base band signals from one place to another on communication channel This is done by number of modulation schemes Information theory is a branch of probability theory which may be applied to the study of the communication system Information theory deals with mathematical modelling and analysis of communication system rather than with physical sources and physical channels

Discrete memory less channels(DMC) DMC is a statistical model with an input ‘x’ and an output ‘y’. During each unit of time,the channel accepts an input symbol from ‘x’ and in response it generates an output symbol ‘y’. The channel is ‘discrete ’when alphabets of x & y are both finite. The channel is “memory less” when current output depends on only the current input and not on any of the previous input. Each possible input-to-output path is indicated along with a conditional probability p(y i /x i )

INFORMATION: Information contained in a message depends on its probability of occurrence. If the probability of occurrence of a particular message is more, then it contains less amount of information and vice versa and approaches to zero for a probability of unity. The probability occurrence of each possible message is known such that the given set of probabilities satisfies the condition given as

Information sources A Practical source communication system is a device which produces messages(analog or discrete). The set of source symbols is called the Source alphabet and the elements of the set are called Symbols or letters. Information Sources can be classified as a) memory b) memory less A source with memory is one for which current symbol depends on the previous symbols. A memory less source is one for which each symbol produced is independent of the previous symbols.

Amount of information The amount of information contents contained in the signal is measured with the extent of uncertainty in the information If pi=0 and pk=1 then there is no information generated. If source symbol occur with low probability then there is more amount of information. If the source symbol occur with high probability then there is less amount of information.

INFORMATION Definition: If two independent messages are generated in sequence, then total information should equal to the sum of individual information contents of two messages. Total probability of composite message is the product of two individual probabilities The probability of information must be such the when probability are multiplied together, the corresponding channels are added, the logarithmic function satisfies these requirements

The amount of information I(sk) gained after the event s=sk occurs with probability pk is defined as the logarithmic function. I(sk) = log(1/pk) for k=0,1,2,3…….(k-1) I(s k ) = -log 2 (p k ) The unit of information is a bit, its base is 2 The amount of information can be expressed as

Entropy(Average information) Entropy of a source is a measure of the average amount of information per source symbol in a long message. Units: bits/symbol. Let the amount of information I(sk) generated by a discrete source takes on the values I(s o ),I(s 1 ),I(s 2 )…….I(S k-1 ) with respective probabilities P 0,P 1,P 2,...P(S K-1 ). The entropy of source alphabet ‘s’ is given as H(S)=

H(S)= H(S)= Since the entropy is a measure of uncertainty,the probability distribution that generates the maximum uncertainty will have the maximum entropy.

Entropy of binary & multilevel memory less source Consider a binary source for which symbol ‘0’ occurs with probability p 0 and symbol ‘1’ occurs with probability ‘1’=(1-p 0 ) i.e. the case of two equiprobable events. The source entropy, H(s) = p 0 log 2 (1/p 0 )+ p 1 log 2 (1/p 1 ) In case of binary level, the entropy becomes maximum (1bit/symbol) when both the symbols are equiprobable. Similarly in case of multilevel, the entropy becomes maximum when all the symbols are equiprobable. For equiprobable symbols, H(S)=log 2 M, which gives minimum number of bits needed to encode the symbol.This ensures the minimum transmission bandwidth. The source entropy H(S) satisfies the following relation 0≤ H(S) ≤ log 2 M Lower bound occurs when P k =1 and Upper bound occurs when p(S i )=1/m i.e. for equiprobable events.

Various specific channels 1. Lossless channels: A channel described by a channel matrix with only one non-zero element in each column is called a “lossless channel” 2. Deterministic channel: A channel described by a channel matrix with only one non-zero element in each row. 3. Noiseless channel: A channel is called noiseless if it is both lossless and deterministic. 4. Binary symmetric channel(BSC):This channel is symmetric because the probability of receiving ‘1’if ‘o’ is send is same as probability of receiving a ‘0’ if a ‘1’ is sent. p[y/x]=

Joint entropy Definition : H(x, y) is the average uncertainty of the communication channel as a whole considering the entropy due to channel input as well as channel output. Where p(x i,y j ) is the joint probability of the average uncertainty of the channel input H(x) and the average uncertainty of the channel output H(y). The entropy functions for a channel with ‘m’ inputs and ‘n’ outputs are given as Where p(x i ) represents the input probabilities and p(y j )represents the output probabilities.

Conditional entropy Conditional entropy H(x/y) and H(y/x),is a measure of the average uncertainty remaining about the channel input after the channel output has been observed and the channel output after the channel input has been observed respectively.

Mutual information The mutual information I(x;y) of a DMC with the input alphabet x, the output alphabet y and conditional probabilities p(y k /x j ), where j=0,1,2,…….(j-1) and k=0,1,2, ……(k-1) is defined as I{x;y}= where p(x j,y k ) is the joint probability distribution of random variables x and y and is defined as p(x j,y k )=p(y k /x j )p(x j ) where p(y k /x j ) is the conditional probabilities and given as p(y k /x j )=p(y=y k /x=x j ); for all values of ‘j’ and ‘k’.

P(y k ) is the marginal probability distribution of output random variable y and is given as p(y k ) = In order to calculate mutual information, it is necessary to know the input probability distribution, p(x j ) where j=0,1,2,3,…..(j-1) Mutual information of a channel depends not only on the channel but also on the way in which the channel is used.

Mutual information in terms of entropy I(x;y)=H(x)-H(x/y) b/symbol, where H(x) is the entropy of channel input and H(x/y) is the conditional entropy of channel input given the channel output. llly, I(y;x)=H(y)-H(y/x) The difference between entropy and conditional entropy must represent uncertainty about the channel output which is nothing but mutual information of the channel.

Properties I(x;y)=I(y;x) -----symmetric I(x;y)≥ non negative property I(x;y)=H(y)-H(y/x) I(x;y)=H(y)+H(y)-H(x,y) -----joint entropy of input/output channel

Information rate If the time rate at which source ‘s’ emits symbols is r(symbol, s) the information rate ‘R’ of the source is given by R= r H(s) b/second here, R=information rate H(s) is entropy and ‘r’ is the rate at which symbols are generated. Units: ‘R’ is represented in average number of bits of information per second.it is calculated as R=[r in symbols/sec]*[H(s) in information bits/symbol] R=information bits/second

Shannon limit for channel capacity Definition: The maximum data rate at which information can be transmitted over a given communication channel, under given condition is referred to as the channel capacity C=Blog 2 (1+S/N) bps for a given level of noise, the data rate or capacity can be increased by increasing either bandwidth or signal strength. If bandwidth B is increased, more noise is introduced to the system. Since noise is assumed to be white noise (N=KTB). Thus as B increases, SNR decreases.

Shannon’s theorem says that if R≤C, it is possible to transmit information without any error and if noise present coding techniques are used to correct and detect the errors. The negative statement of shannon’s theorem says that if R>C then every message will be in error. Noise less channel has infinite capacity i..e, if N=0 then S/N  ∞ then c=∞ Infinite bandwidth channel has finite capacity. This is because as bandwidth increases noise power also increases i.e. N=N O B Due to this increase in noise power, signal to noise ratio decreases. Hence even if B  ∞ capacity does not approaches infinity.

Bandwidth and channel capacity In a noise-free communication channel, the limitation on data rate of the transmitted signal is mainly constrained by the available bandwidth of the communication channel. A communication channel cannot propagate a signal that contains a frequency which is changing at a rate greater than bandwidth of the channel. The greater the bandwidth available the higher the information carrying capacity. For binary information, the highest signal rate that can be supported is 2B bps in a given bandwidth of BHz.