§4 Continuous source and Gaussian channel

1. Differential entropy §4.1 Continuous source Definition: Let X be a random variable with cumulative distribution function F(x) = Pr(X≤x). If F(x) is continuous, the random variable is said to be continuous. Let when the derivative is defined. If , then p(x) is called the probability density function for X. The set where p(x) > 0 is called the support set of X.

§4.1 Continuous source 1. Differential entropy Definition: The differential entropy h(X) of a continuous random variable X with a density p(x) is defined as where S is the support set of the random variable.

1. Differential entropy §4.1 Continuous source Example 4.1.1 (X~N (m,σ2), Normal distribution) please calculate the differential entropy.

§4.1 Continuous source 1. Differential entropy Definition: The differential entropy of a set X,Y of random variables with density p(xy) is defined as If X,Y have a joint density p(xy), we can define the conditional differential entropy h(X|Y) as

2. Properties of differential entropy §4.1 Continuous source 2. Properties of differential entropy 1) h (XY) = h(X) + h(Y|X) = h(Y) + h(X|Y)

2. Properties of differential entropy §4.1 Continuous source 2. Properties of differential entropy 2) h(X) can be negative. Example 4.1.2 Consider a random variable distributed uniformly from a to b. If (b-a)<1, h(X) < 0.

2. Properties of differential entropy §4.1 Continuous source 2. Properties of differential entropy 3) h(X) is a convex function of the input probabilities p(x), it has the maximum. Theorem 4.1 If the peak power of the random variable X is restricted, the maximizing distribution is the uniform distribution. If the average power of the random variable X is restricted, the maximizing distribution is the normal distribution.

2. Properties of differential entropy §4.1 Continuous source 2. Properties of differential entropy 4) let Y=g(X), the differential entropy of Y may be different with h(X). Example 4.1.3 Let X is a random variable distributed uniformly from -1 to 1, and Y=2X. h(X)=? h(Y)=? Theorem 4.2 Theorem 4.3

Review Differential entropy Chain rule of differential entropy KeyWords: Differential entropy Chain rule of differential entropy Conditioning reduces entropy Independent bound of differential entropy may be negative convex function transformative

a) h (XY) = h(X) + h(Y|X) = h(Y) + h(X|Y) Homework Prove the following conclusions: a) h (XY) = h(X) + h(Y|X) = h(Y) + h(X|Y) 2009.03.23到此，次12，时间正好

§4 Continuous source and Gaussian channel

1.The model of Gaussian channel X Z Y Normal, mean 0, variance σz2 Y=X+Z X and Z are independent

I(X;Y) = h(Y) – h(Y|X) = h(Y) – h(Z|X) = h(Y) – h(Z) §4.2 Gaussian channel 2. Average mutual information I(X;Y) = h(Y) – h(Y|X) (Y=X+Z) = h(Y) – h(Z|X) = h(Y) – h(Z) Example 4.2.1 Let X~N(0,σx2), Y~N(0,σx2+σz2),

3. The channel capacity I(X;Y) = h(Y) – h(Z) §4.2 Gaussian channel Definition: The information capacity of the Gaussian channel with power constraint P is I(X;Y) = h(Y) – h(Z)

§4.2 Gaussian channel 3. The channel capacity

3. The channel capacity §4.2 Gaussian channel (bits/sample ) Thinking about the band-limited channels, transmission bandwidth is W, (bits/sample ) There are 2W samples per second,

§4.2 Gaussian channel 4. Shannon’s formula (bits/sec) Shannon’s famous expression for the capacity of a band-limited, power-limited Gaussian channel.

4. Shannon’s formula 1）Ct、W、SNR can be interchanged. 2） §4.2 Gaussian channel 4. Shannon’s formula Remarks: 1）Ct、W、SNR can be interchanged. 2）

4. Shannon’s formula 3） shannon limit Ct (bps) W §4.2 Gaussian channel For infinite bandwidth channels Ct (bps) W

4. Shannon’s formula §4.2 Gaussian channel Let Eb is the energy per bit, then As

Information rate of Gaussian channel Review KeyWords: Information rate of Gaussian channel Capacity of Gaussian channel Shannon’s fomula (Band limited, power limited) Shannon limit

Homework 1. In image transmission, there are 2.25*106 pixels per frame. Reproducing image needs 4 bits per pixel (assume that each bit has equal probability to choose ‘0’ and ‘1’ ). Compute the channel bandwidth needed for transmitting 30 frames image per second . (P/N = 30dB) 2. Consider a power-limited Gaussian channel , bandwidth is 3kHz, and (P + N)/N = 10dB. (1) Compute the maximum rate of this channel. (bps) (2) If SNR decreases to 5 dB, give the channel bandwidth with the same maximum rate.