The Channel and Mutual Information Chapter 7 The Channel and Mutual Information

Information Through a Channel symbols can’t be swallowed A a1 : : aq b1 : : bs B alphabet of symbols sent alphabet of symbols received P(bj|ai) or randomly generated For example, in an error correcting code over a noisy channel, s ≥ q. If two symbols sent are indistinguishable when received, s < q. Characterize a stationary channel by a matrix of conditional probabilities: Compare: noise (randomness) versus distortion (permutation) received row column sent s Pi,j Pi,j = P(bj | ai) P = q 7.1, 7.2, 7.3

[p(a1) … p(aq)]P = [p(b1) … p(bs)] For p(ai) = probability of source symbols, let p(bj) = probability of being received [p(a1) … p(aq)]P = [p(b1) … p(bs)] no noise: Pi,j = I; p(bj) = p(aj) all noise: Pi,j = 1/s; p(bj) = 1/s The probability that ai was sent and bj was received is: Baye’s Theorem Intuition: given b_j, some a_i must have been sent P(ai, bj) = p(ai) ∙ P(bj | ai) = p(bj) ∙ P(ai | bj). [coincidental probability] So if p(bj) ≠ 0, the backwards conditional probabilities are: 7.1, 7.2, 7.3

Binary symmetric Channel P0,0 a = 0 b = 0 p(a = 0) = p p(a = 1) = 1 − p P0,0 = P1,1 = P P0,1 = P1,0 = Q P0,1 P1,0 P1,1 a = 1 b = 1 a = 0 a = 1 p(b = 0) p(b = 1) P Q Q P ( pP + (1 − p)Q pQ + (1 − p)P ) (p 1−p) = 7.4

Backwards conditional probabilities No noise All noise P(a = 0 | b = 0) = Pp 1 p Pp + Q(1−p) P(a = 1 | b = 0) Q(1−p) 1 − p P(a = 0 | b = 1) Qp Qp + P(1−p) P(a = 1 | b = 1) P(1−p) Is this the only situation where the forwards and backwards matrix is the same? P = 1 Q = 0 P = Q = ½ If p = 1 − p = ½ (equiprobable) then: P(a = 1 | b = 0) = P(a = 0 | b = 1) = Q P(a = 0 | b = 0) = P(a = 1 | b = 1) = P P Q Q P 7.4

System Entropies H(A) Input entropy H(B) Output entropy H(A| B) H(B| A) H(A) Input entropy H(B) Output entropy condition on bj average over all bj Similarly The information loss in the channel, called equivocation (or noise entropy). The average uncertainty about the symbol sent or received. 7.5

H(A, B) = H(A, B) Joint Entropy Define : H(B | A) H(A) H(A | B) H(B) Intuition: taking snapshots A B Define : H(A| B) H(B| A) H(B | A) H(A) The signal H(A) is the information you want. The signal and noise H(B) is the information you have. H(B|A) is the information you don’t want (from noise in the channel). H(A | B) H(A, B) = H(B) 7.5

a priori a posteriori H(A, B) joint Mutual Information P(ai | bj) H(B| A) P(ai | bj) p(ai) I(A; B) The amount of information they are sharing corresponds to Information gain upon receiving bj : I(ai) − I(ai | bj) . shared I(A; B) is the reduction in the uncertainty of A due to the knowledge of B. By symmetry: If ai and bj are independent (all noise), then P(ai , bj) = p(ai) ∙ p(bj) and hence P(ai | bj) = p(ai)  I(ai ; bj) = 0. No information gained in channel. 7.6

= H(A) + H(B) − H(A, B) ≥ 0  H(A, B)  H(A) + H(B) Average over all ai: Similarly: from symmetry By Gibbs I(A; B) ≥ 0. Equality only if P(ai, bj) = p(ai)∙p(bj) [independence]. Can derive these inequalities from the figure on the previous slide. = H(A) + H(B) − H(A, B) ≥ 0  H(A, B)  H(A) + H(B) We know H(A, B) = H(A) + H(B | A) = H(B) + H(A | B).  I(A ; B) = H(A) − H(A | B) = H(B) − H(B | A) ≥ 0  H(A | B)  H(A) and H(B | A)  H(B). 7.6