Chapter 8 Channel Capacity

bits of useful info per bits actually sent Change in entropy going through the channel (drop in uncertainty): average uncertainty being sent: before receiving after receiving 8.1

Uniform Channel channel probabilities do not change from symbol to symbol I.e. the rows of the probability matrix are permutations of each other. So the following is independent of a: Consider no noise: P(b | a) = 1for some b 0all others W = 0 I(A ; B) = H(B) = H(A) (conforms to intuition only if permutation matrix) 8.2 All noise implies H(B) = W

Capacity of Binary Symmetric Channel P Q p(a = 0) p(a = 1) p(b = 0) p(b = 1) where x = pP + (1 p)Qp = p(a = 0) maximum occurs when x = ½, p = ½ also (unless all noise). C = 1 H 2 (P) 8.5

Numerical Examples If P = ½ + ε, then C(P) 3 ε 2 is a great approximation. ProbabilityCapacity P = ½Q = ½C = 0 % P = 0.6Q = 0.4C ~ 3 % P = 0.7Q = 0.3C ~ 12 % P = 0.8Q = 0.2C ~ 28 % P = 0.9Q = 0.1C ~ 53 % P = 0.99Q = 0.01C ~ 92 % P = 0.999Q = 0.001C ~ 99 % 8.5

Error Detecting Code Use a uniform channel with uniform input: p(a 1 ) = … = p(a q ). Apply to n-bit single error detection, with one parity bit among c i {0, 1}: 8.3 P Q Q P |A| = 2 n1 a = c 1 … c n (even parity) |B| = 2 n c 1 … c n = b (any parity) For blocks of size n, we know the probability of k errors = every b B can be obtained from any a A by k = 0 … n errors:

|| n th term = 0 || 0 th term = 0 || n … | B | = 2 n This is W for one bit W

Error Correcting Code × 3 noisy channel 3 encode triplicate decode majority P Q Q P think of this as the channel originalvs.new prob. of no errors = P 3 = probability of no error prob. of 1 error = 3P 2 Q prob. of 2 errors = 3PQ 2 = probability of an error prob. of 3 errors = Q 3 uncoded coded P 3 +3P 2 Q 3PQ 2 +Q 3 3PQ 2 +Q 3 P 3 +3P 2 Q let P = P 2 (P + 3Q) 8.4

PC(P)C(P)PC(P)/3.9992% %.953% %.828%.89617%.712% %.63%.6482%.51.03% % Shannons Theorem will say that as n = 3, there are codes that take P 1 while C(P)/n C(P). 8.4