The Cutoff Rate and Other Limits: Passing the Impassable Richard E. Blahut University of Illinois UIUC 5/4/20151

Shannon’s Ideal Channel Example: Binary Memoryless Channel Stationary Discrete Memoryless 5/4/20152

… … … … … … …. 1 1 … … … … … … A Large Code 5/4/20153

+ + + A convolutional encoder 5/4/20154

Information theory asserts existence of good codes Coding theory wants practical codes and decoders There are binary codes 5/4/20155

Brief History of Codes Algebraic Block Codes 1948 Reed-Solomon codes (1960) Convolutional Codes 1954 Sequential decoding (1951) Viterbi algorithm (1967) Euclidean Trellis Codes 1982 Turbo Codes 1993 Gallager (LDPC) codes (1960) 5/4/20156

Decoders Maximum Likelihood Maximum Block Posterior Maximum Symbol Posterior Typical Sequence Iterative Posterior Minimum Distance Bounded Distance 5/4/20157

My View 1)Channel Capacity 2)Cutoff Rate 3)Critical Rate Distance -based codes Likelihood -based codes Posterior -based codes Polar codes 5/4/20158

For any fixed there is a sequence of codes for which exponentially in blocklength. This sequence does not approach Channel Error Exponent Fact #2 Every code satisfies Fact #1 Codes exist such that 5/4/20159

E(R) 5/4/201510

A sequence of codes drawn from a set of ensembles 5/4/201511

Channel Capacity Channel Critical Rate Channel Cutoff Rate 5/4/201512

Binary Hypotheses Testing Type 1 Error Type 2 Error 5/4/201513

Binary Hypotheses Testing Change Notation 5/4/201514

Bounds onand Upper Bounds on Sphere Packing Bound Minimum Distance Bound Lower Bounds on Random Coding Bound Expurgated Bound 5/4/201515

Bhattacharyya Distance 5/4/201516

175/4/2015

is quadratic near Let be a sequence with Then with so A Code Sequence Approaching Capacity with if 5/4/201518

Capacity: C Shannon(1948) Cutoff Rate: Jacobs & Berlekamp(1968) Massey(1981) Arikan(1985/1988) Error Exponent: Gallager(1965) Forney(1968) Blahut(1972) 5/4/201519

Gallager (1965) Forney (1968) Blahut (1972) where is the Kullback divergence 5/4/201520

Forney’s List Decoding Likelihood Function Likelihood Ratio 5/4/201521

Sequential Decoding Exponential waiting time Work exponential in time Pareto Distribution with Work unbounded if Sequential decoding fails if Is maximum likelihood decoding sequential decoding? 5/4/201522

Two Pareto parameters and Pareto Distribution 5/4/201523

Start with an exponential distribution If is exponential, then is a Pareto distribution The Origin of a Pareto Distribution 5/4/201524

The Origins of Graph-Based Codes Brillouin deBrogle Shannon Battail (1987) Hagenauer (1989) Berrou et al (1993) 5/4/201525

Coding Beyond the Cutoff Rate Parallel – Pinsker Hybrid – Jelinek Turbo – Berrou/Glavieux LDPC – Gallager/Tanner/Wiberg Polar - Arikan 5/4/201526

The Massey Distraction (1981) QECBEC QEC 2 BEC 5/4/201527

Performance Measures Bit Error Rate vs. Message Error Rate 5/4/201528

The Arikan Retraction 5/4/201529

5/4/201530

5/4/201531

The Arkan Redistraction* *Rhetorical 5/4/201532