Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference

Outline 1) Thresholds in coding, the large size limit (definition and density evolution characterization) 2) The inversion of limits (length to infty vs size to infty) 3) Phase transitions in measurements (compressed sensing versus message passing, dense versus sparse matrices) 4) Phase transitions in collaborative filtering (the low-rank matrix model)

Model Shannon ’48 binary symmetric channel capacity: R≤1-h(ε) binary erasures channel capacity: R≤1-ε

Channel Coding code decoding C={000, 010, 101, 111} n... blocklength x MAP (y)=argmax X in C p(x | y) x i MAP (y)=argmax Xi p(x i |y)

Factor Graph Representation of Linear Codes (7, 4) Hamming code every linear code Tanner, Wiberg, Koetter, Loeliger, Frey parity-check matrix

Low-Density Parity Check Codes (3, 4)-regular codes Gallager ‘60 number of edges is linear in n

Ensemble

Variations on the Theme irregular LDPC ensembleregular RA ensembleirregular MN ensembleirregular RA ensembleARA ensembleturbo code degree distributions as well as structure protographirregular LDGM ensemble (Luby, Mitzenmacher, Shokrollahi, Spielman, and Stehman)Divsalar, Jin, and McElieceJin, Khandekar, and McElieceAbbasfar, Divsalar, KungBerrou and GlavieuxThorpe, Andrews, DolinarDavey, MacKay

Message-Passing Decoding -- BEC ? ? ? ? ? 0+?0+?0+? =?? 0 0 ? ? ? ? ? 0=00=00? ? ? 0 ? decoded =00

Message-Passing Decoding -- BSCGallager Algorithm

Asymptotic Analysis: Computation Graph probability that computation graph of fixed depth becomes tree tends to 1 as n tends to infinity

Asymptotic Analysis: Density Evolution -- BEC x 1-(1-x) r-1 xx ε (1-(1-x) r-1 ) l-1 ε Luby,Mitzenmacher, Shokrollahi, Spielman, and Steman ‘97

Asymptotic Analysis: Density Evolution -- BEC ε phase transition: ε BP so that x t → 0 for ε< ε BP x t → x ∞ >0 for ε> ε BP

Asymptotic Analysis: Density Evolution -- BSC, Gallager Algorithm phase transition: ε BP so that x t → 0 for ε< ε BP x t → x ∞ >0 for ε> ε BP x t =ε (1-p + (x t-1 ))+(1-ε) p - (x t-1 ) p + (x)=((1+(1-2x) r-1 )/2) l-1 p - (x)=((1-(1-2x) r-1 )/2) l-1

Asymptotic Analysis: Density Evolution -- BP

Inversion of Limits size versus number of iterations

Density Evolution Limit

“Practical” Limit

The Two Limits Easy: (Density Evolution Limit) Hard(er): (“Practical Limit”)

Binary Erasure Channel DE Limit “Practical” Limit implies

What about “General” Case expansion probabilistic methods Korada and U.

Expansion Miller and Burshtein: Random element of LDPC(l, r, n) ensemble is expander with expansion close to 1- 1/l with high probability expansion ~ 1-1/l

Why is Expansion Useful?

Setting: Channel

Setting: Ensemble

Setting: Algorithm

Aim: Show for this setting that... DE Limit “Practical” Limit implies

Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Linearized Decoding Algorithm

Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Combine with Density Evolution

Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Correlation and Interaction Expected growth: (r-1) 2 ε ? < 1 Problem: interaction correlation (r-1) 2 ε

Correlation and Interaction

Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Witness

Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Monotonicity

Randomizing the Noise Outside randomizing noise outside the witness increases the probability of error FKG → ← ⁄ ≤

Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Expansion random graph has expansion close to expansion of a tree with high probability ⇒ this limits interaction

References For a list of references see:

Results

Open Problems P b channel entropy