1. Andrea Montanari and Ruediger Urbanke TIFR Tuesday, January 6th, 2008 Phase Transitions in Coding, Communications, and Inference

2. Outline 1) Thresholds in coding, the large size limit (definition and density evolution characterization) 2) The inversion of limits (length to infty vs size to infty) 3) Phase transitions in measurements (compressed sensing versus message passing, dense versus sparse matrices) 4) Phase transitions in collaborative filtering (the low-rank matrix model)

3. Model Shannon ’48 binary symmetric channel capacity: R≤1-h(ε) binary erasures channel capacity: R≤1-ε

4. Channel Coding code decoding C={000, 010, 101, 111} n... blocklength x MAP (y)=argmax X in C p(x | y) x i MAP (y)=argmax Xi p(x i |y)

5. Factor Graph Representation of Linear Codes (7, 4) Hamming code every linear code Tanner, Wiberg, Koetter, Loeliger, Frey parity-check matrix

6. Low-Density Parity Check Codes (3, 4)-regular codes Gallager ‘60 number of edges is linear in n

7. Ensemble

8. Variations on the Theme irregular LDPC ensembleregular RA ensembleirregular MN ensembleirregular RA ensembleARA ensembleturbo code degree distributions as well as structure protographirregular LDGM ensemble (Luby, Mitzenmacher, Shokrollahi, Spielman, and Stehman)Divsalar, Jin, and McElieceJin, Khandekar, and McElieceAbbasfar, Divsalar, KungBerrou and GlavieuxThorpe, Andrews, DolinarDavey, MacKay

9. Message-Passing Decoding -- BEC ? ? 0 0 0 ? ? ? 0+?0+?0+? =?? 0 0 ? ? ? ? ? 0=00=00? ? 0 0 0 ? 0 ? decoded 0+00+00+0 =00

10. Message-Passing Decoding -- BSCGallager Algorithm

11. Asymptotic Analysis: Computation Graph probability that computation graph of fixed depth becomes tree tends to 1 as n tends to infinity

12. Asymptotic Analysis: Density Evolution -- BEC x 1-(1-x) r-1 xx ε (1-(1-x) r-1 ) l-1 ε Luby,Mitzenmacher, Shokrollahi, Spielman, and Steman ‘97

13. Asymptotic Analysis: Density Evolution -- BEC ε phase transition: ε BP so that x t → 0 for ε< ε BP x t → x ∞ >0 for ε> ε BP

14. Asymptotic Analysis: Density Evolution -- BSC, Gallager Algorithm phase transition: ε BP so that x t → 0 for ε< ε BP x t → x ∞ >0 for ε> ε BP x t =ε (1-p + (x t-1 ))+(1-ε) p - (x t-1 ) p + (x)=((1+(1-2x) r-1 )/2) l-1 p - (x)=((1-(1-2x) r-1 )/2) l-1

15. Asymptotic Analysis: Density Evolution -- BP

16. Inversion of Limits size versus number of iterations

17. Density Evolution Limit

19. “Practical” Limit

21. The Two Limits Easy: (Density Evolution Limit) Hard(er): (“Practical Limit”)

22. Binary Erasure Channel DE Limit “Practical” Limit implies

23. What about “General” Case expansion probabilistic methods Korada and U.

24. Expansion Miller and Burshtein: Random element of LDPC(l, r, n) ensemble is expander with expansion close to 1- 1/l with high probability expansion ~ 1-1/l

25. Why is Expansion Useful?

26. Setting: Channel

27. Setting: Ensemble

28. Setting: Algorithm

29. Aim: Show for this setting that... DE Limit “Practical” Limit implies

30. Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

31. Linearized Decoding Algorithm

32. Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

33. Combine with Density Evolution

34. Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

35. Correlation and Interaction 01 1000 Expected growth: (r-1) 2 ε ? < 1 Problem: interaction correlation (r-1) 2 ε

36. Correlation and Interaction

37. Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

38. Witness

41. Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

42. Monotonicity

43. Randomizing the Noise Outside randomizing noise outside the witness increases the probability of error FKG → ← ⁄ ≤

44. Proof Outline linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

45. Expansion random graph has expansion close to expansion of a tree with high probability ⇒ this limits interaction 01 1000

46. References For a list of references see: http://ipg.epfl.ch/doku.php?id=en:courses:2007-2208:mct

47. Results

48. Open Problems 0.0 0.4 0.3 0.2 0.1 0.20.40.60.8 P b channel entropy