1. Daphne Koller Message Passing Loopy BP and Message Decoding Probabilistic Graphical Models Inference

2. Daphne Koller Message Coding & Decoding U 1, …, U k Encoder X 1, …, X n Noisy Channel Y 1, …, Y n Decoder V 1, …, V k Rate = k/n Bit error rate =

3. Daphne Koller Channel Capacity Noisy Channel 0 1 0 1 0.9 0.1 0 1 0 1 0.9 0.1 ? Binary symmetric channel Binary erasure channel capacity = 0.531capacity = 0.9 0 X capacity =

4. Daphne Koller Shannon’s Theorem McEliece Rate in multiples of capacity Bit error probability

5. Daphne Koller How close to C can we get? Signal to noise ratio (dB) Log bit error prob McEliece -2 -3 -4 -5 -6

6. Daphne Koller Turbocodes (May 1993)

7. Daphne Koller How close to C can we get? Signal to noise ratio (dB) Log bit error prob. Shannon limit = -0.79 McEliece -2 -3 -4 -5 -6 -7

8. Daphne Koller Turbocodes: The Idea U 1, …, U k Noisy Channel Encoder 1 Encoder 2 Y 1 1, …, Y 1 n Y 2 1, …, Y 2 n Compute P(U | Y 1, Y 2 ) Y 1 1, …, Y 1 n Y 2 1, …, Y 2 n Decoder 1 Decoder 2 Bel(U | Y 2 )Bel(U | Y 1 )

9. Daphne Koller Iterations of Turbo Decoding Signal to noise ratio (dB) Log bit error prob. -2 -3 -4 -5 -6

10. Daphne Koller X7X7 Y5Y5 Y7Y7 Y6Y6 X5X5 X6X6 U1U1 U4U4 U2U2 X4X4 X1X1 X2X2 Y4Y4 Y1Y1 Y2Y2 U3U3 X3X3 Y3Y3 Low-Density Parity Checking Codes Invented by Robert Gallager in 1962! original message bits parity bits

11. Daphne Koller Decoding as Loopy BP X7X7 Y5Y5 Y7Y7 Y6Y6 X5X5 X6X6 U1U1 U4U4 U2U2 X4X4 X1X1 X2X2 Y4Y4 Y1Y1 Y2Y2 U3U3 X3X3 Y3Y3 X 5 cluster X 7 cluster X 6 cluster U2U2 U3U3 U4U4 U1U1

12. Daphne Koller Decoding in Action Information Theory, Inference, and Learning Algorithms, David MacKay

13. Daphne Koller Turbo-Codes & LDPCs 3G and 4G mobile telephony standards Mobile television system from Qualcomm Digital video broadcasting Satellite communication systems New NASA missions (e.g., Mars Orbiter) Wireless metropolitan network standard

14. Daphne Koller Summary Loopy BP rediscovered by coding practitioners Understanding turbocodes as loopy BP led to development of many new and better codes – Current codes coming closer and closer to Shannon limit Resurgence of interest in BP led to much deeper understanding of approximate inference in graphical models – Many new algorithms

15. Daphne Koller END END END