Grids A1,1 A1,2 A1,3 A1,4 A2,1 A2,2 A2,3 A2,4 A3,1 A3,2 A3,3 A3,4 A4,1 A4,2 A4,3 A4,4

1: A,B 4: A,D A B D 2: B,C 3: C,D C 1,4 = 1 4,1 = 1 1,2 = 1 1 3,4 = B D 2,1 = 1 4,3 = 1 2: B,C 3: C,D C 2,3 = 1 3,2 = 1

1: A,B 4: A,D A B D 2: B,C 3: C,D C 1,4 = B 2,1(B) y1(A,B) 4,1 = D 3,4(D) y4(A,D) 1: A,B 4: A,D A 1,2 = A 4,1(A) y1 (A,B) 3,4 = C 2,3(C) y3(C,D) B D 2,1 = C 3,2(C) y2(B,C) 4,3 = A 1,4(A) y4(A,D) 2: B,C 3: C,D C 2,3 = B 1,2(B) y2(B,C) 3,2 = D 4,3(D) y3(C,D)

Loopy BP Run P(a1) Iteration # True posterior 0.85 0.8 0.75 0.7 0.65 0.55 5 10 15 20 Iteration #

Grid Cluster Graph A1,1 A1,1 , A1,2 A1,2 A1,2 , A1,3 A1,3 A1,1 , A2,1

Cluster Graphs 1: A, B, C C 3: B,D,F B D E 2: B, C, D 5: D, E 4: B, E

Loopy in Practice Synchronous BP: all messages are updated in parallel asynchronous order 1 Time (seconds) 2 4 6 8 10 12 14 # messages converged Ising Grid x 100 asynchronous order 2 synchronous 11 11 12 12 13 13 21 21 22 22 23 23 31 31 32 32 33 33 Asynchronous is faster than synchronous Order of messages has a significant effect on behaviour

Shannon’s Theorem Goal: Transmit bits over a noisy channel How efficient can we make our transmission, for arbitrarily low probability of making an error Shannon’s result: Define Channel Capacity = bound on code rate for a given signal to noise ratio all rates under this are achievable, for arbitrarily low error rate (simply make messages long enough) no rate above that is achievable # bits per message Rate of a code = # bits sent

Y1 Y2 Y3 Y4 X1 X2 X3 X4 U1 U2 U3 U4 X5 X6 X7 Y5 Y6 Y7

U1 U2 Un Z1 Z2 Zn X1 X2 Xn Y1 Y2 Yn

Permuter Y4 Y8 X4 X8 W1 W2 W3 W4 U1 U2 U3 U4 Y1 Y3 Y5 Y7 Z1 Z2 Z3 Z4

Coding: Post 1993 Shannon limit = -0.79