1. 236601 - Coding and Algorithms for Memories Lecture 4 1

2. Array of cells, made of floating gate transistors ─ Each cell can store q different levels ─ Today, q typically ranges between 2 and 16 ─ The levels are represented by the number of electrons ─ The cell’s level is increased by pulsing electrons ─ To reduce a cell level, all cells in its containing block must first be reset to level 0 A VERY EXPENSIVE OPERATION Rewriting Codes 2

3. Write-Once Memories (WOM) Introduced by Rivest and Shamir, “How to reuse a write-once memory”, 1982 The memory elements represent bits (2 levels) and are irreversibly programmed from ‘0’ to ‘1’ Q: How many cells are required to write 100 bits twice? P1: Is it possible to do better…? P2: How many cells to write k bits twice? P3: How many cells to write k bits t times? P3’: What is the total number of bits that is possible to write in n cells in t writes? 1 st Write 2 nd Write 3

4. Binary WOM Codes k 1,…,k t :the number of bits on each write – n cells and t writes The sum-rate of the WOM code is R = (Σ 1 t k i )/n – Rivest Shamir: R = (2+2)/3 = 4/3 4

5. Definition: WOM Codes Definition: An [n,t;M 1,…,M t ] t-write WOM code is a coding scheme which consists of n cells and guarantees any t writes of alphabet size M 1,…,M t by programming cells from zero to one – A WOM code consists of t encoding and decoding maps E i, D i, 1 ≤i≤ t –E 1 : {1,…,M 1 }  {0,1} n – For 2 ≤i≤ t, E i : {1,…,M i }×{0,1} n  {0,1} n such that for all (m,c) ∊ {1,…,M i }×{0,1} n, E i (m,c) ≥ c – For 1 ≤i≤ t, D i : {0,1} n  {1,…,M i } such that for D i ( E i (m,c)) =m for all (m,c) ∊ {1,…,M i }×{0,1} n The sum-rate of the WOM code is R = (Σ 1 t logM i )/n Rivest Shamir: [3,2;4,4], R = (log4+log4)/3=4/3 5

6. Definition: WOM Codes There are two cases – The individual rates on each write must all be the same: fixed-rate – The individual rates are allowed to be different: unrestricted-rate We assume that the write number on each write is known. This knowledge does not affect the rate – Assume there exists a [n,t;M 1,…,M t ] t-write WOM code where the write number is known – It is possible to construct a [Nn+t,t;M 1 N,…,M t N ] t-write WOM code where the write number is not-known so asymptotically the sum-rate is the same 6

7. The Entropy Function How many vectors are there with at most a single 1? – How many bits is it possible to represent this way? – What is the rate? How many vectors are there with at most k 1’s? – How many bits is it possible to represent this way? – What is the rate? Is it possible to approximate the value ? Yes! ≈ h(p), where p=k/n and h(p) = -plog(p)-(1-p)log(1-p): the Binary Entropy Function h(p) is the information rate that is possible to represent when bits are programmed with prob. p 7 n+1 log(n+1) log(n+1)/ n log( ) log( )/n

8. 8

9. The Binary Symmetric Channel When transmitting a binary vector, with probability p, every bit is in error Roughly pn bits will be in error The amount of information which is lost is h(p) Therefore, the channel capacity is C(p)=1-h(p) – The channel capacity is an indication on the amount of rate which is lost, or how much is necessary to “pay” in order to correct the errors in the channel 9 0 0 0 0 0 0 0 0 p p 1-p

10. The Capacity of WOM Codes The Capacity Region for two writes C 2-WOM ={(R 1,R 2 )| ∃ p ∊ [0,0.5],R 1 ≤h(p), R 2 ≤1-p} h(p) – the entropy function h(p) = -plog(p)-(1-p)log(1-p) – p – the prob to program a cell on the 1 st write, thus R 1 ≤ h(p) – After the first write, 1-p out of the cells aren’t programmed, thus R 2 ≤ 1-p The maximum achievable sum-rate is max p ∊ [0,0.5] {h(p)+(1-p)} = log3 achieved for p=1/3: R 1 = h(1/3) = log(3)-2/3 R 2 = 1-1/3 = 2/3 10

11. The Capacity of WOM Codes The Capacity Region for two writes C 2-WOM ={(R 1,R 2 )| ∃ p ∊ [0,0.5],R 1 ≤h(p), R 2 ≤1-p} h(p) – the entropy function h(p) = -plog(p)-(1-p)log(1-p) The Capacity Region for t writes: C t-WOM ={(R 1,…,R t )| ∃ p 1,p 2,…p t-1 ∊ [0,0.5], R 1 ≤ h(p 1 ), R 2 ≤ (1–p 1 )h(p 2 ),…, R t-1 ≤ (1–p 1 )  (1–p t–2 )h(p t–1 ) R t ≤ (1–p 1 )  (1–p t–2 )(1–p t–1 )} p 1 - prob to prog. a cell on the 1 st write: R 1 ≤ h(p 1 ) p 2 - prob to prog. a cell on the 2 nd write (from the remainder): R 2 ≤(1-p 1 )h(p 2 ) p t-1 - prob to prog. a cell on the (t-1) th write (from the remainder): R t-1 ≤ (1–p 1 )  (1–p t–2 )h(p t–1 ) R t ≤ (1–p 1 )  (1–p t–2 )(1–p t–1 ) because (1–p 1 )  (1–p t–2 )(1–p t–1 ) cells weren’t programmed The maximum achievable sum-rate is log(t+1) 11

12. The Capacity for Fixed Rate The capacity region for two writes C 2-WOM ={(R 1,R 2 )| ∃ p ∊ [0,0.5],R 1 ≤h(p), R 2 ≤1-p} When forcing R 1 =R 2 we get h(p) = 1-p The (numerical) solution is p = 0.2271, the sum-rate is 1.54 Multiple writes: A recursive formula to calculate the maximum achievable sum-rate R F (1)=1 R F (t+1) = (t+1)  root{h(zt/R F (t))-z} where root{f(z)} is the min positive value z s.t. f(z)=0 – For example: R F (2) = 2  root{h(z)-z} = 2  0.7729=1.5458 R F (3) = 3  root{h(2z/1.54)-z}=3  0.6437=1.9311 12

13. WOM Codes Constructions Rivest and Shamir ‘82 – [3,2; 4,4] (R=1.33); [7,3; 8,8,8] (R=1.28); [7,5; 4,4,4,4,4] (R=1.42); [7,2; 26,26] (R=1.34) – Tabular WOM-codes – “Linear” WOM-codes – David Klaner: [5,3; 5,5,5] (R=1.39) – David Leavitt: [4,4; 7,7,7,7] (R=1.60) – James Saxe: [n,n/2-1; n/2,n/2-1,n/2-2,…,2] (R≈0.5*log n), [12,3; 65,81,64] (R=1.53) Merkx ‘84 – WOM codes constructed with Projective Geometries – [4,4;7,7,7,7] (R=1.60), [31,10; 31,31,31,31,31,31,31,31,31,31] (R=1.598) – [7,4; 8,7,8,8] (R=1.69), [7,4; 8,7,11,8] (R=1.75) – [8,4; 8,14,11,8] (R=1.66), [7,8; 16,16,16,16, 16,16,16,16] (R=1.75) Wu and Jiang ‘09 - Position modulation code for WOM codes – [172,5; 2 56, 2 56,2 56,2 56,2 56 ] (R=1.63), [196,6; 2 56,2 56,2 56,2 56,2 56,2 56 ] (R=1.71), [238,8; 2 56,2 56,2 56,2 56,2 56,2 56,2 56,2 56 ] (R=1.88), [258,9; 2 56,2 56,2 56,2 56,2 56,2 56,2 56,2 56,2 56 ] (R=1.95), [278,10; 2 56,2 56,2 56,2 56,2 56,2 56,2 56,2 56,2 56,2 56 ] (R=2.01) 13

14. James Saxe’s WOM Code [n,n/2-1; n/2,n/2-1,n/2-2,…,2] WOM Code – Partition the memory into two parts of n/2 cells each – First write: input symbol m ∊ {1,…,n/2} program the i th cell of the 1 st group – The i th write, i≥2: input symbol m ∊ {1,…,n/2-i+1} copy the first group to the second group program the i th available cell in the 1 st group – Decoding: There is always one cell that is programmed in the 1 st and not in the 2 nd group Its location, among the non-programmed cells, is the message value – Sum-rate: (log(n/2)+log(n/2-1)+ … +log2)/n=log((n/2)!)/n ≈ (n/2log(n/2))/n ≈ (log n)/2 14

15. James Saxe’s WOM Code [n,n/2-1; n/2,n/2-1,n/2-2,…,2] WOM Code – Partition the memory into two parts of n/2 cells each Example: n=8, [8,3; 4,3,2] – First write: 3 – Second write: 2 – Third write: 1 Sum-rate: (log4+log3+log2)/8=4.58/8=0.57 0,0,0,0|0,0,0,0  0,0,1,0|0,0,0,0  0,1,1,0|0,0,1,0  1,1,1,0|0,1,1,0 15

16. The Coset Coding Scheme Cohen, Godlewski, and Merkx ‘86 – The coset coding scheme – Use Error Correcting Codes (ECC) in order to construct WOM-codes – Let C[n,n-r] be an ECC with parity check matrix H of size r×n – Write r bits: Given a syndrome s of r bits, find a length-n vector e such that H ⋅ e T = s – Use ECC’s that guarantee on successive writes to find vectors that do not overlap with the previously programmed cells – The goal is to find a vector e of minimum weight such that only 0s flip to 1s 16

17. The Coset Coding Scheme C[n,n-r] is an ECC with an r×n parity check matrix H Write r bits: Given a syndrome s of r bits, find a length-n vector e such that H ⋅ e T = s Example: H is the parity check matrix of a Hamming code – s=100, v 1 = 0000100: c = 0000100 – s=000, v 2 = 1001000: c = 1001100 – s=111, v 3 = 0100010: c = 1101110 – s=010, …  can’t write! This matrix gives a [7,3:8,8,8] WOM code 17

18. Binary Two-Write WOM-Codes C[n,n-r] is a linear code w/ parity check matrix H of size r×n For a vector v ∊ {0,1} n, H v is the matrix H with 0’s in the columns that correspond to the positions of the 1’s in v v 1 = (0 1 0 1 1 0 0) 18

19. Binary Two-Write WOM-Codes First Write: program only vectors v such that rank(H v ) = r V C = { v ∊ {0,1} n | rank(H v ) = r} – For H we get |V C | = 92 - we can write 92 messages – Assume we write v 1 = 0 1 0 1 1 0 0 v 1 = (0 1 0 1 1 0 0) 19

20. Binary Two-Write WOM-Codes First Write: program only vectors v such that rank(H v ) = r, V C = { v ∊ {0,1} n | rank(H v ) = r} Second Write Encoding: Second Write Decoding: Multiply the received word by H: H ⋅ (v 1 + v 2 ) = H ⋅ v 1 + H ⋅ v 2 = s 1 + (s 1 + s 2 ) = s 2 v 1 = (0 1 0 1 1 0 0) 1.Write a vector s 2 of r bits 2.Calculate s 1 = H ⋅ v 1 3.Find v 2 such that H v 1 ⋅ v 2 = s 1 +s 2 a 4. v 2 exists since rank(H v 1 ) = r a 5.Write v 1 +v 2 to memory 1. s 2 = 001 2. s 1 = H ⋅ v 1 = 010 3. H v 1 ⋅ v 2 = s 1 +s 2 = 011 a 4. v 2 = 0 0 0 0 0 1 1 5. v 1 +v 2 = 0 1 0 1 1 1 1 20

21. Example Summary Let H be the parity check matrix of the [7,4] Hamming code First write: program only vectors v such that rank(H v ) = 3 V C = { v {0,1} n | rank(H v ) = 3} – For H we get |V C | = 92 - we can write 92 messages – Assume we write v 1 = 0 1 0 1 1 0 0 – Write 0’s in the columns of H corresponding to 1’s in v 1 : H v 1 d Second write: write r = 3 bits, for example: s 2 = 0 0 1 – Calculate s 1 = H ⋅ v 1 = 0 1 0 – Solve: find a vector v 2 such that H v 1 ⋅ v 2 = s 1 + s 2 = 0 1 1 d – Choose v 2 = 0 0 0 0 0 1 1 – Finally, write v 1 + v 2 = 0 1 0 1 1 1 1 – Decoding: 1 1 1 0 1 0 0 1 0 1 1 0 1 0 1 1 0 1 0 0 1 H = 1 0 1 0 0 0 0 H v 1 = 1 0 1 0 0 1 0 1 0 0 0 0 0 1 1 1 1 0 1 0 0 1 0 1 1 0 1 0 1 1 0 1 0 0 1. [0 1 0 1 1 1 1] T = [0 0 1] 21

22. Sum-rate Results The construction works for any linear code C For any C[n,n-r] with parity check matrix H, V C = { v ∊ {0,1} n | rank(H v ) = r} The rate of the first write is: R 1 (C) = (log 2 |V C |)/n The rate of the second write is: R 2 (C) = r/n Thus, the sum-rate is: R(C) = (log 2 |V C | + r)/n In the last example: – R 1 = log(92)/7=6.52/7=0.93, R 2 =3/7=0.42, R=1.35 Goal: Choose a code C with parity check matrix H that maximizes the sum-rate 22

23. Capacity Achieving Results The Capacity region C 2-WOM ={(R 1,R 2 )| ∃ p ∊ [0,0.5],R 1 ≤h(p), R 2 ≤1-p} Theorem: For any (R 1, R 2 ) ∊ C 2-WOM and ε>0, there exists a linear code C satisfying R 1 (C) ≥ R 1 -ε and R 2 (C) ≥ R 2 –ε By computer search – Best unrestricted sum-rate 1.4928 (upper bound 1.58) – Best fixed sum-rate 1.4546 (upper bound 1.54) 23

24. Capacity Region and Achievable Rates of Two-Write WOM codes 24