1. 236601 - Coding and Algorithms for Memories Lecture 5

2. Write-Once Memories (WOM)
Introduced by Rivest and Shamir, “How to reuse a write-once memory”, 1982
The memory elements represent bits (2 levels) and are irreversibly programmed from ‘0’ to ‘1’
Q: How many cells are required to write 100 bits twice?
P1: Is it possible to do better…?
P2: How many cells to write k bits twice?
P3: How many cells to write k bits t times?
P3’: What is the total number of bits that is possible to write in n cells in t writes?
1st
Write
2nd
Write

3. Definition: WOM Codes
Definition: An [n,t;M1,…,Mt] t-write WOM code is a coding scheme which consists of n cells and guarantees any t writes of alphabet size M1,…,Mt by programming cells from zero to one
A WOM code consists of t encoding and decoding maps Ei, Di, 1 ≤i≤ t
E1: {1,…,M1}  {0,1}n
For 2 ≤i≤ t, Ei: {1,…,Mi}×Im(Ei-1)  {0,1}n such that for all (m,c)∊{1,…,Mi}×Im(Ei-1), Ei(m,c) ≥ c
For 1 ≤i≤ t, Di: {0,1}n  {1,…,Mi} such that for Di(Ei(m,c)) =m for all (m,c)∊{1,…,Mi}×Im(Ei-1) The sum-rate of the WOM code is R = (Σ1t logMi)/n
Rivest Shamir: [3,2;4,4], R = (log4+log4)/3=4/3

4. Example Summary Let H be the parity check matrix
Let H be the parity check matrix
of the [7,4] Hamming code
First write: program only vectors v such that rank(Hv) = 3
VC = { v ϵ {0,1}n | rank(Hv) = 3}
For H we get |VC| = 92 - we can write 92 messages
Assume we write v1 =
Write 0’s in the columns of H
corresponding to 1’s in v1: Hv1 d
Second write: write r = 3 bits, for example: s2 = 0 0 1
Calculate s1 = H⋅v1 = 0 1 0
Solve: find a vector v2 such that Hv1⋅v2 = s1 + s2 = d
Choose v2 =
Finally, write v1 + v2 =
Decoding:
H =
Hv1 = .
[ ]T = [0 0 1]

5. Sum-rate Results The construction works for any linear code C
For any C[n,n-r] with parity check matrix H, VC = { v ∊ {0,1}n | rank(Hv) = r}
The rate of the first write is: R1(C) = (log2|VC|)/n
The rate of the second write is: R2(C) = r/n
Thus, the sum-rate is: R(C) = (log2|VC| + r)/n
In the last example:
R1= log(92)/7=6.52/7=0.93, R2=3/7=0.42, R=1.35
Goal: Choose a code C with parity check matrix H that maximizes the sum-rate

6. Capacity Achieving Results
The Capacity region
C2-WOM={(R1,R2)|∃p∊[0,0.5],R1≤h(p), R2≤1-p}
Theorem: For any (R1, R2)∊C2-WOM and ε>0, there exists a linear code C satisfying R1(C) ≥ R1-ε and R2(C) ≥ R2–ε
By computer search
Best unrestricted sum-rate (upper bound 1.58)
Best fixed sum-rate (upper bound 1.54)

7. Capacity Region and Achievable Rates of Two-Write WOM codes
Flash Memory Summit
Santa Clara, CA USA

8. The Capacity of WOM Codes
The Capacity Region for two writes
C2-WOM={(R1,R2)|∃p∊[0,0.5],R1≤h(p), R2≤1-p}
h(p) – the entropy function h(p) = -plog(p)-(1-p)log(1-p)
The Capacity Region for t writes:
Ct-WOM={(R1,…,Rt)| ∃p1,p2,…pt-1∊[0,0.5], R1 ≤ h(p1), R2 ≤ (1–p1)h(p2),…,
Rt-1≤ (1–p1)(1–pt–2)h(pt–1)
Rt ≤ (1–p1)(1–pt–2)(1–pt–1)}
p1 - prob to prog. a cell on the 1st write: R1 ≤ h(p1)
p2 - prob to prog. a cell on the 2nd write (from the remainder): R2≤(1-p1)h(p2)
pt-1 - prob to prog. a cell on the (t-1)th write (from the remainder): Rt-1 ≤ (1–p1)(1–pt–2)h(pt–1)
Rt ≤ (1–p1)(1–pt–2)(1–pt–1) because (1–p1)(1–pt–2)(1–pt–1) cells weren’t programmed
The maximum achievable sum-rate is log(t+1)
Flash Memory Summit
Santa Clara, CA USA

9. Non-Binary WOM Capacity
The capacity of non-binary WOM-codes was given by Fu and Han Vinck, ‘99
The maximal sum-rate using t writes and q-ary cells is
There is no tight upper bound on the sum-rate in case equal amount of information is written on each write
Flash Memory Summit
Santa Clara, CA USA

10. Basic Constructions
Construction 1: Assume t=q–1 and there are n q-ary cells
On each write, n binary bits are written
Write to the cells level by level
The sum-rate is n(q–1)/n = q-1 (upper bound ≈ 2(q-1))
Construction 2: Assume t=2 and q is odd
First write: use only levels 0,1,…,(q-1)/2
Second write: use only levels (q-1)/2,…,q-1
The sum-rate is 2log((q+1)/2)
The difference between the upper bound is
Flash Memory Summit
Santa Clara, CA USA

11. Construction 3 Assume q is a power of 2, q=2m
Every cell is converted into m bits
For 1 ≤ i ≤ m, the ith bits from each cell comprise a t-write binary WOM code
The m WOM codes write into their corresponding bits in the cells independently
Since every bit can only change from 0 to 1, the level of each cell cannot decrease
Use the binary expansion ϕ:{0,1}m→{0,…,2m-1}
x=(x0,…,xm-1), ϕ(x) = ∑j 2m-j xj
Flash Memory Summit
Santa Clara, CA USA

12. Example – Construction 3
Let q = 8 = 23 and n = 3
Every cell corresponds to 3 bits
Use Rivest-Shamir WOM-code
The ith bits from each cell comprise a WOM-code
The sum-rate is 6∙2/3 = 4 (upper bound is 5.17)
Write Number
Data bits
Encoding by the RS base-code
Encoded values in the 8-ary cells 1
(11,01,10)
(100,001,010)
(4,1,2) 2
(00,11,01)
(101,111,110)
(7,3,6)
Flash Memory Summit
Santa Clara, CA USA

13. Construction 3
Theorem: If q=2m and there exists a t-write binary WOM-code of sum-rate R, then there exists a t-write q-ary WOM-code of sum-rate mR
Flash Memory Summit
Santa Clara, CA USA

14. Binary Two-Write WOM-Codes
First Write: program only vectors v such that rank(Hv) = r, VC = { v ∊ {0,1}n | rank(Hv) = r}
Second Write Encoding:
Second Write Decoding: Multiply the received word by H:
H⋅(v1 + v2) = H⋅v1 + H⋅v2 = s1+ (s1 + s2) = s2
Write a vector s2 of r bits
Calculate s1 = H⋅v1
Find v2 such that Hv1⋅v2 = s1+s2 a
v2 exists since rank(Hv1) = r a
Write v1+v2 to memory
s2 = 001
s1 = H⋅v1 = 010
Hv1⋅v2 = s1+s2 = 011 a
v2 =
v1+v2 =
v1 = ( )
Flash Memory Summit
Santa Clara, CA USA

15. Non-Binary Two-Write WOM-Codes
Let C[n,n-r] be a linear code w/ p.c.m H of size r×n over GF(q)
For a vector v ∊ GF(q)n, let Hv be the matrix H with 0’s in the columns that correspond to the positions of the nonzeros’s in v
First Write: program only vectors v such that rank(Hv)=r, VC = { v ∊ GF(q)n | rank(Hv) = r}
Second Write Encoding: Write a vector s2 of r symbols over GF(q)
Let v1 be the programmed vector on the first write, and let s1 = H⋅v1
Remember that rank(Hv1) = r a
Find v2 such that Hv1⋅v2 = -s1+s2 F
A solution v2 exists since rank(Hv1) = r a
Write v1+v2 to memory
Second Write Decoding: Multiply the received word by H:
H⋅(v1 + v2) = H⋅v1 + H⋅v2 = s1+ (-s1 + s2) = s2
This construction works for any linear code over GF(q)
Flash Memory Summit
Santa Clara, CA USA

16. Code Rate The construction works for any linear code C over GF(q)
For any C[n,n-r] with parity check matrix H, VC = { v ∊ GF(q)n | rank(Hv) = r}
The rate of the first write is: R1(C) = (log2|VC|)/n
The rate of the second write is: R2(C) = r⋅log2q/n
Thus, the total rate is: R(C) = (log2|VC| + r⋅log2q)/n
By choosing the parity check matrix H uniformly at random, it is possible to achieve the following capacity region:
{(R1,R2)|∃p∊[0,1-1/q], R1≤h(p)+p⋅log2(q-1), R2≤(1-p)⋅log2q}
The maximum achievable rate log2(2q-1) is achieved for p=(q-1)/(2q-1)
Remarks:
This is not an optimal non-binary two-write WOM code construction
Cells cannot be programmed twice
However, these codes are useful to construct multiple-write WOM codes

17. Three-Write WOM Codes m1∊{1,…, 2nR1} C3 Encoder v1∊GF(3)n Φ u1∊{0,1}2n
Let C3 be an [n,2; 2nR1,2nR2] two-write WOM code over GF(3)
Construct a binary [2n,3; 2nR1,2nR2,2n] 3-write WOM code as follows:
The code has 2n cells, which are broken into n 2-cell blocks
First write: use the first write of the code C3
Write a message m1 from {1,…, 2nR1}
This message is supposed to be written as a ternary vector v1 ∊ GF(3)n
Write the vector v1 into the n 2-cell blocks, using the mapping Φ :GF(3)→ (GF(2),GF(2)), where Φ(0)=(0,0), Φ(1)=(1,0), Φ(2)=(0,1)
Second write: use the second write of the code C3
Write a message m2 from {1,…, 2nR2}
The ternary vector v2 is written into the n 2-cell blocks using the mapping Φ
Each 2-cell block is written at most once and at most one cell is programmed
m1∊{1,…, 2nR1}
C3 Encoder
v1∊GF(3)n Φ
u1∊{0,1}2n
m2∊{1,…, 2nR2}
C3 Encoder
v2∊GF(3)n Φ
u2∊{0,1}2n

18. Three-Write WOM Codes
Let C3 be an [n,2; 2nR1,2nR2] two-write WOM code over GF(3)
Construct a binary [2n,3; 2nR1,2nR2,2n] 3-write WOM code as follows:
The code has 2n cells, which are broken into n 2-cell blocks
First write: use the first write of the code C3
Write a message m1 from {1,…, 2nR1}
This message is supposed to be written as a ternary vector v1 ∊ GF(3)n
Write the vector v1 into the n 2-cell blocks, using the mapping Φ :GF(3)→ (GF(2),GF(2)), where Φ(0)=(0,0), Φ(1)=(1,0), Φ(2)=(0,1)
Second write: use the second write of the code C3
Write a message m2 from {1,…, 2nR2}
The ternary vector v2 is written into the n 2-cell blocks using the mapping Φ
Each 2-cell block is written at most once and at most one cell is programmed
Third Write: write n bits
Each bit is stored in a 2-cell block: the bit value is 1 iff both cells are programmed
It is possible to write a bit in each block since at most one cell in each block was programmed

19. Three-Write WOM Code: Example
Ex: C3 is a 2-write WOM code over GF(3) with n=7 cells
First Write
Write a vector v1 over GF(3) according to the first write of C3
Assume the vector is v1 =
Using the mapping Φ, v1 is written into the 2n = 14 binary cells →Φ
Second Write
Write a vector v2 over GF(3) according to the second write of C3
Assume the vector is v2 =
Using the mapping Φ, v2 is written into the 2n = 14 binary cells →Φ
Third Write: Write 7 bits
Assume the message is v3 = , then write to memory

20. Code Rate
Theorem: Let C3 be an [n,2; 2nR1,2nR2] 2-write WOM code over GF(3) of sum-rate R1+R2, then there exists a binary [2n,3; 2nR1,2nR2,2n] 3-write WOM code of sum-rate (R1+R2+1)/2
The best sum-rate for the code C3 is log5 = 2.32
Conclusion: It is possible to find a binary 3-write WOM code of sum-rate (log5 +1)/2 = 1.66
Using a computer search, the best sum-rate found for WOM codes over GF(3) is 2.22 and thus there exists a 3-write WOM code of sum-rate 1.61

21. Four-Write WOM Codes
Let C3 be an [n,2; 2nR3,1,2nR3,2] two-write WOM code over GF(3) and let C2 be an [n,2; 2nR2,1,2nR2,2] binary two-write WOM code
Construct a [2n,4; 2nR1,2nR2, 2nR2,1,2nR2,2] 4-write WOM code as follows:
The code has 2n cells, which are broken into n 2-cell blocks
First write: use the first write of the code C3
Write a message from {1,…,2nR3,1}
This message is supposed to be written as a ternary vector v ∊ GF(3)n
Write the vector v into the n 2-cell blocks, using the mapping Φ :GF(3)→ (GF(2),GF(2)), where Φ(0)=(0,0), Φ(1)=(1,0), Φ(2)=(0,1)
Second write: use the second write of the code C3
Write a message from {1,…,2nR3,2}
The ternary vector is written into the n 2-cell blocks using the mapping Φ
Note that each 2-cell block is written only once
Third and fourth writes: use the first and second writes of C2
Each bit is stored in a 2-cell block: the bit value is 1 iff both cells are programmed
It is possible to write a bit in each block since at most one cell in each block was programmed

22. Code Rate
Theorem: Let C3 be an [n,2; 2nR3,1,2nR3,2] 2-write WOM code over GF(3) (found before) of sum-rate R1+R2, and let C2 be an [n,2; 2nR2,1,2nR2,2] binary two-write WOM code then there exists a binary [2n,4; 2nR3,1,2nR3,2,2nR2,1,2nR2,2] 4-write WOM code of sum-rate (R3,1+R3,2+R2,1+R2,2)/2 a
The best sum-rate for the code C3 is log5 = 2.32 and for the code C2 is log3 = 1.58
Conclusion: It is possible to find a binary 3-write WOM code of sum-rate (log5 + log3)/2 = 1.95
Using a computer search, the best sum-rate found for codes over GF(3) is 2.22 and for codes over GF(2) is Thus there exists a 4-write WOM code of sum-rate 1.855

23. Five-Write WOM-Codes
Use a 2-write WOM code C3 over GF(3) of sum-rates R3,1, R3,2 and a binary 3-write WOM code C2 of sum-rates R2,1, R2,2, R2,3 a
The first and second writes are implemented as before using the first and second writes of the code C3
The third, fourth, and fifth writes are also implemented as before by using the first, second, and third writes of C2
It is possible to achieve a five-write WOM code of rate (log5+(log5+1)/2)/2=1.99 and there exists a code of rate 1.915
For six writes, we will change the code C2 to be a 4-write WOM code such that it is possible to achieve WOM-rate (log5+1.95)/2 = 2.14 and there exists a code of rate

24. Capacity Achieving Results
The Capacity region
C2-WOM={(R1,R2)|∃p∊[0,0.5],R1≤h(p), R2≤1-p}
Theorem: For any (R1, R2)∊C2-WOM and ε>0, there exists a linear code C satisfying R1(C) ≥ R1-ε and R2(C) ≥ R2–ε
By computer search
Best unrestricted sum-rate (upper bound 1.58)
Best fixed sum-rate (upper bound 1.54)

25. Capacity Region and Achievable Rates of Two-Write WOM codes
Flash Memory Summit
Santa Clara, CA USA

26. The Capacity of WOM Codes
The Capacity Region for two writes
C2-WOM={(R1,R2)|∃p∊[0,0.5],R1≤h(p), R2≤1-p}
h(p) – the entropy function h(p) = -plog(p)-(1-p)log(1-p)
The Capacity Region for t writes:
Ct-WOM={(R1,…,Rt)| ∃p1,p2,…pt-1∊[0,0.5], R1 ≤ h(p1), R2 ≤ (1–p1)h(p2),…,
Rt-1≤ (1–p1)(1–pt–2)h(pt–1)
Rt ≤ (1–p1)(1–pt–2)(1–pt–1)}
p1 - prob to prog. a cell on the 1st write: R1 ≤ h(p1)
p2 - prob to prog. a cell on the 2nd write (from the remainder): R2≤(1-p1)h(p2)
pt-1 - prob to prog. a cell on the (t-1)th write (from the remainder): Rt-1 ≤ (1–p1)(1–pt–2)h(pt–1)
Rt ≤ (1–p1)(1–pt–2)(1–pt–1) because (1–p1)(1–pt–2)(1–pt–1) cells weren’t programmed
The maximum achievable sum-rate is log(t+1)
Flash Memory Summit
Santa Clara, CA USA

27. The Capacity for Fixed Rate
The capacity region for two writes
C2-WOM={(R1,R2)|∃p∊[0,0.5],R1≤h(p), R2≤1-p}
When forcing R1=R2 we get h(p) = 1-p
The (numerical) solution is p = , the sum-rate is 1.54
Multiple writes: A recursive formula to calculate the maximum achievable sum-rate RF(1)=1 RF(t+1) = (t+1)root{h(zt/RF(t))-z} where root{f(z)} is the min positive value z s.t. f(z)=0
For example: RF(2) = 2root{h(z)-z} = 2 = RF(3) = 3root{h(2z/1.54)-z}=3 =1.9311
Flash Memory Summit
Santa Clara, CA USA

28. More Constructions
Shpilka, “New constructions of WOM codes using the Wozencraft ensemble”
An efficient capacity-achieving two-write construction
1st write – program any vector of weight at most m (fixed)
2nd write – instead of using one matrix, use a set of matrices such that at least one of them succeeds on the second write
Need to index the matrix for the 2nd write – negligible if the number of matrices is small
Use the Wozencraft ensemble of linear codes to construct a good set of matrices
Flash Memory Summit
Santa Clara, CA USA

29. Polar WOM Codes
A probabilistic approach to construct WOM codes which works with high probability
Similar to the one by Wu
On each write, encode more bits and write a vector that matches the bits which were already programmed
Can combine with ECC so the redundancy is used both for rewriting and error correction
Another recent construction using LDPC codes