1. Redundant Data Update in Server-less Video-on-Demand Systems Presented by Ho Tsz Kin

2. Agenda Background Redundant Data Regeneration Sequential Redundant Data Update(SRDU) Update Overhead Analysis Performance Evaluation Conclusion and Future Works

3. Background SLVoD System Data Nodes Redundancy Nodes Receiver

4. Background Data Placement v0v0 v1v1 v2v2 v3v3 c 0,0 c 0,1 d2d2 d1d1 d0d0 d3d3 r0r0 r1r1 v4v4 v8v8 v 12 c 1,0 c 2,0 c 3,0 c 1,1 c 2,1 c 3,1 v5v5 v6v6 v7v7 v9v9 v 10 v 11 v 13 v 14 v 15 v 16 c 4,0 c 4,1 v 17 v 18 v 19 Same Parity Group Data NodesRedundancy Nodes Q-bytes data blocks Q-bytes redundant data blocks

5. Background New nodes join the system Data blocks are reorganized to utilize storage and streaming capacity Data blocks in Parity Group change Redundant data blocks need to update/re-compute v0v0 v1v1 v2v2 v3v3 c’ 0,0 c’ 0,1 d2d2 d1d1 d0d0 d3d3 r0r0 r1r1 v5v5 v 10 v 15 c’ 1,0 c’ 2,0 c’ 3,0 c’ 1,1 c’ 2,1 c’ 3,1 v6v6 v7v7 v8v8 v 11 v 12 v 13 v 16 v 17 v 18 v4v4 d4d4 v9v9 v 14 v 19 Data blocks in parity group CHANGE! Redundant data blocks CHANGE!

6. Redundant Data Regeneration Regenerate new redundant blocks by all the data blocks Data blocks are distributed among nodes Send all the data blocks to redundancy node Compute the new redundant blocks in redundancy node Overhead is too high Redundancy Node Data Nodes

7. Redundant Data Regeneration Data blocks are stored in central archive server Regenerate all the new redundant data in server Send to redundancy nodes Server may be costly, not desirable, not feasible Redundancy Node Server

8. Sequential Redundant Data Update Idea Old redundant data blocks contain valuable information on the data blocks Data blocks are different after reorganization But some blocks in parity group is the SAME Possible to reuse for linear code v0v0 v1v1 v2v2 v3v3 c’ 0,0 c’ 0,1 v4v4 New Redundant data blocks v0v0 v1v1 v2v2 v3v3 c 0,0 c 0,1 Old Redundant data blocks d2d2 d1d1 d0d0 d3d3 r0r0 r1r1 d4d4 d2d2 d1d1 d0d0 d3d3 r0r0 r1r1

9. Sequential Redundant Data Update Linear erasure correction code Can be analyzed using linear algebra Linear matrix multiplications in computing redundant data Reed-Solomon Erasure Correction Code Redundant data by Vandermonde matrix Number of nodes, N Number of redundancy node, h Element at row i, column j = j i-1 Data from data node 0 Redundant data in redundancy node 0

10. Rounding or truncating would prevent a correct reconstruction of source data 1 / 3 = 0.3333, decoding fails Arithmetic over Galois Fields Finite Fields Finite elements Add, subtract, multiply, divide similar to algebra Closed under these operations GF(2 w ) are integers from zero to 2 w -1 Fixed bits to represent an element Assume a Q-bytes data block can be represented by a field element Sequential Redundant Data Update

11. Sequential Redundant Data Update(SRDU) consists of 3 techniques Direct reuse of old redundant data Parity group reshuffling Caching of transmitted data Sequential Redundant Data Update

12. SRDU – Direct Reuse v0v0 v1v1 v2v2 v3v3 c’ 0,0 c’ 0,1 v4v4 v0v0 v1v1 v2v2 v3v3 c 0,0 c 0,1 d2d2 d1d1 d0d0 d3d3 r0r0 r1r1 d4d4 d2d2 d1d1 d0d0 d3d3 r0r0 r1r1 Only need to send v 4, Save 4 blocks Construct 1 st parity group Before node addition After node addition

13. SRDU – Direct Reuse Construct 2 nd parity group Before node addition v5v5 v6v6 v7v7 v8v8 c’ 1,0 c’ 1,1 v9v9 v4v4 v5v5 v6v6 v7v7 c 1,0 c 1,1 d2d2 d1d1 d0d0 d3d3 r0r0 r1r1 d4d4 d2d2 d1d1 d0d0 d3d3 r0r0 r1r1 Terms are different. Direct Reuse Fails! After node addition

14. SRDU – Parity Group Reshuffling The order during encoding is not important Parity group reshuffling Before node addition After node addition Only need to send v 4, v 8 and v 9 Save 2 blocks

15. SRDU – Parity Group Reshuffling Choices of old redundant data for reusing Before node addition After node addition v5v5 v6v6 v7v7 v8v8 c’ 1,0 c’ 1,1 v9v9 v8v8 v9v9 v 10 v 11 c 2,0 c 2,1 d2d2 d1d1 d0d0 d3d3 r0r0 r1r1 d4d4 d2d2 d1d1 d0d0 d3d3 r0r0 r1r1 5 blocks needed, More overhead

16. SRDU - Caching Caching of transmitted data in constructing previous redundant block Reuse them to construct current redundant blocks Construct 1 st parity group Construct 2 nd parity group v 4 is cached in the redundancy node v 4 can be reused from locally cached data, Only v 8 and v 9 need to send

17. SRDU - Caching The buffer for caching is at most (N-h) Data blocks from earlier than (N-h) blocks are not useful for reusing Construct the 4 th parity group c 4,1 c 3,1 All previous blocks are useless c’ 3,1 At most 4 data blocks are cached Old redundant block New redundant block

18. SRDU - Summary For construction of each new redundant block Reuse old redundant block by Direct reuse Parity group reshuffling Caching of transmitted data Select the one with lowest overhead to reuse Send the required blocks only

19. Update Overhead Analysis Total data blocks, B Redundant Data Regeneration Data blocks are distributed among nodes Total overhead = B + (h-1) * B/(N-h) All data blocks are sent, overhead = B All redundant data are constructed here overhead = B / (N-h)

20. Update Overhead Analysis Data blocks are stored in central archive server Total overhead = h * B/(N-h) overhead = B / (N-h) All redundant data are constructed in server

21. Update Overhead Analysis Sequential Redundant Data Update(SRDU) Total overhead = block movement under SRDU + (h -1) * B/(N-h) Data blocks are sent under decision of SRDU Compute partial result of each new redundant blocks overhead = B / (N-h) Combine the old redundant data and received partial result

22. Update Overhead Analysis Redundant data Regeneration Data blocks are distributed among nodes Total overhead = B + (h-1) * B/(N-h) Data blocks are stored in central archive server Total overhead = B/(N-h) + (h-1) * B/(N-h) Sequential Redundant Data Update(SRDU) Total overhead = blocks movement under SRDU + (h-1) * B/(N-h) Sufficient to consider overhead for one redundant update as performance metric

23. Performance Evaluation Update overhead in continuous system growth Number of data blocks = 40000 Grow from 1 node to 400 nodes, one node added each

24. Performance Evaluation Update overhead in batched redundancy update Defer update until a fixed number of nodes, W Initial system size = 80, W ranges from 1 to 200

25. Conclusion Identify the redundant data update problem Investigate redundant data regeneration Propose SRDU to reduce the overhead substantially (about 75%) Further reduce the overhead by batched redundancy update (about 97% for a batch size of 10)

26. Future Works Distributed Redundant Data Update and its scheduling Mathematical Modeling Optimality of redundant data update Redundant Data Addition