A Row-Permutated Data Reorganization Algorithm for Growing Server-less VoD Systems Presented by Ho Tsz Kin

Agenda Background Existing solutions Row-Permutated (RP) Algorithm Multi-RP Algorithm Performance Evaluation Conclusion

Background Each node keeps balance video data blocks Nodes join the system Data must be reorganized to utilize storage and streaming capacity n0n0 n1n1 n3n3 n2n2 node n 4 joins n0n0 n1n1 n3n3 n2n n3n3

Background Data reorganization Require data block movement Consume bandwidth Should not disrupt services Achieve storage and streaming balance

Existing Solutions Round-robin Reorganization Round-robin placement policy Advantages: Perfect storage and streaming balance Drawbacks: Nearly all the data blocks must be reorganized n0n0 n1n1 n3n3 n2n2 node n 4 joins n0n0 n1n1 n3n3 n2n n3n3

Existing Solutions Randomized Reorganization Randomized placement policy Blocks are distributed to nodes randomly n0n0 n1n1 n3n3 n2n2 0 Assign to each node with equal probability n0n0 n1n1 n3n3 n2n2

Existing Solutions Reorganization Algorithm Number of nodes, N Probability of residing in same node = Probability of moving to new node = n0n0 n1n1 n3n3 n2n2 n4n4 P =

Existing Solutions n0n0 n1n1 n3n3 n2n2 node n 4 joins Randomized Reorganization Advantages: Block movement is minimized, achieve reasonable storage balance Drawbacks: Streaming load is imbalance n0n0 n1n1 n3n3 n2n2 n4n4 imbalance row

Goal Two extreme cases Round-robin Reorganization Overhead is maximum, balance streaming load Randomized Reorganization Overhead is minimum, imbalance streaming load Two Goals: Maintain balance streaming load but lower the overhead of round-robin reorganization Allow controllable tradeoff between overhead and streaming load balance

Row-Permutated (RP) Algorithm Idea: the sequence of blocks within each row is not important in streaming load Row-permutated placement policy Streaming load is still balanced 1032 n0n0 n1n1 n3n3 n2n n0n0 n1n1 n3n3 n2n2 Both maintain balanced streaming load Round-robin PlacementRow-Permutated Placement

Row-Permutated (RP) Algorithm Reorganization Algorithm Reorganize one row per iteration Identify overflow and underflow nodes Overflow if more than 1 block Underflow if no block Move excess block from overflow nodes to underflow nodes n0n0 n1n1 n3n3 n2n2 n4n4 Overflow Node Underflow Node Excess block Target row in this iteration

Row-Permutated (RP) Algorithm Perfect streaming and storage balance Significantly lower down number of block movement during reorganization n0n0 n1n1 n3n3 n2n node n 4 joins n0n0 n1n1 n3n3 n2n n4n4

Multi-RP Algorithm Tradeoff between overhead and streaming balance Control streaming balance by window size, w n0n0 n1n1 n3n3 n2n n0n0 n1n1 n3n3 n2n n4n4 w =2 Consider 2 rows

Multi-RP Algorithm Reorganization Algorithm Reorganize w rows per iteration Identify overflow and underflow nodes Overflow if more than w blocks Underflow if fewer than w blocks n0n0 n1n1 n3n3 n2n n4n4 w =2 Overflow Nodes Underflow Nodes

Multi-RP Algorithm In each overflow node Choose row with largest number of block Take blocks in this row as excess blocks Move to underflow nodes Contains smallest number of blocks in this row n0n0 n1n1 n3n3 n2n n4n4 n0n0 n1n1 n3n3 n2n n4n4 randomly

Multi-RP Algorithm Idea: Spread out blocks within row n0n0 n1n1 n3n3 n2n n4n4 row with largest number of blocks n0n0 n1n1 n3n3 n2n n4n4 n0n0 n1n1 n3n3 n2n n4n4

Performance Evaluation Experiment Details Number of data blocks = 4000 Grow from 1 node to 200 nodes Metrics Data Reorganization Overhead Number of block movement Streaming Load Balance Proportion of missing data block within one row, given that each node can only send out one block each round

Data Reorganization Overhead

Streaming load balance

Conclusion Identify the shortcomings of round-robin and randomized reorganization RP and multi-RP reorganization are proposed Perfect streaming load balance with lower overhead Controllable tradeoff between overhead and streaming load balance