Locality-Aware Content Distribution Danny Bickson, Dahlia Malkhi, David Rabinowitz

Lecture outline  Problem statement  Previous work  The Julia algorithm  Algorithm deployment

Problem Statement  Large scale content distribution  High speed data dissemination  Application layer multicast The problem: We would like to transfer a file F, from a source s to a group of n recipients.

Measurements  Minimize worst completion time  Minimize total work (the number of bits * link distance) over all links  Average fair sharing ratio  Connectivity

Naïve solutions * Client / Server * Mirroring / replication * Multicast tree

State-of-the-art solutions  SRM  SplitStream / Coopnet / Bayeux  Bullet  FastReplica  Zigzag / TMesh / Narada  BitTorrent

Julia Algorithm  Load balance  Fair sharing  Optimal finishing time  Local transfer of most parts of the file

Locality motivation Stretch – Overlay distance / Underlying network distance

Locality motivation First phase

Locality motivation Phase 2

Julia algorithm Round 0

Julia algorithm ,52,63,74,81,52,63,74,8 Round 1 – Exchange 1 part along longest links

Julia algorithm ,52,63,74,81,52,63,74,8 Round 2 1,3,5,72,4,6,81,3,5,7 2,4,6,8

Julia algorithm ,52,63,74,81,52,63,74,8 Round logN – exchange half of the file along shortest links 1,3,5,72,4,6,81,3,5,7 2,4,6,8 1-8

Comparison Summary Download time (worst case) Number of edges Total work Application multicast tree K|F|log k (n) n-1|F|D/2(n-1) SplitStream protocol |F|(log k (n)+1) (n-1)k  (|D|n 1-   FastReplica protocol ( 2-1 /k )|F|log k (n) (n-1)*(k+1)/2  (|D|n 1-   Our protocol |F| nlog 2 (n)  (|F|D log 2 (n))  =1/log 2 (k)

The deployment  Distance estimation  Level categorization  Node selection algorithm  Chunk selection algorithm

Implementation  C++ client consisting of 15,000 lines of code.  Event queue model.  Did both LAN and Planetlab experiments

Protocol Messages  Request file info  Reply file info  Request chunk  Reply chunk  Data / Error msgs

Distance estimation  Distance measurements are collected on the fly – no spare bandwidth allocated.  Nodes are categorized into 8 levels 12Mbps 21.5Mpbs 3750Kbps 4500Kbps 5250Kbps 6100Kbps 750Kbps 8Unknown

Node selection algorithm  Progress depended.  Up to 25% progress – connect random nodes.  Up to 50% progress – connect close nodes with probability p1.  Above 50% progress – connect close nodes with probability p2.

Chunks selection algorithm  Rarest first  Random  Round robin  Mixed

Optimizations  Pipelining of the Julia algorithm  Message batching

The END  Thank You!