1. Locality-Aware Content Distribution Danny Bickson, Dahlia Malkhi, David Rabinowitz

2. Lecture outline  Problem statement  Previous work  The Julia algorithm  Algorithm deployment

3. Problem Statement  Large scale content distribution  High speed data dissemination  Application layer multicast The problem: We would like to transfer a file F, from a source s to a group of n recipients.

4. Measurements  Minimize worst completion time  Minimize total work (the number of bits * link distance) over all links  Average fair sharing ratio  Connectivity

5. Naïve solutions * Client / Server * Mirroring / replication * Multicast tree

6. State-of-the-art solutions  SRM  SplitStream / Coopnet / Bayeux  Bullet  FastReplica  Zigzag / TMesh / Narada  BitTorrent

7. Julia Algorithm  Load balance  Fair sharing  Optimal finishing time  Local transfer of most parts of the file

8. Locality motivation Stretch – Overlay distance / Underlying network distance

9. Locality motivation First phase

10. Locality motivation Phase 2

11. Julia algorithm 12345678 Round 0

12. Julia algorithm 12345678 1,52,63,74,81,52,63,74,8 Round 1 – Exchange 1 part along longest links

13. Julia algorithm 12345678 1,52,63,74,81,52,63,74,8 Round 2 1,3,5,72,4,6,81,3,5,7 2,4,6,8

14. Julia algorithm 12345678 1,52,63,74,81,52,63,74,8 Round logN – exchange half of the file along shortest links 1,3,5,72,4,6,81,3,5,7 2,4,6,8 1-8

15. Comparison Summary Download time (worst case) Number of edges Total work Application multicast tree K|F|log k (n) n-1|F|D/2(n-1) SplitStream protocol |F|(log k (n)+1) (n-1)k  (|D|n 1-   FastReplica protocol ( 2-1 /k )|F|log k (n) (n-1)*(k+1)/2  (|D|n 1-   Our protocol |F| nlog 2 (n)  (|F|D log 2 (n))  =1/log 2 (k)

16. The deployment  Distance estimation  Level categorization  Node selection algorithm  Chunk selection algorithm

17. Implementation  C++ client consisting of 15,000 lines of code.  Event queue model.  Did both LAN and Planetlab experiments

18. Protocol Messages  Request file info  Reply file info  Request chunk  Reply chunk  Data / Error msgs

19. Distance estimation  Distance measurements are collected on the fly – no spare bandwidth allocated.  Nodes are categorized into 8 levels 12Mbps 21.5Mpbs 3750Kbps 4500Kbps 5250Kbps 6100Kbps 750Kbps 8Unknown

20. Node selection algorithm  Progress depended.  Up to 25% progress – connect random nodes.  Up to 50% progress – connect close nodes with probability p1.  Above 50% progress – connect close nodes with probability p2.

21. Chunks selection algorithm  Rarest first  Random  Round robin  Mixed

22. Optimizations  Pipelining of the Julia algorithm  Message batching

26. The END  Thank You!