1. Efficient Large Scale Content Distribution WDAS 2004 By Danny Bickson, Dahlia Malkhi, David Rabinowitz

2. Lecture outline Problem definition Problem definition Previous work Previous work Julia algorithm Julia algorithm Comparison to other algorithms Comparison to other algorithms Future directions Future directions Demonstration Demonstration

3. Problem Statement Several names: Several names: Large scale content distributionLarge scale content distribution High speed data disseminationHigh speed data dissemination Application multicastApplication multicast

4. Problem model Network of N nodes, network fully connected. Network of N nodes, network fully connected. A single source node S has complete file Foo of size |Foo| bits. A single source node S has complete file Foo of size |Foo| bits. All nodes want to download the file Foo. All nodes want to download the file Foo. All links have the same bandwidth (However not the same cost). All links have the same bandwidth (However not the same cost).

5. Performance Analysis Common criteria for comparison between algorithms: Common criteria for comparison between algorithms: Download timeDownload time ConnectivityConnectivity

6. Locality motivation – Tree example

7. Locality Motivation – Tree example

8. Performance Analysis Fair Sharing- upload/ download ratio Fair Sharing- upload/ download ratio Work – number of bits X link distance Work – number of bits X link distance All criteria are taken in the worst case. All criteria are taken in the worst case.

9. Naïve solutions Application Multicast tree Application Multicast tree Download time: klog k (n) Connectivity: n-1 Fair sharing: k Work: |Foo|(D/2)(n-1) Locality awareness improves this to be : |Foo|Dn 1-  1/log(k)

10. FastReplica Cherkasova,Lee -HP Labs - 2003 Phase 1 Phase 1 Source Clients

11. FastReplica Cherkasova,Lee -HP Labs - 2003 Phase 2 Phase 2 Source Clients

12. FastReplica Cherkasova,Lee -HP Labs - 2003 Scalability? Scalability?

13. FastReplica Cherkasova,Lee -HP Labs - 2003 Scalability? Scalability? Time: O(|Foo|log k (n)) Connectivity: (n-1)(k-1)/2 Fair sharing: 1 Work: |Foo|Dn 1-, =1/log(k)

14. SplitStream Castro, Kermarrec, Ms Research, Rice - 2003 Time: O(|Foo|log k (n)) Connectivity: (n-1)k Fair sharing: 1 Work: |Foo|Dn 1-, =1/log(k)

15. Our approach Previous approaches do not take locality into account. Previous approaches do not take locality into account. Enforce “fair sharing” / load balancing Enforce “fair sharing” / load balancing Reduce download time to optimal Reduce download time to optimal

16. Julia algorithm motivation: “Divide and conquer” First phase

17. 2 nd phase Julia algorithm motivation: “Divide and conquer”

18. Julia algorithm 12345678 Round 0

19. Julia algorithm 12345678 1,52,63,74,81,52,63,74,8 Round 1 – Exchange 1 part along longest links

20. Julia algorithm 12345678 1,52,63,74,81,52,63,74,8 Round 2 – Exchange 2 parts along half length links 1,3,5,72,4,6,81,3,5,7 2,4,6,8

21. Julia algorithm 12345678 1,52,63,74,81,52,63,74,8 Round logN – exchange half of the file along shortest links 1,3,5,72,4,6,81,3,5,7 2,4,6,8 1-8

22. Algorithm properties Load balance / Fair sharing Load balance / Fair sharing Optimal finishing time Optimal finishing time Local transfer of most parts of the file Local transfer of most parts of the file

23. Comparison Summary Download time Number of edges Total work Fair sharing Application multicast tree K|F|log k (n) n-1|F|(D/2)(n-1)k SplitStream protocol |F|(log k (n)+1) (n-1)k (|F|Dn 1-   FastReplica protocol (2-1/k)|F|log k (n) (n-1)*(k+1)/2 (|F|Dn 1-  1 Our protocol 2|F|nlog(n) (|F|D log(n)) 1  =1/log 2 (k)

24. Implementation issues and future work Fuzzy vs. structured network Fuzzy vs. structured network How to measure distances in the network How to measure distances in the network Fault tolerance, dynamic membership Fault tolerance, dynamic membership Scalability Scalability Demonstration of software tools Demonstration of software tools

25. THE END Thank you! Thank you!