Peer-Assisted Content Distribution Pablo Rodriguez Christos Gkantsidis

2 Traditional Content Distribution Often, large content needs to be distributed to millions of clients: Currently: Huge server farms Infrastructure-based solutions (e.g. Akamai) slow, expensive, non scalable Server Farm

3 Content Distribution Evolution Hype Realism Growth Caching IP Multicast CDNs Akamai Enterprise CDNs Layer-7 Switches Satellite CDNs P2P Disappointment

4 Peer-Assisted Content Distribution

5 Desktop PCs can help each other! Clients become new servers Capacity increases with the number of clients Limitless scalability and fast speeds at extremely low cost!! Server Farm 4 MB file. Server 100 Mbps. Client 1 Mbps

6 Examples Updates/Critical Patches –Adding large servers and egress capacity to absorb pick load is quite expensive –Alternative solution is to delay clients »Patches do not arrive on-time Software Distribution TV On-Demand. Movie/Music downloads PodCasting Enterprise content distribution

7 P2P Content Distribution Benefits: –Dramatically improves speed –Limitless scalability –Minimum server requirements –Very cheap Challenges: –Requires incentives for cooperation –Hard to ensure end2end full connectivity –Security –Manageability –Lack of locality increases transit costs for ISPs –Asymmetric links (traffic engineering) –Variable bandwidth, peers come and go –Need for more sophisticated distribution algorithms

8 P2P Swarming File is divided into many small pieces for distribution Clients request different pieces from the server or from other clients Clients become servers for those pieces downloaded When all pieces are downloaded, clients can re-construct the whole file Server [Rodriguez, Biersack, Infocom’00]

The Challenge Server If there are many users, deciding which is the best piece to download can be very hard!!  Incorrect decisions result in low throughput, nodes not able to finish, bandwidth wasted, etc. Solutions that require to have full knowledge of who has what are non- scalable

10 Avalanche: Improving file swarming using Coding Techniques

11 Goal Provide a very fast and robust Peer-Assisted solution for the distribution of legal content Current problems in existing File Swarming solutions: Rare-blocks are hard to obtain Tit-for-tat incentive mechanisms decrease speeds Arrival of new users slows down old users Heterogeneous nodes do not interact well Same information travels repeatedly over bottleneck links Too much dependency from seeds Sudden departures can prevent peers from finishing

12 Source The Problem of Efficient Scheduling of Information Node ANode B Block 1 Block 2 Node C Block 1 Block 1, or 2, or 1  2?

13 The Avalanche Magic To solve problems of existing P2P file distribution solutions, Avalanche uses special encoding algorithms Each encoded piece has the “DNA” of all pieces in the file. => A given encoded piece can be used by any peer in place of any piece Encoded pieces are created using linear equations that involve all pieces in the file Reconstructing the file requires collecting enough encoded pieces and solving the set of mathematical equations

14 Coding in general Assume file: F = [x 1 x 2 ], where x i is a block. Define code E i (a i,1, a i,2 ) = a i,1 *x 1 + a i,2 *x 2, where a i,1, a i,2 are numbers. “Infinite” number of E i ’s. Any two linearly independent E i (a i,1, a i,2 ) can recover [x 1 x 2 ]. –Similar as solving a system of linear equations. Operations in finite fields [such as GF(2 16 )].

15 Avalanche Coding B1B1 B2B2 BnBn Server 11 22 Client A 11 22 nn E1E1 E2E2 Client B 11 22 E3E3 [Chou et al., ’03] Content is encoded at the server Clients can produce new encoded packets out of partial files nn File

16 Avalanche Robustness If server suddenly goes down (after serving the full file one), all Avalanche users are able to complete the download. Only 10% of users using typical file-swarming techniques are able to complete. Typical file-swarming systems Avalanche

17 Avalanche Download Time Finish Times Nodes (sorted by order of arrival) Avalanche Typical swarming Peers using typical file- swarming techniques that did not finish. => Much lower and predictable download times

18 No need for nodes to stay around… With Avalanche, there is no need for nodes to stay after they finish the download to help other nodes (the performance remains unchanged) Nodes stay for ever Nodes leave immediately Nodes (sorted by order of arrival) Finish Times

19 Minimum Server Requirements Less than half the server requirements compared to systems based on current file-swarming techniques.

20 Decoding Performance Avalanche trades-off better speeds and less server load for more processing power at each node File Size (MB)BlocksTime sec sec 100 2m 21 sec m 38 sec Note: Pentium III, 650MHz, 512MB RAM. Decoding time is less than 4% of the total download

21 Summary Adding resources in an arbitrary fashion is not efficient or cost effective We are witnessing a new Revolution Peer-Assisted solutions can be used by content providers to provide hugely scalable, and very fast distribution of legal content at low cost