1. Adaptive File Transfers for Diverse Environments Himabindu Pucha, Purdue University David G. Andersen, Carnegie Mellon University Michael Kaminsky, Intel Research Pittsburgh Michael Kozuch, Intel Research Pittsburgh

2. 2 Goal Correctly and efficiently transfer files in wide range of scenarios

3. 3 Goal Correctly and efficiently transfer files in wide range of scenarios Sender Receiver Network Data backup, code update Software synchronization Different network speeds file in-place Search for similar files Scenario: Data backup, code update Network peers Scenario: Software synchronization Gigabit LAN – DSL links Different disk loads

4. 4 Problem: Existing Tools Scenario-specific Tool Files in-place Other files Identical peers Peers rsync  BitTorrent  rsync-batch + BitTorrent   dsync

5. 5 Challenges Resources have widely varying performance Resource performance changes dynamically Support receivers with different initial state Do not require resources to be set up in advance

6. 6 dsync: Design dsync uses all available resources effectively Network Disk dsync scheduler

7. 7 dsync: Design Discovers available resources using exposed backpressure information From disk: “I’m busy writing, don’t read from me.” From network: “I have lots of incoming packets, don’t spend time doing IO or computation.” Schedules intelligently across available resources Disk: use a pre-computed index and/or search entire disk using heuristics Network: Schedule remaining chunks, least likely to be found on disk

8. 8 dsync: Preliminary Results dsync defers disk operations when network is faster than disk Bonus: dsync provides best of BitTorrent, rsync, scp … Throughput for 1 GB file on a 1 Gbps link

9. 9 Average download time across 45 receivers, 50% similar file in-place dsync: Preliminary Results dsync rapidly locates similar files and effectively combines them with peering dsync correctly uses backpressure to defer disk operations when network is faster than disk dsync speedup: 5x vs. rsync 2x vs. SET

10. BACKUP

11. 11 dsync: Preliminary Results dsync correctly uses backpressure to defer disk operations when network is faster than disk

12. 12 Goal Correctly and efficiently transfer files in wide range of scenarios Sender Receiver Network Data backup Code update Software synchronization Different network speeds file in-place Search for similar files Scenario: Data backupScenario: Code update Network peers Scenario: Software synchronization Gigabit LAN – DSL links Different disk loads

13. 13 Problem: Existing Tools Scenario-specific Tooldisknetwork peers rsync (files in-place)  BitTorrent  rsync-batch + BitTorrent (files in-place) ~ (all receivers in identical state)

14. 14 dsync: Design Discovers available resources using exposed backpressure information From disk: “I’m busy writing, don’t read from me.” From network: “I have lots of incoming packets, don’t spend time doing IO or computation.” Schedules intelligently across available resources Disk: use a pre-computed index and/or search entire disk using heuristics Network: Schedule remaining chunks, least likely to be found on disk

15. 15 Challenges Correctly use resources with widely varying performance characteristics Dynamically adapt to changes in resource performance Support receivers with different initial state Do not require resources to be set up in advance