1. George Kola Computer Sciences Department University of Wisconsin-Madison kola@cs.wisc.edu http://www.cs.wisc.edu/condor DiskRouter: A Mechanism for High Performance Large Scale Data Transfers

2. www.cs.wisc.edu/condor 2 Outline › Problem › DiskRouter Overview › Details › Real life DiskRouters › Experiments

3. www.cs.wisc.edu/condor 3 Problem SDSC to NCSA Bottleneck Bandwidth : 12.5 MBPS Latency67 ms Transfer Rate got by applications for a 1GB file Scp : 0.66MBPS GridFTP(1 stream) : 0.85 MBPS GridFTP(10 streams) : 3.52 MBPS

4. www.cs.wisc.edu/condor 4 DiskRouter Overview › Mechanism to efficiently move large amounts of data (order of terabytes) › Uses disk as a buffer to aid in large scale data transfers › Application-level overlay network used for routing › Ability to use higher level knowledge for data movement

5. www.cs.wisc.edu/condor 5 A Simple Case AB A is transferring a large amount of data to B

6. www.cs.wisc.edu/condor 6 A Simple Case AB C DiskRouter C is an intermediate node between A and B

7. www.cs.wisc.edu/condor 7 A Simple Case with DiskRouter AB C Improves performance when bandwidth fluctuation between A and C is independent of the bandwidth fluctuation between C and B DiskRouter With DiskRouter Without DiskRouter

8. www.cs.wisc.edu/condor 8 Data Mover/Distributed Cache Source writes to the closest DiskRouter and Destination receives it up from its closest DiskRouter Source Destination DiskRouter Cloud

9. www.cs.wisc.edu/condor 9 Outline › Problem › DiskRouter Overview › Details › Real life DiskRouters › Experiments

10. www.cs.wisc.edu/condor 10 Routing Between DiskRouters C need not be in the path between A and B DiskRouter A DiskRouter B DiskRouter C

11. www.cs.wisc.edu/condor 11 Network Monitoring › Uses ‘Pathrate’ for estimating network capacity › Performs actual transfers for measurement › Logging the data rate seen by different components › Generate network interface stats on the machines involved in the transfers

12. www.cs.wisc.edu/condor 12 Implementation Details › Uses multiple sockets and explicitly sets TCP buffer sizes › Overlaps disk I/O and socket I/O

13. www.cs.wisc.edu/condor 13 Client Side › Client library provided › Applications can call library functions for network I/O › Functions provided for common case file transfer (overlaps network I/O and disk I/O) › Third party transfer support

14. www.cs.wisc.edu/condor 14 Outline › Problem › DiskRouter Overview › Details › Real life DiskRouters › Experiments

15. www.cs.wisc.edu/condor 15 Real Life DiskRouters UW Madison StarLight MCS ANL INFN Italy 90 Mbps 3.3 ms 90 Mbps 5.5 ms 514 Mbps 0.85 ms 30 Mbps 126.6 ms 411 Mbps 8 ms SDSC UW Milwaukee NCSA 518 Mbps 67 ms 94 Mbps 2.7 ms

16. www.cs.wisc.edu/condor 16 Outline › Overview › Details › Real Life DiskRouters › Experiments

17. www.cs.wisc.edu/condor 17 Testing Multiroute UW Madison StarLight 90 Mbps 3.3 ms 90 Mbps 5.5 ms 411 Mbps 8 ms UW Milwaukee

18. www.cs.wisc.edu/condor 18 Multiroute Improves Performance Total Data into Starlight Data From Milwaukee Data From Madison Megabits/second

19. www.cs.wisc.edu/condor 19 SRB to Unitree Transfer Using Stork › Data movement from SDSC to NCSA via Starlight (3 TB of data had to be moved) › Integrated into Stork › Found significant performance gain

20. www.cs.wisc.edu/condor 20 Link between SDSC and NCSA StarLight SDSC NCSA 518 Mbps 67 ms 94 Mbps 2.7 ms

21. www.cs.wisc.edu/condor 21 Starlight DiskRouter Stats Data Inflow Data Outflow Memory Used Disk Used

22. www.cs.wisc.edu/condor 22 GridFTP vs DiskRouter Megabytes/second End-to-End Data Rate Seen by Stork(MBPS) vs. Time GridFTP DiskRouter

23. www.cs.wisc.edu/condor 23 A Glimpse of Performance Transfer of 1 GB file from SDSC (SanDiego) to NCSA (Urbana-Champaign) ToolTransfer Rate Scp0.66 MBPS GridFTP(1 stream) 0.85MBPS GridFTP(10 streams) 3.52 MBPS DiskRouter 10.77 MBPS

24. www.cs.wisc.edu/condor 24 Work In Progress › Computation on data streams in the DiskRouter › Ability to perform computation in the nodes attached locally to the DiskRouter › Working together with Stork to add intelligence to data movement

25. www.cs.wisc.edu/condor 25 Questions › Thanks for listening