1. Enabling High Speed Data Transfer in High Energy Physics
05/12/08
FDT/dCache
Enabling High Speed Data Transfer in High Energy Physics
Kamran Soomro
Caltech
May 15, 2008

2. Goals Meet the computing needs of the LHC
05/12/08
Goals
Meet the computing needs of the LHC
Manage terabyte to petabyte scientific data transfers worldwide from
Add pictures
Mention CMS

3. CERN Center PBs of Disk; Tape Robot
05/12/08
Data Grid Hierarchy
~PByte/sec
~ MBytes/sec
Online System
CERN Center PBs of Disk; Tape Robot
Tier 0 +1
Tier 1
~40 Gbps
IN2P3 Center
RAL Center
INFN Center
FNAL Center
~n x 10 Gbps
Tier 2
Network infrastructure employs two components, dCache and FDT
Many of the Tier 1’s have deployed dCache
350 TB of storage at Caltech
450 Batch slots for computing
Tier2 Center
Tier2 Center
Tier2 Center
Tier2 Center
Tier2 Center
~ Gbps
Tier 3
Tens of PB in ; 1 EB by 2015
Institute
Institute
Institute
Institute
Physics data cache
0.1 to 10 Gbps
Workstations
Tier 4

4. 05/12/08
dCache
Joint venture between the Deutsches Elektronen- Synchrotron (DESY) and the Fermi National Accelerator Laboratory (FNAL)
Mass storage system able to distribute data over a large number of heterogeneous nodes
Presents single virtual filesystem view
Provides access to Tertiary Storage System
Space management
Pool attraction (Set aside pools for specific purposes)
File-level Replication
Disk and node failure recovery
Filesystem namespace operations available through nfs v2 interface
Elaborate on Pool Attraction

5. gsidcapdoor-host1Domain
05/12/08
gsidcapdoor-host1Domain
gsigridftpdoor-host1Domain
httpdDomain
adminDoorDomain
dCacheDomain
Pool Node
PoolManager
TCP Connections
Admin Node
host1Domain
Pool Node
pnfsDomain
PnfsManager
PNFS
Door Node

6. FDT Developed at Caltech by team led by Iosif Legrand
05/12/08
FDT
Developed at Caltech by team led by Iosif Legrand
Cross platform, efficient data transfer application
Asynchronous multithreaded system
Streams a dataset continuously through TCP sockets
Parallelizes reading and writing on multiple physical devices
Parallelizes data transfer over the network between multiple TCP streams
Uses appropriate-sized buffers to match disk I/O with the network
Restores files from the buffers asynchronously
Resumes a file transfer session without loss, when needed
Pipeline transfers to avoid TCP ramp up time
Supports various security mechanisms, e.g. SSH Encryption, GSI etc
Provides hooks for monitoring tools like MonALISA
Pipeline transfers to avoid TCP ramp up time

7. FDT Client/Server Setup
05/12/08
FDT Client/Server Setup
Wide Area
Network
Mention different types of data transfers
Transfer Scenarios
Memory to memory
Disk to memory
Disk to disk
Storage to memory
Storage to disk
Storage to Storage

8. FDT @ Supercomputing ‘07 05/12/08 Managed distributed storage
Motivational slide

9. Motivation We already had an efficient data transfer tool
05/12/08
Motivation
We already had an efficient data transfer tool
US CMS has officially adopted dCache as a storage management solution
Facilitate data transfer between Tier 2 and Tier 3
Easily deployable
Manage bandwidth efficiently
May not have expertise/manpower for complicated setup

10. Approaches for integration
05/12/08
Approaches for integration
FDT Adaptor
FDT Door
FDT Preload
Java dCapJ Library
Mention the need for integration

11. FDT Adaptor Wide Area Network FDT Client FDT Server 05/12/08
Mention dcap
Mention dcap door
Explain dccp
Mention FDT runs on disk server(s)
Storage needs to be high performance disk server

12. 05/12/08
FDT Adapter Test
Transferred ~480 GB in files of GB distributed over several dozen pools from production storage.
4 Gbps for storage→disk and 5 Gbps by FDT for disk→disk (using 4 dcap doors with 4 dccp’s per door)

13. Drawbacks Extra latency of copying file to intermediate location
05/12/08
Drawbacks
Extra latency of copying file to intermediate location
Need to deploy high-speed disk server

14. Approaches for integration
05/12/08
Approaches for integration
FDT Adaptor
FDT Door
FDT Preload
Java dCapJ Library
Mention the need for integration

15. FDT Door 1. Client contacts FDT door
05/12/08
FDT Door
FDT Server
dCache Pools
FDT Client
FDT Door
FDT Server
1. Client contacts FDT door
2. Door starts up FDT servers on pools
3. Door returns file locations to client
4. Client connects with respective FDT servers for file transfer

16. Drawbacks Prevents FDT from doing intelligent scheduling
05/12/08
Drawbacks
Prevents FDT from doing intelligent scheduling
Requires multiple instances of FDT server possibly running on the same pool node
No pipelining of files
Doesn’t allow for pipelining of files
Doesn’t allow FDT to do intelligent scheduling
Not a valid approach

17. Approaches for integration
05/12/08
Approaches for integration
FDT Adaptor
FDT Door
FDT Preload
Java dCapJ Library
Mention the need for integration

18. 05/12/08
FDT Preload
Wide Area
Network
Mention LD_PRELOAD

19. FDT Preload Test Number of files = 3425 Avg rate ~4.2 Gbps 05/12/08
Explain discrepancy in units

20. Drawbacks Distribution of files is hidden from FDT
05/12/08
Drawbacks
Distribution of files is hidden from FDT
dCap C library (libdcap.so) is in not thread-safe, resulting in problems during parallel transfers
The library provides limited POSIX functionality, FDT needs more, e.g. stat replacement
Mention why files are hidden
Mention stat

21. Approaches for integration
05/12/08
Approaches for integration
FDT Adaptor
FDT Door
FDT Preload
Java dCapJ Library
Mention the need for integration

22. 05/12/08
FDT/dCapJ

23. Java dCapJ Java language implementation of dCap protocol
05/12/08
Java dCapJ
Java language implementation of dCap protocol
Uses Java NIO library for efficient data transfer
Thread-safe
Developed as an independent library for use by application programmers to interact with dCache files via java
Does not require deployment of extra hardware 25

24. FDT Storage to Disk with the Java dcapJ Library (3/9/08)‏
FDT/dCache (with dCapJ) sustains a steady [4.2] Gbps with just six 1U pool nodes + 1 disk server
4.8 TByte dataset composed of GByte files is sent in just over Hours
[Throughput drops off only as files finish] 4
3. 5 3
2. 5
Gbps 2
1. 5 1
0. 5

25. Source = dCache cluster Max number of parallel transfers = 5
05/12/08
Source = dCache cluster
Max number of parallel transfers = 5
Sink = High performance disk system
Network = WAN loop from Caltech <-> Chicago
Number of files = 260
Round Trip Time = ~160 ms
File size = 40 GB
Total data transferred = ~10 TB

26. 05/12/08
Future Work
Continue to test scalability for large dCache installations
Integrate FDT with SRM
Develop a scheduling service
Manage scheduling of transfers to minimize effects of overhead involved in opening files
Aggregate transfers queued by different users
Integrate FDT with SRM
Facilitate data transfer b/w Tier 2 and Tier 3