1. BNL Facility Status and Service Challenge 3 Zhenping Liu, Razvan Popescu, Xin Zhao and Dantong Yu USATLAS Computing Facility Brookhaven National Lab

2. HEPiX Karlsruhe, Germany, May 09-13, 2005 2 Outline  Lessons learned from SC2.  Goals of BNL Service Challenges  Detailed SC3 planning  Throughput Challenge (Simple)  Network Upgrade Plan. z USATLAS dCache system at BNL. z MSS.  Tier 2 Integration Planning.  File Transfer System.  Evaluation of LFC and DPM.  Service Phase Challenge to include ATLAS applications (difficult)

3. HEPiX Karlsruhe, Germany, May 09-13, 2005 3 One day data transfer of SC2

4. HEPiX Karlsruhe, Germany, May 09-13, 2005 4 Lessons Learned From SC2  Four file transfer servers with 1 Gigabit WAN network connection to CERN.  Meet the performance/throughput (70~80MB/second disk to disk).  Enabled data transfer between dCache/SRM and CERN SRM at openlab  Design our own script to control SRM data transfer.  Enabled data transfer between BNL GridFtp servers and CERN openlab GridFtp servers controlled by Radiant software.  Many components need to be tuned  250 ms RRT, high packet dropping rate, has to use multiple TCP streams and multiple file transfers to fill up network pipe.  Sluggish parallel file I/O with EXT2/EXT3, lot of processes with “D” state, more file streams, worse the performance on file system.  Slight improvement with XFS system. Still need to tune file system parameter

5. HEPiX Karlsruhe, Germany, May 09-13, 2005 5 Goals  Network, disk, and tape service  Sufficient network bandwidth: 2Gbit/sec  Quality of service: performance: 150Mbtype/sec to Storage, upto 60 Mbytes/second to tape, Has to be done with efficiency and effectives.  Functionality/Services, high reliability, data integration, high performance  Robust file transfer service  Storage Servers  File Transfer Software (FTS)  Data Management software (SRM, dCache)  Archiving service: tapeservers, taperobots, tapes, tapedrives,  Sustainability  Weeks in a row un-interrupted 24/7 operation  Involve ATLAS Experiment Applications

6. HEPiX Karlsruhe, Germany, May 09-13, 2005 6 BNL network Topology

7. HEPiX Karlsruhe, Germany, May 09-13, 2005 7 Network Upgrade Status  WAN connection OC-48.  Dual GigE links connect the BNL boarder router to the Esnet router.  Work on LAN upgrade from 1 GigE to 10 GigE was completed: June 10, 2005. USATLAS network connection to BNL campus network was upgraded to 3x1Gbps.  USATLAS network feed to BNL campus network was separated from RHIC Computing Facility.

8. HEPiX Karlsruhe, Germany, May 09-13, 2005 8 BNL Storage Element: dCache System TAllows transparent access to large amount of data files distributed on disk in dCache pools or stored on HPSS. q Provides the users with one unique name-space for all the data files.  Significantly improve the efficiency of connected tape storage systems, through caching, i.e. gather & flush, and scheduled staging techniques. TClever selection mechanism. q The system determines whether the file is already stored on one or more disks or on HPSS. q The system determines the source or destination dCache pool based on storage group and network mask of clients, also CPU load and disk space, configuration of the dcache pools. TOptimizes the throughput to and from data clients as well as balances the load of the connected disk storage nodes by dynamically replicating files upon the detection of hot spots. TTolerant against failures of its data servers. TVarious access protocols, including gridftp, SRM and dccp.

9. HEPiX Karlsruhe, Germany, May 09-13, 2005 9 Read pools DCap doors SRM door doors GridFTP doors doors Control Channel write pools Data Channel DCap Clients Pnfs ManagerPool Manager HPSS GridFTP Clientsd SRM Clients Oak Ridge Batch system DCache System BNL dCache Architecture

10. HEPiX Karlsruhe, Germany, May 09-13, 2005 10 dCache System, Continued TBNL USATLAS dCache system works as a disk caching system as a front end for Mass Storage System TCurrent configuration: Total 72 nodes with 50.4 TB disks: q Core server nodes, database server q Internal/External Read pool: 65 x 49.45 TB q Internal write pool nodes 4 x 532 GB q External write pool nodes 2 x 420 GB q dCache version: V1.2.2.7-2 q Access protocols: GridFTP, SRM, dCap, gsi-dCap

11. HEPiX Karlsruhe, Germany, May 09-13, 2005 11 Immediate dCache Upgrade June 22  Existing dCache has 50 TB data storage.  230 new dual-CPU 3.4 Ghz dell hosts will be on-site on May/11/2005  2 x 250G SATA drives  2GB memory and dual Gigbit on-board ports  These hosts will be put into one dCache system. We might evolve the system based on our experience of running such a scale, i.e. more dCache instances.  One of system will be used to SC3. The disk pool nodes will be connected directly to ATLAS router which has 3G uplink.  SL3 will be installed on all these dell hosts.  File System to be installed: XFS, need to tune to improve disk utilization per host.

12. HEPiX Karlsruhe, Germany, May 09-13, 2005 12 BNL ATLAS MSS  Two 9940B tape drivers. Data transfer rate is between 10MB~30MB/second. These two tape drives are saturated with daily USATLAS production.  200 GB tapes.  We need to borrow tape drives from other BNL in- house experiments on July to meet 60MByte/second performance target.

13. HEPiX Karlsruhe, Germany, May 09-13, 2005 13 File Transfer Service ATLAS sees benefits on trying gLite FTS as soon as possible  To see ASAP whether it meet data transfer requirements  Data transfer requires significant effort to ramp up, learn from SC2  Help debugging gLite FTS  Transfers between Tier 0, Tier 1 and a few Tier 2.  A real usage with ROME production data.  Uniform low-level file transfer layer to interface with several implementations of SRM: dCache/SRM, DPM, even vanilla GridFtp.  Xin deployed the FTS service. Xin has done successfully the data transfer test with FTS.  We are ready for the Prime Time of July 1,2005.

14. HEPiX Karlsruhe, Germany, May 09-13, 2005 14 Tier 2 Plans  Choose two USATLAS tier 2 sites: University of Chicago/IU and Boston University.  Each site can choose to deploy dCache/DPM serverswith SRM v.1 interface as appropriate.  gLite FTS (file transfer service) will transfer data from BNL to each of two chosen sites in the speed of 75M byte/second.  Files will be kept in BNL Tier 1 dCache until they are read once to Tier 2 center.

15. HEPiX Karlsruhe, Germany, May 09-13, 2005 15 ATLAS and SC3 Service Phase  September  ATLAS release 11 (mid September)  Will include use of conditions database and COOL  We intend to use COOL for several sub-detectors  Not clear how many sub-detectors will be ready  Not clear as well how we will use COOL  Central COOL database or COOL distributed database  Debug scaling for distributed conditions data access calibration/alignment, DDM, event data distribution and discovery  Tier 0 exercise testing  A dedicated server is requested for the initial ATLAS COOL service  Issues on FroNtier are still under discussion and ATLAS is interested  Data can be thrown away.

16. HEPiX Karlsruhe, Germany, May 09-13, 2005 16 ATLAS & SC3 Service Phase  April-July: Preparation phase  Test of FTS (“gLite-SRM”)  Integration of FTS with DDM  July: Scalability tests (commissioning data; Rome Physics workshop data)  September: test of new components and preparation for real use of the service  Intensive debugging of COOL and DDM  Prepare for “scalability” running  Mid-October  Use of the Service  Scalability tests of all components (DDM)  Production of real data (MonteCarlo; Tier-0; …)  Later  “continuous” production mode  Re-processing  Analysis

17. HEPiX Karlsruhe, Germany, May 09-13, 2005 17 Conclusion  Storage Element and network go well with upgrade.  The whole chain of system will be tuned before the end of June.  Continue to work with USATLAS Tier 2 sites on SC3.  Continue to discuss on how the experiment software can be involved in the second phase of SC.

18. HEPiX Karlsruhe, Germany, May 09-13, 2005 18 Future Work After SC3 Phase 1  Deploy DQ for ATLAS Data Management  Dataset File Catalog: LFC, Globus Replica catalog, or Pool File Catalog  One File Catalog (LFC?) at each Tier 2 site to catalog site files and collection of files.  Storage Element and SRM for Tier 2  File Transfer Tools:  Does FTS satisfy out data transfer requirement after one month throughput test?  FTS data transfer among more sites within the control of DQ.  Prepare for integrating ATLAS applications with Grid middleware/DMS exercised in Phase 1 of SC3.

19. HEPiX Karlsruhe, Germany, May 09-13, 2005 19 Thank You!