BNL Facility Status and Service Challenge 3 HEPiX Karlsruhe, Germany May 9~13, 2005 Zhenping Liu, Razvan Popescu, and Dantong Yu USATLAS/RHIC Computing Facility Brookhaven National Lab

HEPiX Karlsruhe, Germany, May 09-13, Outline  Lessons learned from SC2.  Goals of BNL Service Challenges  Detailed SC3 planning  Throughput Challenge (Simple)  Network Upgrade Plan z USATLAS dCache system at BNL z MSS  Tier 2 Integration Planning  File Transfer System  Service Phase Challenge to include ATLAS applications (difficult)

HEPiX Karlsruhe, Germany, May 09-13, One day data transfer of SC2

HEPiX Karlsruhe, Germany, May 09-13, Lessons Learned From SC2  Four file transfer servers with 1 Gigabit WAN network connection to CERN.  Meet the performance/throughput challenges (70~80MB/second disk to disk).  Enabled data transfer between dCache/SRM and CERN SRM at openlab  Design our own script to control SRM data transfer.  Enabled data transfer between BNL GridFtp servers and CERN openlab GridFtp servers controlled by Radiant software.  Many components need to be tuned  250 ms RRT, high packet dropping rate, has to use multiple TCP streams and multiple file transfers to fill up network pipe.  Sluggish parallel file I/O with EXT2/EXT3, lot of processes with “D” state, more file streams, worse the performance on file system.  Slight improvement with XFS system. Still need to tune file system parameter

HEPiX Karlsruhe, Germany, May 09-13, Goals  Network, disk, and tape service  Sufficient network bandwidth: 2Gbit/sec  Quality of service: performance: 150Mbtype/sec to Storage, upto 60 Mbytes/second to tape, Has to be done with efficiency and effectives.  Functionality/Services, high reliability, data integration, high performance  Robust file transfer service  Storage Servers  File Transfer Software (FTS)  Data Management software (SRM, dCache)  Archiving service: tapeservers, taperobots, tapes, tapedrives,  Sustainability  Weeks in a row un-interrupted 24/7 operation  Involve ATLAS Experiment Applications

HEPiX Karlsruhe, Germany, May 09-13, BNL network Topology

HEPiX Karlsruhe, Germany, May 09-13, Network Upgrade Status and Plan  WAN connection OC-48.  Dual GigE links connect the BNL boarder router to the Esnet router.  Work on LAN upgrade from 1 GigE to 10 GigE, date to complete: Middle of June, 2005

HEPiX Karlsruhe, Germany, May 09-13, BNL Storage Element: dCache System TAllows transparent access to large amount of data files distributed on disk in dCache pools or stored on HPSS. q Provides the users with one unique name-space for all the data files.  Significantly improve the efficiency of connected tape storage systems, through caching, i.e. gather & flush, and scheduled staging techniques. TClever selection mechanism. q The system determines whether the file is already stored on one or more disks or on HPSS. q The system determines the source or destination dCache pool based on storage group and network mask of clients, also CPU load and disk space, configuration of the dcache pools. TOptimizes the throughput to and from data clients as well as balances the load of the connected disk storage nodes by dynamically replicating files upon the detection of hot spots. TTolerant against failures of its data servers. TVarious access protocols, including gridftp, SRM and dccp.

HEPiX Karlsruhe, Germany, May 09-13, Read pools DCap doors SRM door doors GridFTP doors doors Control Channel External Write Pools Internal write pools Data Channel DCap Clients Pnfs ManagerPool Manager HPSS GridFTP Clientsd SRM Clients Oak Ridge Batch system DCache System BNL dCache Architecture

HEPiX Karlsruhe, Germany, May 09-13, dCache System, Continued TBNL USATLAS dCache system works as a disk caching system as a front end for Mass Storage System TCurrent configuration: Total 72 nodes with 50.4 TB disks: q Core server nodes, database server q Internal/External Read pool: 65 x TB q Internal write pool nodes 4 x 532 GB q External write pool nodes 2 x 420 GB q dCache version: V q Access protocols: GridFTP, SRM, dCap, gsi-dCap

HEPiX Karlsruhe, Germany, May 09-13, Immediate dCache Upgrade  Existing dCache has 50 TB data storage.  288 new dual-CPU 3.4 Ghz dell hosts will be on-site on May/11/2005  2 x 250G SATA drives  2GB memory and dual Gigbit on-board ports  These hosts will be split into more than two dCache system.  One of system will be used to SC3. The disk pool nodes will be connected directly to ATLAS router which has 10 G uplink.  SL3 will be installed on all these dell hosts.  File System to be installed: XFS, need to tune to improve disk utilization per host.

HEPiX Karlsruhe, Germany, May 09-13, BNL ATLAS MSS  Two 9940B tape drivers. Data transfer rate is between 10MB~30MB/second. These two tape drives are saturated with daily USATLAS production.  200 GB tapes.  We need to borrow tape drives from other BNL in- house experiments on July to meet 60MByte/second performance target.

HEPiX Karlsruhe, Germany, May 09-13, File Transfer Service ATLAS sees benefits on trying gLite FTS as soon as possible  To see ASAP whether it meet data transfer requirements  Data transfer requires significant effort to ramp up, learn from SC2  Help debugging gLite FTS  Transfers between Tier 0, Tier 1 and a few Tier 2.  A real usage with ROME production data.  Uniform low-level file transfer layer to interface with several implementations of SRM: dCache/SRM, DPM, even vanilla GridFtp.  Xin deployed the FTS service. Xin has done successfully the data transfer test with FTS.  We are ready for the Prime Time of July 1,2005.

HEPiX Karlsruhe, Germany, May 09-13, Tier 2 Plans  Choose two USATLAS tier 2 sites.  Each site will deploy DPM server as storage element with SRM interface.  gLite FTS (file transfer service) will transfer data from BNL to each of two chosen sites in the speed of 75M byte/second.  Files will be kept in BNL Tier 1 dCache until they are read once to Tier 2 center.

HEPiX Karlsruhe, Germany, May 09-13, ATLAS and SC3 Service Phase  September  ATLAS release 11 (mid September)  Will include use of conditions database and COOL  We intend to use COOL for several sub-detectors  Not clear how many sub-detectors will be ready  Not clear as well how we will use COOL  Central COOL database or COOL distributed database  Debug scaling for distributed conditions data access calibration/alignment, DDM, event data distribution and discovery  Tier 0 exercise testing  A dedicated server is requested for the initial ATLAS COOL service  Issues on FroNtier are still under discussion and ATLAS is interested  Data can be thrown away.

HEPiX Karlsruhe, Germany, May 09-13, ATLAS & SC3 Service Phase  April-July: Preparation phase  Test of FTS (“gLite-SRM”)  Integration of FTS with DDM  July: Scalability tests (commissioning data; Rome Physics workshop data)  September: test of new components and preparation for real use of the service  Intensive debugging of COOL and DDM  Prepare for “scalability” running  Mid-October  Use of the Service  Scalability tests of all components (DDM)  Production of real data (MonteCarlo; Tier-0; …)  Later  “continuous” production mode  Re-processing  Analysis

HEPiX Karlsruhe, Germany, May 09-13, Conclusion  Storage Element and network go well with upgrade.  The whole chain of system will be tuned before the end of May.  Wait for FTS software to control data transfer.  Talk with USATLAS Tier 2 sites to participate SC3.  Discuss on how the experiment software can be involved.

HEPiX Karlsruhe, Germany, May 09-13, Thank You!