1. Storage at the ATLAS Great Lakes Tier-2 Tier-2 Storage Administrator Talks Shawn McKee / University of Michigan OSG Storage ForumShawn McKee1

2. AGLT2 Storage: Outline TOverview of AGLT2 T Description, hardware and configuration TStorage Implementation T Original options: AFS, NFS, Lustre, dCache T Choose dCache (broad support in USATLAS/OSG) TIssues T Monitoring needed T Manpower to maintain (including creating custom cron scripts) T Reliability and performance TFuture? T NFSv4(.x), Lustre (1.8.1+), Hadoop OSG Storage Forum2Shawn McKee

3. The ATLAS Great Lakes Tier-2 TWithin the US there are five Tier-2 computing centers, most split over two physical locations. They support “production” and user analysis tasks. TThe ATLAS Great Lakes Tier-2 (AGLT2) is hosted at the University of Michigan and Michigan State University. TAGLT2 design goals: q Incorporate 10GE networks for high-performance data transfers q Utilize 2.6 custom kernels (UltraLight) and SL[C]4 OS (soon to SL[C]5) q Deploy inexpensive high-capacity storage systems with large partitions using the XFS filesystem --- Still must address SRM q Take advantage of MiLR to the extent possible OSG Storage Forum3Shawn McKee

4. AGLT2 Storage Node Information TAGLT2 has 13 dCache storage nodes distributed between MSU and UM with either 40 or 52 useable TB each. TTotal dCache production storage is 500TB TWe use space-tokens to manage space allocations for ATLAS and currently have 7 space-token areas. TWe use XFS for the filesystem on our 50 pools. Each pool varies from 10-20 TB in size. TdCache version is 1.9.2.5 and we are running Chimera TWe also use AFS (1.4.10) and NFSv3 for storage OSG Storage ForumShawn McKee4

5. AGLT2 Storage Node UMFS05 OSG Storage Forum5Shawn McKee Current Node 2xE5450 processors 2xE5450 processors 32GB, 10GE Myricom CX4 1TB disks (15 disks/shelf) 4xRAID6 array (13TB each)

6. AGLT2 Network Architecture Example TGood pricing from Dell for access layer switches: q Managed with 10GE ports, lots of 10/100/1000 copper ports q QoS and layer 3 capabilities, redundant power supply OSG Storage Forum6Shawn McKee

7. dCache at AGLT2 TChoose dCache because BNL and OSG both supported it. TReally using a system outside of its original design purpose q We wanted a way to tie multiple storage locations into a single user accessible name-space providing unified storage. q dCache was intended as a front-end for a HSM system TLots of manpower and attention has been required q Significant effort required to “watch for” and debug problems early q For a while we felt like the little Dutch boy…every time we fixed a problem, another popped up! q 1.9.2.5/Chimera has improved the situation OSG Storage ForumShawn McKee7

8. AGLT2 Lustre Storage TWe have been exploring options for alternative storage systems. BestMan provides an SRM for a posix-filesystem TLustre was originally tested by us in 2007: Too early! q Recently setup a 200TB (4 server) Lustre configuration with dual MDT+MGS headnodes configured with heartbeat using V1.6.6 z Performance very good (faster for clients than /tmp) z Some “issues” with huge load on one storage server q Need 2.6.24 (or 2.6.30) support for “patchless” client. Should be available in v1.8.1. q Planning to re-test once we have the needed kernel support OSG Storage ForumShawn McKee8

9. Monitoring Storage TMonitoring our Tier-2 is a critical part of our work and storage monitoring is one of the most important aspects TWe use a number of tools to track status: q Cacti q Ganglia q Custom web pages q Brian’s dCache billing web interface q Email alerting (low space, failed auto-tests, specific errors noted) TSee http://head02.aglt2.org/cgi-bin/dashboard or https://hep.pa.msu.edu/twiki/bin/view/AGLT2/Monitors http://head02.aglt2.org/cgi-bin/dashboard https://hep.pa.msu.edu/twiki/bin/view/AGLT2/Monitorshttp://head02.aglt2.org/cgi-bin/dashboard https://hep.pa.msu.edu/twiki/bin/view/AGLT2/Monitors OSG Storage ForumShawn McKee9

10. Scripts to Manage Storage TAs we have found problems we sometimes needed to automate the solution q Rephot – Automated replica creation developed by Wenjing Wu scans for “hot” files and automatically adds/removes replicas q Pool-balancer – Rebalances pools within a group TNumerous scripts (cron) for maintenance q Directory ownership q Consistency checking, including Adler32 checksum verification q Repairing inconsistencies q Tracking usage (real space and space-token) q Auto-loading PNFSID into Chimera DB for faster access OSG Storage ForumShawn McKee10

11. AGLT2 Storage Issues TInformation is scattered across server nodes and log files TJava based components have verbose output which makes it hard to find the real problem TWith dCache we find the large number of components have a large phase-space for problems TErrors are not always indicative of the real problem… TIs syslog-ng a part of the solution? TCan error messages be significantly improved? OSG Storage ForumShawn McKee11

12. Benchmarking and Optimization TWe have spent some time trying to optimize our storage performance. TSee https://hep.pa.msu.edu/twiki/bin/view/AGLT2/IOTestOnRaid Systems https://hep.pa.msu.edu/twiki/bin/view/AGLT2/IOTestOnRaid Systems https://hep.pa.msu.edu/twiki/bin/view/AGLT2/IOTestOnRaid Systems TExplored system/kernel/network and I/O tunings for our hardware. Have achieved good performance for single read/write (>700MB/sec) per partition. Multiple readers/writers at a few hundred MB/sec. OSG Storage ForumShawn McKee12

13. AGLT2 dCache Pool Transfer Rates STEP09 OSG Storage ForumShawn McKee13 Avg 616 MBytes/sec over 8 days

14. Future AGLT2 Storage TWe are interested in exploring BestMan+X where ‘X’ is q NFSv4[.x] q Lustre 1.8.x q Hadoop TWe are interested in the following characteristics for each option: q Expertise required to install/configure q Manpower required to maintain q Robustness over time q Performance (single and multiple read/writer) TOur plan is to test these during the rest of 2009 OSG Storage ForumShawn McKee14

15. ?Questions? OSG Storage ForumShawn McKee15

16. Backup Slides OSG Storage Forum16Shawn McKee

17. AGLT2 “Server” 10GE Activity STEP09 OSG Storage ForumShawn McKee17 Shown are the aggregated graphs for our 10GE storage servers during STEP09 (Units: Bytes/sec) Results are a combination of local and remote traffic

18. MiLR 10GE Protected Network TWe have a single “/23” network for the AGL-Tier2 q Internally each site (UM/MSU) has a /24 TOur network will have 3 10GE wavelengths on MiLR in a “triangle” q Loss of any of the 3 waves doesn’t impact connectivity for both sites OSG Storage Forum18Shawn McKee