Storage at the ATLAS Great Lakes Tier-2 Tier-2 Storage Administrator Talks Shawn McKee / University of Michigan OSG Storage ForumShawn McKee1

AGLT2 Storage: Outline TOverview of AGLT2 T Description, hardware and configuration TStorage Implementation T Original options: AFS, NFS, Lustre, dCache T Choose dCache (broad support in USATLAS/OSG) TIssues T Monitoring needed T Manpower to maintain (including creating custom cron scripts) T Reliability and performance TFuture? T NFSv4(.x), Lustre (1.8.1+), Hadoop OSG Storage Forum2Shawn McKee

The ATLAS Great Lakes Tier-2 TWithin the US there are five Tier-2 computing centers, most split over two physical locations. They support “production” and user analysis tasks. TThe ATLAS Great Lakes Tier-2 (AGLT2) is hosted at the University of Michigan and Michigan State University. TAGLT2 design goals: q Incorporate 10GE networks for high-performance data transfers q Utilize 2.6 custom kernels (UltraLight) and SL[C]4 OS (soon to SL[C]5) q Deploy inexpensive high-capacity storage systems with large partitions using the XFS filesystem --- Still must address SRM q Take advantage of MiLR to the extent possible OSG Storage Forum3Shawn McKee

AGLT2 Storage Node Information TAGLT2 has 13 dCache storage nodes distributed between MSU and UM with either 40 or 52 useable TB each. TTotal dCache production storage is 500TB TWe use space-tokens to manage space allocations for ATLAS and currently have 7 space-token areas. TWe use XFS for the filesystem on our 50 pools. Each pool varies from TB in size. TdCache version is and we are running Chimera TWe also use AFS (1.4.10) and NFSv3 for storage OSG Storage ForumShawn McKee4

AGLT2 Storage Node UMFS05 OSG Storage Forum5Shawn McKee Current Node 2xE5450 processors 2xE5450 processors 32GB, 10GE Myricom CX4 1TB disks (15 disks/shelf) 4xRAID6 array (13TB each)

AGLT2 Network Architecture Example TGood pricing from Dell for access layer switches: q Managed with 10GE ports, lots of 10/100/1000 copper ports q QoS and layer 3 capabilities, redundant power supply OSG Storage Forum6Shawn McKee

dCache at AGLT2 TChoose dCache because BNL and OSG both supported it. TReally using a system outside of its original design purpose q We wanted a way to tie multiple storage locations into a single user accessible name-space providing unified storage. q dCache was intended as a front-end for a HSM system TLots of manpower and attention has been required q Significant effort required to “watch for” and debug problems early q For a while we felt like the little Dutch boy…every time we fixed a problem, another popped up! q /Chimera has improved the situation OSG Storage ForumShawn McKee7

AGLT2 Lustre Storage TWe have been exploring options for alternative storage systems. BestMan provides an SRM for a posix-filesystem TLustre was originally tested by us in 2007: Too early! q Recently setup a 200TB (4 server) Lustre configuration with dual MDT+MGS headnodes configured with heartbeat using V1.6.6 z Performance very good (faster for clients than /tmp) z Some “issues” with huge load on one storage server q Need (or ) support for “patchless” client. Should be available in v q Planning to re-test once we have the needed kernel support OSG Storage ForumShawn McKee8

Monitoring Storage TMonitoring our Tier-2 is a critical part of our work and storage monitoring is one of the most important aspects TWe use a number of tools to track status: q Cacti q Ganglia q Custom web pages q Brian’s dCache billing web interface q alerting (low space, failed auto-tests, specific errors noted) TSee or OSG Storage ForumShawn McKee9

Scripts to Manage Storage TAs we have found problems we sometimes needed to automate the solution q Rephot – Automated replica creation developed by Wenjing Wu scans for “hot” files and automatically adds/removes replicas q Pool-balancer – Rebalances pools within a group TNumerous scripts (cron) for maintenance q Directory ownership q Consistency checking, including Adler32 checksum verification q Repairing inconsistencies q Tracking usage (real space and space-token) q Auto-loading PNFSID into Chimera DB for faster access OSG Storage ForumShawn McKee10

AGLT2 Storage Issues TInformation is scattered across server nodes and log files TJava based components have verbose output which makes it hard to find the real problem TWith dCache we find the large number of components have a large phase-space for problems TErrors are not always indicative of the real problem… TIs syslog-ng a part of the solution? TCan error messages be significantly improved? OSG Storage ForumShawn McKee11

Benchmarking and Optimization TWe have spent some time trying to optimize our storage performance. TSee Systems Systems Systems TExplored system/kernel/network and I/O tunings for our hardware. Have achieved good performance for single read/write (>700MB/sec) per partition. Multiple readers/writers at a few hundred MB/sec. OSG Storage ForumShawn McKee12

AGLT2 dCache Pool Transfer Rates STEP09 OSG Storage ForumShawn McKee13 Avg 616 MBytes/sec over 8 days

Future AGLT2 Storage TWe are interested in exploring BestMan+X where ‘X’ is q NFSv4[.x] q Lustre 1.8.x q Hadoop TWe are interested in the following characteristics for each option: q Expertise required to install/configure q Manpower required to maintain q Robustness over time q Performance (single and multiple read/writer) TOur plan is to test these during the rest of 2009 OSG Storage ForumShawn McKee14

?Questions? OSG Storage ForumShawn McKee15

Backup Slides OSG Storage Forum16Shawn McKee

AGLT2 “Server” 10GE Activity STEP09 OSG Storage ForumShawn McKee17 Shown are the aggregated graphs for our 10GE storage servers during STEP09 (Units: Bytes/sec) Results are a combination of local and remote traffic

MiLR 10GE Protected Network TWe have a single “/23” network for the AGL-Tier2 q Internally each site (UM/MSU) has a /24 TOur network will have 3 10GE wavelengths on MiLR in a “triangle” q Loss of any of the 3 waves doesn’t impact connectivity for both sites OSG Storage Forum18Shawn McKee