1. STORAGE EXPERIENCES AT MWT2 (US ATLAS MIDWEST TIER2 CENTER) Aaron van Meerten University of Chicago Sarah Williams Indiana University OSG Storage Forum, Sept 21-22, 2010 1

2. Introduction to MWT2 Midwest Tier2 Facts 2 sites: UChicago and Indiana U (Indianapolis campus) ATLAS Production and User Analysis jobs D0, and HCC jobs opportunistically (in progress) ~2000 cores available for running jobs 1.41 Petabytes of available storage 6.3 Million files in dCache storage 60K+ datasets registered and stored at UC 10Gb/s circuits between sites and to BNL dCache Golden Release 1.9.5-21 PNFS / Chimera Namespaces xrootd testbed (across two sites) – data access, Tier 3 evaluation 2

3. Network Configuration at MWT2 Cisco 6509 (UC) Force10 (IU) 192 x 1GB Switch ports 8 x 10GB Switch ports (CX4) ~160 Workers 6 Storage Nodes 12 Head Nodes Dell 6248 96 x 1GB Switch ports 8 x CX4 10GB Switch ports ~70 workers 4 Storage Nodes 10 GB TRUNK Full Duplex 10 GB UPLINK Full Duplex 3

4. Storage Services at MWT2 DC1 - SRM dCache SRM SRMWatch PostgreSQL DC3 - ADMIN dCache AdminDomain PoolManager PostgreSQL DC4 - NAMES pnfsd dCache PnfsDomain PostgreSQL S-NODES - STORAGE dCache PoolDomain GridFTP door DCAP door x10 4

5. Storage Hardware at MWT2 DC1 - DC3 4 Cores: Opteron 275 @2.2GHz 8GB RAM 21GB System Disk 45GB Partition for DB 2 x 1Gb/s Network DC4 Dell R610 with PERC H700 HBA 8 Cores HT: Xeon 5560 @ 2.8GHz 32GB RAM 120GB RAID-0 System Disk 120GB SSD 3-Disk RAID5 for DB 2 x 1Gb/s Network S-NODES - STORAGE Dell R710 with 2 x PERC6/E HBAs 8 Cores HT: Xeon X5570 @ 2.93GHz 24GB RAM 125GB RAID-0 System Disk 6 x 26TB RAID-6 MD1000 Dual-Cabled via SAS 15 x 2TB 5400 RPM SATA per MD1000 10Gb/s Network + 1Gb/s Network C-NODES – COMPUTE & STORAGE (IU ONLY) 4 Cores: Opteron 285 @ 2.59GHz 8GB RAM 66GB SATA System Disk 2TB RAID-5(3Ware or LSI MegaRAID) 1Gb/s Network 5

6. Storage Cabling at MWT2 Front ViewSingle CabledDual Cabled 6

7. DD Benchmarks at MWT2 7

8. Bonnie++ Benchmarks at MWT2 8

9. Hardware Blessing at MWT2 Basic Configuration & Functionality Power to all devices, working beo_power for power switches Boots with no errors or prompts Serial Console functional, all output including BIOS redirected to it Network connection and IP (pingable) All external devices report connected and healthy (RAID) All services start correctly Log files for services clean Network names all resolve All certificates and CA’s confirmed Entered into monitoring systems (ganglia, nagios, cacti, etc) Performance RAID Bandwidth: Single DD I/O tests for read & write to file Bandwidth: Bonnie++ I/O tests I/O Scheduling: 120-thread DD tests reading/writing files Network Iperf tests between 10Gb/s nodes reach at least 9Gb/s dCache Single transfer throughput test Internal/external network transfer test 40-thread srmcp read and write tests 9

10. Typical Benchmarks at MWT2 Storage Benchmarks One Shelf – No Load Bonnie Write: 513 MB/s DD Write: 560 MB/s Bonnie Read: 837 MB/s DD Read: 914 MB/s Six Shelves-120 Threads DD writes: 4-5 MB/s per thread – 600 MB/s total DD reads: 20-22 MB/s per thread – 2400 MB/s total Network Benchmarks iperf on 10Gb Interface 9.92 Gbits/sec SRM Writes 40 writes x 1GB file Average Speed per thread: 13 MB/s Fastest thread: 76 MB/s Overall Throughput: 197 MB/s 10

11. Typical Performance at MWT2 Site Transfers per Hour over 2 days Site Transfers on Uplink over 2 days Network Throughput for a single node Movers on a single pool Disk Throughput on a single pool 11

12. Throughput Performance at MWT2 Space Token Sizes for 5 MonthsThroughput to Tier-2 sites from BNL Storage Load Test writing from IU to UC Typical High-Throughput Day 12

13. Lessons at MWT2 dCache Tunings Increase # of allowed movers Multiple mover queues (fast and slow) Increase ulimits on head nodes Increase thread count for pnfsd Metadata for pools stored in BDB format Metadata stored on separate filesystem from pool data Tunings in /opt/dcache/dCacheSetup Other Tunings Jumbo Frames – MTU 9000 is the only way to get 9Gb/s PostgreSQL tuning (hardware-specific but VERY effective) 13

14. Challenges at MWT2 Network & Service Challenges/Issues SRM timeouts during small number of wide area transfers (would like to reduce this to zero) Local data access from analysis jobs High dynamic range of network load for direct access jobs Space token re-balancing (data management issue) Storage Challenges Data consistency DQ2 central catalog LFC – local catalog PNFS ID System loads: pnfsd Greatly improved with hardware upgrade (SSDs) Hot pools and hot files create bottlenecks 14

15. Thanks from MWT2 15

16. Additional slides follow 16

17. Storage Services at MWT2 - IU DC1 - SRM dcache SRM SRMWatch PostgreSQL DC2 - NAMES Chimera-NFS dcache ChimeraDomain PostgreSQL DC3 - ADMIN dcache AdminDomain PoolManager PostgreSQL S-NODES - STORAGE dcache PoolDomain GridFTP door DCAP door C-NODES – STORAGE & COMPUTE dcache PoolDomain Running jobs on 3 cores, 1 reserved for dcache 17

18. HammerCloud at MWT2 Direct Access HC with only UC workers Direct Access HC with IU and UC workers 18