STORM & GPFS on Tier-2 Milan Massimo.Pistolese@mi.infn.it Francesco.Prelz@mi.infn.it Workshop CCR – May 14 2009

Why StoRM ? Security model inherited by ACL support of underlying file system. Light diskspace and authorization manager, no special hardware required. Scalable Easily configurable using yaim. Individual parts of final configuration can still be tuned at a later time. CCR - May 14 2009 STORM&GPFS

Why StoRM ? SOAP web service, roles of all its parts can be assigned separately: front- end, back-end, gridftp, mysql request rate handled up to 40Hz Gridftp throughput 120MB/s on 1Gb/s lan. . Well matched with gpfs architecture. CCR - May 14 2009 STORM&GPFS

Why GPFS ? visibility as a local fs, no remote protocols. Cluster structured Slave clustering Redundancy Scalable Abstraction layer Robustness High performance: concurrent read from 16 clients 380MB/s (over a physical network limit of 412MB/s), concurrent write 320MB/s CCR - May 14 2009 STORM&GPFS

Network and storage setup CCR - May 14 2009 STORM&GPFS

gpfs gpfs gridftp-b1-1 gridftp-b1-2 gridftp-b1-3 gridftp-b1-4 Wn-b1-1 Ce-b1-1 (condor scheduler)‏ glite-UI-nodes.mi.infn.it glite-condor-nodes.mi.infn.it ui.mi.infn.it gpfs Wn-b1-1 (condor exe)‏ gpfs Wn-b1-38 (condor exe)‏ ....... gsiftp://gridftp-b1-1.mi.infn.it:2811/atlas/… /dev/storage_2 /dev/storage_1 /dev/software c13 c14 c2 c3 c1 c6 c7 c8 srm://t2cmcondor.mi.infn.it:8444/srm/managerv2?SFN=/atlas/… c15 c16 c17 c4 c5 c9 c10 c11 c12 mmcrfs /dev/software -F nsd-software.txt -B 64K -m 2 -M 2 -r 2 -R 2 -Q yes -n 512 -A yes -v no -N 50000000 ## definitions for /dev/storage_1 fs #sdb:ts-b1-2,ts-b1-1,ts-b1-4,ts-b1-3::dataAndMetadata:1:c2 c2:::dataAndMetadata:1::: #sdc:ts-b1-3,ts-b1-4,ts-b1-1,ts-b1-2::dataAndMetadata:1:c3 c3:::dataAndMetadata:1::: #sdd:ts-b1-4,ts-b1-3,ts-b1-2,ts-b1-1::dataAndMetadata:1:c4 c4:::dataAndMetadata:1::: #sde:ts-b1-1,ts-b1-2,ts-b1-3,ts-b1-4::dataAndMetadata:1:c5 c5:::dataAndMetadata:1::: storm FE T2cmcondor (central manager)‏ glite-condor.mi.infn.it gpfs gpfs ts-b1-1 ,..., ts-b1-4 ts-b1-5, ts-b1-6 ts-b1-7,ts-b1-8 Fiber channel FC multipath FC multipath storm-BE gridftp (ops)‏ gpfs 40TB 46TB 46TB se-b1-1 /opt/glite/yaim/bin/ig_yaim -c -s siteinfo/ig-site-info.def -n ig_SE_storm_backend gridftp-b1-1 gpfs gridftp-b1-2 gpfs gridftp-b1-3 gpfs gridftp-b1-4 gpfs

Issues with first production run Project Outline: Analyse MSSM A/H  tau tau  l h at 14TeV, to obtain discovery potential for this channel in ATLAS (Publish result as a PUB Note)‏ To do this must produce ALL our own datasets Production: Target Numbers of Events: Total: 35 M events Milano Share: 3.2 M events CCR - May 14 2009 STORM&GPFS

Issues with first production run Job specification: Atlfast II Simulation using job transform (csc_simul_reco_trf.py)‏ Input: evgen -> lcg-cp from SE: INFN-MILANO_LOCALGROUPDISK Output: AOD -> lcg-cp to SE: INFN-MILANO_LOCALGROUPDISK Event Per Job: 250 Total Jobs: 12800 Requirements: 2GB RAM / 2GB swap Running Time: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz cache size: 6144 KB  6 hours (TYPE1)‏ Intel(R) Xeon(TM) CPU 3.06GHz cache size: 512 KB  18 hours (TYPE2)‏ Cluster Performance: 48 CPU (TYPE1) : 192 Jobs/Day 124 CPU (TYPE2): 165 Jobs/Day CCR - May 14 2009 STORM&GPFS

Issues with first production run Failure Rate: 50% Environment setup and variables tuning Machine requirements: huge memory needed Works at typical ATLAS Production failure rate ~3% when functioning correctly Mainly issues with setup, GPFS and storage Gpfs perfectly integrated with StoRM. No special settings required New clients can be easily added CCR - May 14 2009 STORM&GPFS

Issues with first production run Gpfs caching related to OS limits (single- process 2GB virtual memory on 32-bit nodes). 64-bit clients can accommodate larger cache.. C-NFS is good to provide fault-tolerant NFS read-only mounts for better, faster caching, but special care must be taken. Better to separate disk servers from nfs and set gpfs redundancy for related filesystems Local hosts should resolve gridftp servers, to prevent StoRM from generating too much internal traffic. (VLANs could also be used). CCR - May 14 2009 STORM&GPFS

Conclusions Most issues found in GPFS and worker node setup. The GPFS cluster architecture allows better organization for pools of similar machines, with central steering. GPFS performance can be easily enforced in a SAN. Headnodes can be added as needed StoRM scalability: Can manage as many frontend and gridftp servers as needed CCR - May 14 2009 STORM&GPFS