GSIAF "CAF" experience at GSI Kilian Schwarz

GSIAF Present status installation and configuration operation Plans and outview Issues and problems

ALICE::GSI::SE::xrootd Directly attached disk storage (81 TB) GSI – current setup CERN 1 Gbps GridK a 3rd party copy test cluster 10 TB 80 TB ALICE::GSI::SE::xrootd Grid vobox Grid CE LCG RB/CE GSI batchfarm: ALICE cluster (160 nodes/1200 cores for batch: 20 nodes for GSIAF) Directly attached disk storage (81 TB) gStore backend Local xrd Cluster 15 TB PROOF/Batch GSI batchfarm: Common batch queue Lustre Clustre 150 TB GSI

Present Status ALICE::GSI:SE::xrootd 75 TB disk on fileserver (16 FS a 4-5 TB each) 3U 12*500 GB disks RAID 5 6 TB user space per server Batch Farm/GSIAF gave up concept of ALICE::GSI::SE_tactical::xrootd reasons: not good to mix local and Grid access cryptic file names make non Grid access difficult nodes dedicated to ALICE (Grid and local) but used also by FAIR and Theory if ALICE is not present 160 boxes: 1200 cores (to a large extend funded by D-Grid): each 2*2core 2.67 GHz Xeon, 8 GB RAM 2.1 TB local disk space on 3 disks + system disk Additionally 24 new boxes: each 2*4core 2.67 GHz Xeon, 16 GB RAM 2.0 TB local disk space on 4 disks including system up to 2*4 core, 32GB RAM and Dell Blade Centres on all machines: Debian Etch 64bit

ML Monitoring: Storage Cluster

installation shared NFS dir, visible by all nodes xrootd ( version 2.9.0 build 20080621-0000) ROOT (521-01-alice and 519-04) AliRoot (head) all compiled for 64bit reason: due to fast software changes disadvantage: possible NFS stales started to build Debian packages of the used software to install locally

configuration setup: 1 standalone, high end 32 GB machine for xrd redirector and proof master, Cluster: xrd data servers and proof workers, AliEn SE, Lustre so far no authentification/authorization via Cfengine platform independent computer administration system (main functionality: automatic configuration). xrootd.cf, proof.conf, TkAuthz.Authorization, access control, Debian specific init scripts for start/stop of daemons (for the latter also Capistrano and LSF methods for fast prototyping) all configuration files are under version control (subversion)

Cfengine – config files in subversion

monitoring via MonaLisa http://lxgrid3.gsi.de:8080

GSIAF usage experience real life analysis work of staged data by GSI ALICE group (1-4 concurrent users) 2 user tutorials for GSI ALICE users (10 students each training) GSIAF and CAF were used as PROOF clusters during PROOF course at GridKa School 2007 and 2008

PROOF users at GSIAF

GridKa School 2008

ideas came true ... Coexistence of interactive and batch processes (PROOF analysis on staged data and Grid user/production jobs) on the same machines can be handled !!! re”nice” LSF batch processes to give PROOF processes a higher priority (LSF parameter) number of jobs per queue can be increased/decreased queues can be enabled/disabled jobs can be moved from one queue to other queues Currently at GSI each PROOF worker is an LSF batch node optimised I/O. Various methods of data access (local disk, file servers via xrd, mounted lustre cluster) have been investigated systematically. Method of choice: Lustre and eventually xrd based SE. Local disks are not used for PROOF anymore at GSIAF. PROOF nodes can be added/removed easily local disks have no good surviving rate extend GSI T2 and GSIAF according to promised ramp up plan

cluster load

data transfer and user support how to bring data to GSIAF ? used method: GSI users copy data from AliEn individually to Lustre using standard tools inter site transfer is done via AliEn FTD centrally managed user support at GSIAF: private communication or help yourself via established procedures.

GSI Luster Clustre

ALICE T2 – ramp up plans  http://lcg.web.cern.ch/LCG/C-RRB/MoU/WLCGMoU.pdf GSIAF will grow with a similar rate

gLitePROOF – PROOF on demand A number of utilities and configuration files to implement a PROOF distributed data analysis on the gLite Grid. Built on top of RGlite: TGridXXX interface are implemented in RGLite for gLite MW. ROOT team accepted our suggestions to TGridXXX interface. gLitePROOF package – tested against FZK and GSI WMSs. recently also with plain LSF @ GSI It setups “on-the-fly” a PROOF cluster on gLite Grid. It works with mixed type of gLite worker nodes (x86_64, i686...) It supports reconnection. gLitePROOF – PROOF on demand A. Manafov 20

issues and problems

issues and problems instability of PROOF (or our setup). Under certain circumstances, when something goes wrong, e.g. a PROOF session crashes or the user code has bugs PROOF manages to affect the underlying xrootd architecture. (Here eventually some sandbox methods would be useful: crashing sessions of user A should not affect sessions of user B) since quite often xrootd manages to recover by itself after a few hours it may be related to retries and timeouts. If one could reduce these timeouts ... sometimes it keeps hanging so much that xrd needs to be restarted. But this can not be done by the users on the worker nodes. Can a cluster restart be triggered via the redirector ? or more general: how to restart, configure a whole PROOF cluster via one node. Nice to have for admins also.

wishlist, summary and outview