STEINBUCH CENTRE FOR COMPUTING - SCC KIT – University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association GridKa Site Report Andreas Petzold

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 GridKa Batch Farm Univa Grid Engine is running fine ~150kHS06 ~10k job slots 98 replacement machines this summer SysGen 2U 4 node chassis 2x Intel Xeon E (8-core, 2,6 GHz, 312 HS06) 3GB/core 3x 500GB HDD

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 WN Migration to SL6 Migration of GridKa compute fabric to SL6 finished Performance: +5.4% Intel Xeon E (8 cores, 2.6 GHz) HT off / HT on: SL5 + default compiler:267 HS06/335 HS06 SL6 + default compiler:283 HS06(+5.8%)/348 HS06(+3.9 %) SL5 + gcc-4.8.1:289 HS06/353 HS06 AMD Opteron 6168 (12 cores, 1.9 GHz): SL5 + default compiler:183 HS06 SL6 + default compiler:193 HS06 (+ 5.6 %) SL5 + gcc-4.8.1:187 HS06

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 Ivy Bridge Benchmarks New Intel Ivy Bridge processors on the market (E5-26## v2) Manufacturing process: micron Sandy Bridge: micron Up to 12 cores Sandy Bridge: up to 8 cores Increasing HS06 score according to number of cores: E (8 cores, 2.6 GHz, HT on, SL6, default compiler) 348 HS06 E v2 (10 cores, 2.5 GHz, HT on, SL6, default compiler) 411 HS06 Power saving of around % Thanks to DELL for providing test machine

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 Power Efficiency Power Usage (W) per Performance Score (HS06) Worker node class machines at GridKa / E5-2670v2 is a test system provided by DELL AMD 6168 Intel E5430 Intel L5420 AMD 246 AMD 270 Intel 5160 Intel E5345 Intel E5520 (HT on) Intel E (HT on) Watts per HS06 Intel E5-2670v2 (HT on)

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 GridKa dCache & xrootd 6 production dCache instances + pre-production setup 5 instances running 2.6, 1 running PB, 287 pools on 58 servers Upgrade to 2.6 instead of 2.2 recommended by dCache.org last minute decision one week before planned downtime full support for SHA-2 and xrootd monitoring great support from dCache devs CMS disk-tape separation most CMS tape pools converted to disk-only pools last CMS config changes today GridKa 1 st CMS T1 successfully migrated two xrootd instances for ALICE 2.7PB 15 servers

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 GridKa Disk Storage 9x DDN S2AA enclosures 9000 disks 796 LUNs SAN Brocade DCX 1x DDN SFA10K 10 enclosures 600 disks 1x DDN SFA12K 5 enclosures 360 disks 14PB usable storage

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 Evaluating new Storage Solutions DDN SFA12K-E allows to run server VMs directly in storage controller DDN are testing complete dCache instance inside controller expected benefits shortening long IO paths: no SAN + FC HBAs, reduced latency less hardware: less power consumption, improved MTBF possible drawbacks limited resources in storage controllers for VMs loss of redundancy

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 DDN SFA12K-E

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 Glimpse at Performance Preliminary performance evaluation IOZONE testing parallel threads on XFS file system still a lot of work ahead no tuning file system + controller setup tuning

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 GridKa Tape Storage 2x Oracle/Sun/STK SL8500 2x slots 22 LTO5, 16 LTO4 drives 1x IBM TS slots 24 LTO4 drives 1x GRAU XL 5376 slots 16 LTO3, 8 LTO4 >20k cartridges 17PB Migration to HPSS planned for 2014

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor G WAN at GridKa Current WAN setup 7x10Gb/s links to LHCOPN, LHCONE, German research network, FZU Prague + 1x1Gb/s link to Poznan participation in 100G tests at SC G equipment provided by CISCO 100G connection provided by DFN, time-shared by Aachen, Dresden, KIT plan to move LHCOPN, LHCONE to 100G link in 2014 replace old Catalyst border routers procurement of new Nexus 7k with 100G line cards already underway requires new arrangement of LHCOPN operation between KIT and DFN

INSTITUTS-, FAKULTÄTS-, ABTEILUNGSNAME (in der Masteransicht ändern) Steinbuch Centre for Computing Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013 Configuration Management Still mostly using Cfengine 2 Middleware services used as testbed for puppet Started in early 2012 Still based on old homegrown deployment infrastructure CluClo Very smooth operation Now starting to draw up plans for puppet migration wed like to try many new things: git integration, deployment management with Foreman, MCollective, … Will be step by step process

bwLSDF News from SCC/KIT outside GridKa new services for state of Baden-Württemberg run by SCC/KIT bwSync&Share dropbox for scientists winner of software evaluation: PowerFolder start of production Jan 1 st 2014 expect active 55k users from all universities, 10GB quota bwFileStorage simple/overflow storage for scientific data access via SCP, SFTP, HTTPS (r/o) provided by IBM SONAS start of production Dec 1 st 2013 bwBlockStorage iSCSI storage over WAN for universities all services based on storage hosted at Large Scale Data Facility Andreas Petzold – GridKa Site Report – HEPiX Ann Arbor 2013

bwIDM The bwIDM Project: Vision Federated access to services of the State of Baden-Württemberg Access control based on local accounts of the home organizations bwIDM is not about establishing IDM systems, its about federating existing IDM systems and services. bwServices | Federating HPC access via SAML 15

bwIDM bwIDM Overview bwIDM –…federation of 9 universities of the state of Baden-Württemberg (non) web-based services –…federates the access to non web-based services such as grid, cloud, and HPC resources. LDAP Facade –Deployable, operable, and maintainable approach to federate non web-based services: LDAP facade makes active use of the SAML-ECP and AssertionQuery profile LDAP facade offers users a high usability in trustworthy federations LDAP facade facilitates temporary trust for scientific portals Easy-to-deploy solution for service collaborations of universities, research centres or companies Single registration process per service service access Successfully deployed in testing environments Deployed Services –Federated HPC Service bwUniCluster (8640 cores, 40.8 TIB Ram, IB FDR) going live in Q4/2013 –Federated Sync&Share Service going live in Q1/2014 Any Questions? Feel free to contact me: | Federating HPC access via SAML If you have to bring non web-based services together with SAML, make use of the LDAP facade!