1. Monterey 2007-06-25 HPDC Workshop Experiences with MC-GPFS in DEISA Andreas Schott (schott@rzg.mpg.de)

2. Monterey 2007-06-25HPDC Workshop2 Overview Introduction to DEISA Partners in DEISA Aims of DEISA Introduction to Multiple Cluster GPFS Concepts of GPFS Local GPFS Multi-Cluster GPFS Evolution of MC-GPFS in DEISA MC-GPFS until now MC-GPFS in the future Discussion

3. Monterey 2007-06-25HPDC Workshop3 DEISA Partners

4. Monterey 2007-06-25HPDC Workshop4 Aims of DEISA Providing HPC resources to the Scientific Community Offering an add-on value to local facilities optimal hardware selection easy usability transparent data access Achievement of these Aims common network structure

5. Monterey 2007-06-25HPDC Workshop5 DEISA – Network (estimated Q3 / 2007) SURFnet UKERNAFUNET RedIris GARR 1 Gb/s 10 Gb/s RENATER 10 Gb/s GÉANT2 DFN 10 Gb/s Dedicated 10 Gb/s wavelength 1 Gb/s LSP Dedicated 10 Gb/s wavelength (potential) GÉANT LSP DFN/GÉANT Frankfurt ralph.niederberger@fz-juelich.de

6. Monterey 2007-06-25HPDC Workshop6 Aims of DEISA Providing HPC resources to the Scientific Community Offering an add-on value to local facilities optimal hardware selection easy usability transparent data access Achievement of these Aims common network structure using internal features of job schedulers additional middleware for easy access (e.g. UNICORE)

7. Monterey 2007-06-25HPDC Workshop7 AIX LL-MC AIX LL AIX LL-MC user Super-UX NQS II AIX LL job LINUX LSF LINUX PBS Pro CINECA IBM P5 FZJ IBM P4 RZG IBM P4 ECMWF IBM P5 CSC IBM P4 HPCX IBM P5 LRZ SGI ALTIX HLRS NEC SX8 SARA SGI ALTIX BSC IBM PPC IDRIS IBM P4 AIX LL-MC LINUX LL Super-UX NQS II johannes.reetz@rzg.mpg.de

8. Monterey 2007-06-25HPDC Workshop8 NJS CINECA IBM P5 IDB UUDB Gateway CINECA AIX LL-MC AIX LL AIX LL-MC CINECA user Super-UX NQS II AIX LL job LINUX LSF LINUX PBS Pro AIX LL-MC LINUX LL Super-UX NQS II johannes.reetz@rzg.mpg.de

9. Monterey 2007-06-25HPDC Workshop9 Gateway CSC Gateway ECMWF Gateway FZJ Gateway IDRIS Gateway SARA Gateway LRZ Gateway HPCX Gateway HLRS NJS CINECA IBM P5 IDB UUDB Gateway BSC Gateway CINECA NJS FZJ IBM P4 IDB UUDB NJS RZG IBM P4 IDB UUDB NJS ECMWF IBM P5 IDB UUDB NJS CSC IBM P4 IDB UUDB NJS HPCX IBM P5 IDB UUDB NJS LRZ SGI ALTIX IDB UUDB NJS HLRS NEC SX8 IDB UUDB AIX LL-MC AIX LL AIX LL-MC CINECA user Super-UX NQS II AIX LL job NJS SARA SGI ALTIX IDB UUDB NJS BSC IBM PPC IDB UUDB LINUX LSF LINUX PBS Pro Gateway RZG NJS IDRIS IBM P4 IDBUUDB AIX LL-MC LINUX LL Super-UX NQS II johannes.reetz@rzg.mpg.de

10. Monterey 2007-06-25HPDC Workshop10 Gateway CSC Gateway ECMWF Gateway FZJ Gateway IDRIS Gateway SARA Gateway LRZ Gateway HPCX Gateway HLRS NJS CINECA IBM P5 IDB UUDB Gateway BSC Gateway CINECA NJS FZJ IBM P4 IDB UUDB NJS RZG IBM P4 IDB UUDB NJS ECMWF IBM P5 IDB UUDB NJS CSC IBM P4 IDB UUDB NJS HPCX IBM P5 IDB UUDB NJS LRZ SGI ALTIX IDB UUDB NJS HLRS NEC SX8 IDB UUDB AIX LL-MC AIX LL AIX LL-MC CINECA user Super-UX NQS II AIX LL job NJS SARA SGI ALTIX IDB UUDB NJS BSC IBM PPC IDB UUDB LINUX LSF LINUX PBS Pro Gateway RZG NJS IDRIS IBM P4 IDBUUDB AIX LL-MC LINUX LL AIX LL-MC Super-UX NQS II johannes.reetz@rzg.mpg.de

11. Monterey 2007-06-25HPDC Workshop11 Aims of DEISA Providing HPC resources to the Scientific Community Offering an add-on value to local facilities optimal hardware selection easy usability transparent data access Achievement of these Aims common network structure using internal features of job schedulers additional middleware for easy access (e.g. UNICORE) global file system in a network of trust

12. Monterey 2007-06-25HPDC Workshop12 General Concepts of MC-GPFS MC-GPFS = Multiple Cluster General Parallel File System available for all HPC architectures in DEISA servers available for AIX and Linux Principle Structure distributed – shared – striped kernel add-on for file system block oriented data transfer Features achieved shared and high performance access safe and secure data high administrative flexibility

13. Monterey 2007-06-25HPDC Workshop13 General Concepts of MC-GPFS Technical Aspects each site with its own servers possible local disk space locally administered scalability and high performance access by inherent parallelism easy extensible file consistency by sophisticated token management high recoverability and increased data availability simplified storage management storage pools, file sets simplified administration globally acting commands

14. Monterey 2007-06-25HPDC Workshop14 General Concepts of MC-GPFS Security Aspects separate network communication for administration possible remote security authenticated remote access for servers mount and/or data with SSL-keys easy root-mapping easy no-suid functionality userid mapping for remote access via interfaces

15. Monterey 2007-06-25HPDC Workshop15 General Concepts of MC-GPFS Access and Availability transparent access no special data transfer commands required global visibility inside DEISA extended access rights no single point of failure communication delegated locking and other communication

16. Monterey 2007-06-25HPDC Workshop16 Summary of MC-GPFS Local and Remote High Performance Access high parallelism in data and file access very large file and file system support High Availability each site with its own servers redundant access path simply extensible and scalable striped data parallel access path

17. Monterey 2007-06-25HPDC Workshop17 Local GPFS File Servers Network File Server 1 File Server 2 File Server N FC-Switch Disk System 1 Disk System 2 Disk System M...

18. Monterey 2007-06-25HPDC Workshop18 Local GPFS Access Network File Server 1... N FC-Switch Disk System 1... M Compute Server 1 Compute Server N

19. Monterey 2007-06-25HPDC Workshop19 Remote GPFS Access Network Site A Network Site B File Server 1... N FC-Switch Disk System 1... M Compute Server 1 Compute Server N File Server 1... N FC-Switch Disk System 1... M Compute Server N Compute Server 1 WAN

20. Monterey 2007-06-25HPDC Workshop20 Advantages of GPFS (admin) Easy Management Easy Extensibility High Performance Good Security Features Add-On Features like HSM Functionality

21. Monterey 2007-06-25HPDC Workshop21 Advantages of GPFS (user) Standard Access Methods Transparent Access Data globally visible No special actions for data transfer required Simplicity Extended Access Right Features Add-On Features like HSM Functionality

22. Monterey 2007-06-25HPDC Workshop22 GPFS Configuration in DEISA Each AIX-site provides its own server Some non-AIX-sites will provide servers based on Linux RZG hosts disk space for non-AIX-sites without servers RZG provides HSM-functionality on GPFS locally disk space performs like local disk space total of more than 30 TB wide area network connection with 10GBit/s (mostly) remotely disk space no longer limited by network

23. Monterey 2007-06-25HPDC Workshop23 DEISA „proof of concept“ phase Premium IP: IP Priority: LSPs: DFN RENATER GARR GÈANT RENATER GARR DFN 1 Gb/s

24. Monterey 2007-06-25HPDC Workshop24 Evolution of GPFS in DEISA RZG (DE) Power4 AIX FZJ (DE) Power4 AIX IDRIS (FR) Power4 AIX CINECA (IT) Power5 AIX InitialOctober 2004

25. Monterey 2007-06-25HPDC Workshop25 SDSC Chicago New York Amsterdam GEANT Milano Paris Teragrid Frankfurt FZJ Jülich RZG Munich DFN NREN Germany Cineca Bologna GARR NREN Italy IDRIS Orsay RENATER NREN France Internet2/Abilene R.Niederberger@fz-juelich.de 1 Gb/s Premium IP 1 Gb/s LSP 10 Gb/s 30-40 Gb/s 10 Gb/s DEISA – TeraGrid Connection Super Computing 2005

26. Monterey 2007-06-25HPDC Workshop26 DEISA 1 Gb/s network infrastructure RENATER FUNET SURFnet DFN GARR UKERNA RedIris GÉANT LSPs

27. Monterey 2007-06-25HPDC Workshop27 Evolution of GPFS in DEISA RZG (DE) Power4 AIX FZJ (DE) Power4 AIX IDRIS (FR) Power4 AIX CINECA (IT) Power5 AIX BSC (ES) PowerPC Linux CSC (FI) Power4 AIX SARA (NL) SGI-Altix Linux October 2004October 2005May 2006July 2006

28. Monterey 2007-06-25HPDC Workshop28 Upgrade of Multiple Cluster GPFS Problems with GPFS 2.3 Initial MC-functionality not inherently integrated Each-to-Any communication required Limitation of participating nodes Advantages of GPFS 3.1 Better Multi-Cluster Support Better Encapsulation by possible use of private addresses Higher Independence between sites Higher Stability Better Performance

29. Monterey 2007-06-25HPDC Workshop29 Evolution of GPFS in DEISA RZG (DE) Power4 AIX FZJ (DE) Power4 AIX IDRIS (FR) Power4 AIX CINECA (IT) Power5 AIX LRZ (DE) SGI-Altix Linux BSC (ES) PowerPC Linux CSC (FI) Power4 AIX SARA (NL) SGI-Altix Linux ECMWF (GB) Power5+ AIX July 2006September 2006February 2007

30. Monterey 2007-06-25HPDC Workshop30 Status of Multiple Cluster GPFS Site File- server StorageCompute-CPUsTFlopsMemory CINECA2 2 TB 480 Power5 (1.9 GHz)2.61152 GB CSC2 2 TB 512 Power4 (1.1 GHz)2.2 672 GB ECMWF2 1 TB2640 Power5+ (1.9 GHz)20.12250 GB FZJ2 4 TB1288 Power4 (1.7 GHz)8.95152 GB IDRIS2 2 TB1024 Power4 (1.3 GHz)6.73136 GB LRZ(RZG) 0 TB9728 Montecito (1.6 GHz)62.339064 GB RZG2 10 TB 928 Power4 (1.3 GHz)4.62368 GB

31. Monterey 2007-06-25HPDC Workshop31 Evolution of GPFS in DEISA RZG (DE) Power4 AIX FZJ (DE) Power4 AIX IDRIS (FR) Power4 AIX CINECA (IT) Power5 AIX LRZ (DE) SGI-Altix Linux BSC (ES) PowerPC Linux HLRS (DE) NEC-SX8 Super-UX CSC (FI) Power4 AIX SARA (NL) SGI-Altix Linux EPCC (GB) Power4 AIX ECMWF (GB) Power5+ AIX CSC (FI) Cray XT4 Linux SARA (NL) Power5 Linux /deisa/ /home/ / /deisa/ /data / / InitialOctober 2004October 2005May 2006July 2006September 2006February 2007July 2007September 2007Final

32. Monterey 2007-06-25HPDC Workshop32 Discussion Thanks. Questions?