1. Competences & Plans at GSI
LSDMA Kickoff Workshop
22 March 2012, KIT

2. open source and community software budget commodity hardware
GSI: a German National Lab for Heavy Ion Research FAIR: Facility for Ion and Antiproton Research ~2018
GSI computing today
ALICE T2/T3 HADES ~ cores,
~ 5.5 PB lustre
~ 9 PB archive capacity
PANDA
CBM/HADES
NuSTAR
APPA
FAIR computing 2018
CBM
PANDA
NuSTAR
APPA
LQCD
cores
40 PB disk
40 PB archive
open source and community software
budget commodity hardware
support different communities
scarce manpower
& FAIR MAN
& FAIR Grid/Cloud

3. 1Tb/s
1Tb/s
TB/s
1Tb/s
1Tb/s

4. Distribution of Competences
energy efficient high throughput cluster
cluster file system lustre
tape archive, gStore
MAN
Grid, Cloud
Exp. Frameworks
Parallel Programming
Two HEP like experiments: CBM & PANDA,
two research fields with a lot of smaller experiments
~3000 scientists from ~50 countries

5. „General Purpose“ HPC System, optimized for high I/O bandwidth
Compute farms:
4000 cores, ethernet based
10000 cores IB based
Debian, SGE
shared usage, based on priorities by GSI, FAIR, ALICE Tier2/3
Lustre with 3 MDS HA cluster, 48 core, 300 GB RAM 180 Fileservers with 3.5 TB RAM as OSS disks > 5.5 PB raw capacity I/O capacity about 1 Tb/s more to come IB based l

6. We provide a mixture of: a Tier 2 integrated in the AliEn Grid,
The Alice Tier-2 at GSI
We provide a mixture of:
a Tier 2 integrated in the AliEn Grid,
mainly simulation
a Tier 3,
analysis trains
fast interactive responds via PROOF based on PoD
calibration & alignment of TPC
integrated in the standard GSI batch farm (GSI, FAIR).
We are able to readjust the relative size of the different parts on request.
Main contributions from Germany:
Uni Heidelberg
Uni Frankfurt
Uni Münster
Uni Darmstadt
GSI
TPC
TRD
HLT
GridKa Tier-1
GSI Tier-2/3
LoeweCSC

7. New (dark) Datacenter: ready 2014
Minimal floor footprint
Space for “ racks
Planned cooling power 6 MW (building supports more)
Internal temperature 30°
Minimal construction cost
Fast to build
Autonomous rack power mgnt.
Back door rack cooling
Smoke detection in every rack
Use of heat for building heating
Shortest cable lenghts
Water pressure below 1bar avoiding risk of spills
Use of any commercial 19“ architecture
Unmatched packing- and power density
Construction cost about 11 M€

8. Proof of Concept: Testcontainer with PUE < 1.1 in operation since 1.5 years Mini Cube ~ 100 racks, 1.2 MW cooling power: 1Q2012

9. First Step to the FAIR HPC Link
12 * 10 Gb/s GSI – LOEWE CSC
2 * 10 Gb/s GSI – Uni Ffm
4 * 10 Gb/s GSI – Uni Mainz

10. gStore: a lightweight tape access system: 2 3584-L23 ATL, 8 TS1140 tape drives, 17 data mover
resource
max
storage capacities:
Experimentdata:
Copy of rawdata
overall data mover disk cache
8.8 PB
1.3 PB
0.26 PB
overall gStore I/O bandwith:
DM disk <-> tape (SAN)
DM disk <-> clients (LAN)
DAQ online clients ->DM
incl. copy to lustre
2.0 GB/s
5.0 GB/s
1.0 GB/s
0.5 GB/s

11. Exploration of Virtualization & Cloud Computing
GSI as Cloud provider:
SClab: Cloud Prototype at GSI ~ 200 cores
demo production for ALICE
GSI as Cloud user:
FLUKA jobs for Radiation Safety Studies on Amazon and Ffm cloud
IaaS-
cloud VM
CMS server
DRMS master
client
worker
application

12. The FairRoot framework is used by CBM, PANDA and parts of NUSTAR
Virtual MC
Geant3
Geant4
FLUKA
G4VMC
FlukaVMC
G3VMC
Geometry
Close
contact
common developments
Run Manager
Event
Generator
Magnetic
Field
Detector base
IO Manager
Tasks
RTDataBase
SQL
Conf,
Par,
Geo
Root files
Hits,
Digits,
Tracks
Application
Cuts,
processes
Display
Track propagation
Always in
close
contact
STS
TRD
TOF
RICH
ECAL
MVD
ZDC
MUCH
ASCII
Urqmd
Pluto
Track finding
digitizers
Hit
Producers
Dipole Map
Active Map
const. field
CBM Code
STT
MUO
TOF
DCH
EMC
MVD
TPC
DIRC
ASCII
EVT
DPM
Track finding
digitizers
Hit
Producers
Dipole Map
Solenoid
Map
const.
field
Panda Code

13. Challenges for reconstruction
UrQMD, central 25 AGeV
particles per collision of two gold nuclei
results up to now
D0, cτ=127 μm K- π+
Open charm:
Typical signal multiplicities: O(10-6)
No „easy“ trigger signatures
Fast online event reconstruction, no hardware trigger:
time to reconstruct 1-10ms & 107 collisions/s -> – 105 cores

14. Exploring Different Architectures and Parallelization Strategies: eg
Exploring Different Architectures and Parallelization Strategies: eg. PANDA Tracking on CPU and GPU, PROOF, PoD
CPU 1
Event 1
Track Candidates
GPU Track
Fitting
Tracks
CPU 2
Event 2
CPU 3
Event 3
CPU 4
Event 4
No. of Process
50 Track/Event
2000
Track/Event
1 CPU
1.7 E4 Track/s
9.1 E2 Track/s
1 CPU + GPU (Tesla)
5.0 E4 Track/s
6.3 E5 Track/s
4 CPU + GPU (Tesla)
1.2 E5 Track/s
2.2 E6 Track/s

15. Remote access to data and computing via the Grid: example Alice and PANDA Grid with AliEn
Panda Grid Sites

17. http://www.crisp-fp7.eu/ WP16: Common User Identity System
WP19: Distributed Data Infrastructure

18. Long term goal: Distributed data management for FAIR Short/Medium term projects:
Storage Federation Remote lustre SE on top of lustre global xroot redirector for ALICE and FAIR Cloud Storage interface to existing environments (ROOT, AliEn,...) Optimized job/data placement hot vs unused data check for corrupted data sets reading from several SEs 3rd party copy

19. Summary: FAIR Computing
FAIR will serve about 20 scientific collaborations from four major fields of research and application. Basis of the software systems of the larger FAIR experiments is FairRoot - a simulation, reconstruction and analysis framework. It is designed to optimize the accessibility for new as well as experienced users, to be able to cope with future developments, and to enhance the synergy between the different experiments. Parallelization of the software on all levels is essential to harvest the performance gains of future architectures.

20. Data recording for CBM and PANDA require a detector read-out without hardware triggers, relying exclusively on online event filtering.
The first layer of the system consists of a combination of specialized processing elements such as GPUs, CELLs or FPGAs in combination with COTS computers, connected by an efficient high-speed network.
Major building blocks of the infrastructure are
a new datacenter for online computing, storage systems and offline cluster,
linked to a metropolitan area network connecting the compute facilities of the surrounding universities and research centers,
embedded in an international Grid/Cloud infrastructure.