1. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Advanced computing for fusion simulation and modelling Pär Strand Chalmers University of Technology EU Fusion for ITER Applications - EUFORIA 5 th EGEE USER FORUM 2010-04-14 Uppsala

2. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Overview Brief Background – Fusion Modelling Challenges - Use Cases. EUFORIA approach to supporting modellers Summary

3. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Fusion Energy source for the sun and other stars Provides a potential source of base load energy production Been working on this for more than 50 years Has turned out to be a very difficult problem "Every time you look up at the sky, every one of those points of light is a reminder that fusion power is extractable from hydrogen and other light elements, and it is an everyday reality throughout the Milky Way Galaxy." --- Carl Sagan, Spitzer Lecture, October 1991

4. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Fusion Two main lines of research  Inertial confinement Implosion of small pellets NIF at LLNL  Magnetic confinement Two main type of confugrations studied:  Stellarator – W7X Currently under construction in Greifswald in Germany  Tokamak – ITER Under construction in Cadarache in France

5. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 ITER Involves 7 partners representing more than 50% world population Costs > 10 G$ Under construction in Cadarache, France Key element on the path to fusion energy production

6. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 ITER Costs > 10 G$ Budget is for Construction only! Developments for the work programme and securing the exploitation of the device is in the hands of each of the partners.

7. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Units Plasma Major Radius 6.2m Plasma Minor Radius 2.0m Plasma Volume 840m3 Plasma Current 15.0MA Toroidal Field on Axis 5.3T Fusion Power 500MW Burn Flat Top >400s Power Amplification >10 ITER

8. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Units Plasma Major Radius 6.2m Plasma Minor Radius 2.0m Plasma Volume 840m3 Plasma Current 15.0MA Toroidal Field on Axis 5.3T Fusion Power 500MW Burn Flat Top >400s Power Amplification >10 ITER

9. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 ITER has developed from a long sequence of experiments BUT is the first tokamak where modelling is set to play an important role in both design and exploitation. This is even more true for the next stage - a first DEMO reactor.

10. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 The “fast track” to fusion energy – hinges on several parallel developments “Numerical Tokamak” – a comprehensive simulation package with predictive capabilities.

11. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Relation to Experiments Physics analysis and exploration tools ”Semi-open” Operational core tools ”In House” Operational core tools ”In House” Local Data layer Modules Applications Workflows Modules Applications Workflows Grid HPC Local Cluster Local Cluster Local Data Local Data Data exchanges Requires Machine descriptions and Data mappings Universal Access Layer, exp2ITM Experiment Modelling Platform User Gateway Device Independent Exchange of Modules Applications

12. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Modelling Challenges Fusion codes comes with a variety of different computational requirements:  Local calculations (serial or small clusters)  Capacity computing MC, Ray tracing, parameter scans Intermediate level grid parallel applications (~128cores)  Capability computing (pushing the limits of existing HPC) No single “best” computing paradigm  A single application may require different architectures/ or infrastructures depending on particular use!  Plasmas are complex and non-linear and dynamic responses may force higher level fidelity in modelling – on the fly change of application and computing infrastructure requirements Physics are coupled  Rare situation with a single physics codes suffices for deep understanding - coupled applications a necessity!

13. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 1d 2d Real problem is 3d space, 2/3d velocity Simulations EUFORIA scope is ”edge and core turbulence and transport”

14. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Models describing the plasma vary in complexity -Extreme ranges in both time and space scales -Varying dimensionality

15. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Models describing the plasma vary in complexity EUFORIA PHYSICS

16. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Models describing the plasma vary in complexity Turbulence simulation 3d space (can also be kinetic) Edge transport simulation 2d space

17. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Developing a new paradigm for fusion computing Scientific Workflow GRIDHPC Visualization - Building on e-infrastructure tools, middleware and installations - Integrating tools and physics models together with a ”fusion simulation ontology” - At least initially building on fusion de facto standards for data access and communication

18. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 EUFORIA 14 member Institutes 522pms covering - Management - Training - Dissemination - Grid and HPC infra- structure & support - Code adaptation & optimization -Workflows -Visualization

19. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Work plan outline Jan 2008 Dec 2010 Mixed Workflows Grid Testbed Deployment Grid Appliance Proof of concept runs Application Porting Standardization and integration Development and deployment Migrating

20. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Grid Infrastructure Centers with infrastructure in EUFORIA  IFCA in Santander, Spain  KIT in Karlsruhe, Germany  Chalmers University, Sweden  Ciemat in Trujillo and Madrid, Spain Current Status: ~3100 CPUs, ~2TB online #CPU Free Total Jobs Running Waiting ComputingElement ---------------------------------------------------------- 640 1372027 197 1830svea-gl2.c3se.chalmers.se:2119/jobmanager-pbs-svea 568 268 0 0 0iwrce2.fzk.de:2119/jobmanager-lcgpbs-i2gpar 228 139 0 0 0ce01-tic.ciemat.es:2119/jobmanager-lcgpbs-euforia 1616 1616 0 0 0gridce01.ifca.es:2119/jobmanager-sge-euforia 48 48 0 0 0 ce-euforia.ceta-ciemat.es:2119/jobmanager-lcgpbs-euforia Avail Space(Kb) Used Space(Kb) TypeSEs ---------------------------------------------------------- 1000 n.a n.a storm.ifca.es 114395396 100297660 n.a griddpm01.ifca.es 114395091 100297964 n.a griddpm01.ifca.es 1820000000 21058536 n.a iwrse2.fzk.de 13488960 5528882 n.a svea-gl3.c3se.chalmers.se Also shared resources with Fusion VO EUFORIA grid supporting parallel apps and Interactive grid (i2g) extensions.

21. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 HECToR(EPCC, UK) –Cray XT4 (integrated with X2 vector system in a single machine). XT4: 11,328 cores. Processor: 2.8GHz Opteron; 33.2TB RAM. 59Tflops theoretical peak. X2: 112 vector processors; 2.87Tflops theoretical peak–No. 29 in Top 500 (54.65 TflopsLINPACK) MareNostrum(BSC, Spain) –IBM Cluster. 10240 cores. Processor: 2.3GHz PPC 970; 20TB. 94.21Tflops peak. –No.26 in Top 500 (63.83 TflopsLINPACK) Louhi(CSC, Finland)–Cray XT4. 4,048 cores. 4.5TB RAM. 37.68Tflops peak.–No. 70 in Top 500 (26.80 TflopsLINPACK) With DESA ~3Mcpuhours 2009

22. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Example Fusion Workflow One example of a workflow Many variants are possible Different modules have very different computational requirements  sub-second/iteration on a single cpu  hour(s)/iteration on a large number of cpus Need to visualize the results EQUIL ETS ECRHICRHTRANSPNEUTRALS SAW TEETH EQUIL? ELM(c) CORE2EQ SOURCE_COMBINER ELM(t) TRANSPORT_COMBINER NEO Scientific Workflow GRIDHPC Visualization

23. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Coupling codes and applications N modules integrated in N different applications N modules coupled into a dynamic application framework Adapted from David De Roure Number of coupled codes/physics modules in typical fusion applications can be large: - Chain1 model at JET ~85 coupled applications in a DAG like structure - Detailed predicitve transport modelling – 25 different modules with multiple versions and iterative dependencies Still need to support single standalone applications!

24. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 24 (Subset of) codes adapted for Grid use Different code domains and different parallel strategies: GEM: Linear & non linear Turbulence Gyrofluid code (Core Transport). MPI. BIT1: Divertor code (SOL Transport). Parameter Scan EIRENE: Neutral transport for tokamaks & Stellarators (Neutral Transport). MC Code. DAB (Distributed Asinchronous Bees): Tool for the optimization of any concept in fusion devices. Asynchronous Algorithm. Plus the previously gridified Codes (Taken from EGEE code Platform): Suitable for Workflows. - ISDEP (MC-Transport), Mishka & Helena (Equilibrium and MHD), VMEC (3D-Equilibrium suitable for tokamaks and sellarators).

25. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Enhancing the grid usage Application specific support tools for enhancing the grid usage. TAPAS (Tool for Automatic Parameter Studies) and TAPAS4GRID : allows parameter scans automatically on HPCs and grids. Used for EMC3-EIRENE and BIT1. S.t.a.r.t. (Simple Tool for Application Runs and Transfer of data) provides a command line interface that hides all the complexity of job management in the grid. It also simplifies the inclusion of required libraries for the submitted jobs. Tools are also building block for improved inclusion of applications in Workflows

26. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Generic Application Structure Workflow Orchestration “Global” Ontology Modelling Tools/Modules I/O Organisation Resource Interfaces (cluster, grid, HPC,...) Code Interfaces Data transport Data structures Physics Statistics Numerics Large scale fusion applications tend to be long lived: - How to build software that lasts 30+ years?

27. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 European Fusion Implementation KEPLER UAL/CPO Integrated Modelling Projects I/O Organisation Resource Interfaces (cluster, grid, HPC,...) Code Interfaces Data transport Data structures Physics Statistics Numerics Modelling tools are either standalone applications OR modules.i.e., subroutines. Automagic tools wrap modules into workflow components (Kepler Actors)

28. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 KEPLER Transport Workflow P.HUYNH, V.BASIUK Scientific Workflow GRIDHPC Visualization

29. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 KEPLER Transport Workflow P.HUYNH, V.BASIUK Scientific Workflow GRIDHPC Visualization End user only care about available resources, suitability to problem and performance, Particular details of the underpinning hw/sw configurations is in general not of interest:  Hide or at least simplify complexity and promote usability from the end user point of view. Lesson learned: Users much more forgiving towards HPC access complexity than they appear to be towards grid access protocols uesability thresholds.

30. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 The EUFORIA Services in the broader view GRID and HPC infrastructures EUFORIA EGEE DEISA

31. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Launching simple Grid jobs Composite actor

32. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Example of composite actor - details Certificates Job submit Job status Get output Allows for arbitrary complex workflows to be built!

33. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804  GRID workflow Kepler on the local cluster launches a simple Kepler workflow on GRID: Gateway RAS on euforia.efda-itm.eu GRID

34. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Workflow + fusion codes on GRID and HPC MHD equilibrium codes Helena on GRID ILSA on HPC DEMO D10 TODAY EUFORIA-EGEE-DEISA Collaboration

35. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Data Management Data model (CPOs) not directly supported by Kepler workflow  Need to manage data outside WF  Universal Access Layer  Built on fusion de facto standards “MDSplus” for transport and storage  Supports in memory, file (HDF5) and database (MDSplus) access Challenges  Larger datasets (order Gb’s and beyond)  Efficiency in distributed environments?  Access model inherits concept of centralized data server (s) and cataloguing. UAL  Extensible user API built on the CPO (“standardized data structures”)  A transport layer support local and remote access  A set of “backend” storage option:s Mdsplus, HDF5, (memory cache),...!

36. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 ITER – an International Collaboration Experimental facility: -10Gbit/s during discharges, 500-1000s - 20PB/year (lower bound estimates) International partners: - Data replication – at least two offsite rep. - (Near) real time data streaming, inline modelling data to remote centers - “Semi” remote operation - Middleware interoperability  agreement on single technology (most interfaces will be centrally managed/decided!) - Resource sharing/policies - IPR challenging issue. Nuclear installation: - Security - licensing “ITER aim is to demonstrate that it is possible to produce commercial energy from fusion.” First plasma 2019, full operation 2025 (!) ~3000-4000 remote participants

37. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Electron Temperature, DIII-D, with RMPs (initially developed for stellarator applications W7AS, W7X, LHD) was advanced to a more flexible grid structure to allow divertor tokamak + RMP applications. first self-consistent 3D plasma and neutral gas transport simulations for poloidal divertor tokamak configurations with RMPs. Simulation results for ITER similar shape plasmas at DIII-D show a strong 3D spatial modulation of plasma parameter, e.g. in T e. EMC3-EIRENE code verification (by benchmarks with 2D tokamak edge codes) and validation (TEXTOR, DIII-D, JET, LHD experiments) ongoing EMC3-EIRENE is currently foreseen for contractual ITER RMP design studies (jointly by FZJ and IPP, 2010…) T e,1 = 60 eV T e,2 = 120 eV T e,3 = 200 eV Towards fully 3D CFD: The EMC3-EIRENE code (IPP Greifswald – FZ-Juelich) Frerichs, H., Reiter, D. et al., Comm. Phys. Commun. (2010) 181 61-70 and: Nuclear Fusion (2010) 50, in print Scientific Workflow GRIDHPC Visualization

38. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 ASCOT calculations of ITER Tuomas Koskela The ASCOT group used the EUFORIA- DEISA resources for simulating fusion alpha particles in ITER with two computing time - intensive upgrades. 1.an experiment-based anomalous diffusion profile 2.full orbit integration near the wall for precision wall hits The goal was to provide insight on the optimal location of Fast Ion Loss Detectors in ITER, which is still under discussion. Scientific Workflow GRIDHPC Visualization 3D power distribution on the wall under anomalous diffusion

39. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Fusion has turned out to be somewhat more complex to master than initially believed Contrary to popular beliefs - we ARE making rapid progress!

40. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Summary The fusion community is benefitting from e- infrastructures and using input from EUFORIA and collaborating project to do better science!  Improvements and porting of single codes GRID HPC  Using workflows  Using visualization tools  Running on the infrastructure provided by GRID –EUFORIA VO sustainable into EGI DEISA and Project Partners  Training and outreach

41. EUFORIA FP7-INFRASTRUCTURES-2007-1, Grant 211804 Thanks Chalmers University of Technology (Coordinator) from Sweden Max Plank Institute for Plasma Physics (IPP) from Germany Centro Superior de Investigaciones Científicas (CSIC) from Spain Centro de Investigaciones Energéticas, Medio Ambientales y Tecnológicas (CIEMAT) from Spain Forschungszentrum Karlsruhe (FZK) from Germany Finnish IT Center for Science (CSC) from Finland Abo Akademi University (ABO) from Finland University of Edinburgh (UEDIN) from United Kingdom Barcelona Supercomputing Center (BSC) from Spain French Atomic Energy Commission (CEA) from France University of Strasbourg from France University of Ljubljana (UOL) from Slovenia Poznan Supercomputing and Networking Center PSNC from Poland Italian National Agency for New Technologies, Energy and the Environment (ENEA) from Italy EGEE DEISA