1. CCP 2007 Brussels, Sept 5-8, 2007 Computational fusion plasma physics L. Villard 1, T.M. Tran 1, S.J. Allfrey 2, P. Angelino 3, A. Bottino 4, S. Brunner 1, W.A. Cooper 1, T. Dannert 1, G. Darmet 3, X. Garbet 3, Ph. Ghendrih 3, V. Grandgirard 3, J.P. Graves 1, R. Hatzky 5, Y. Idomura 6, F. Jenko 4, S. Jolliet 1, M. Jucker 1, X. Lapillonne 1, B.F. McMillan 1, N. Mellet 1, P. Popovich 2, Y. Sarazin 3, O. Sauter 1, B. Scott 4 (1) Centre de Recherches en Physique des Plasmas, Association Euratom - Suisse, EPFL, 1015 Lausanne, Switzerland (2) Center for Multiscale Dynamics, Dept of Physics and Astronomy, UCLA, USA (3) Association Euratom - CEA/DSM/DRFC, Cadarache, France (4) Max-Planck Gesellschaft, Association Euratom - IPP, Garching, Germany (5) Computer Center, IPP, Max-Planck Gesellschaft, Garching, Germany (6) Japan Atomic Energy Agency, Taitou, Tokyo 110-0015, Japan

2. CCP 2007 Brussels, Sept 5-8, 2007 2 Outline 1. Introduction  Fusion; ITER; tokamak 2. Transport  Turbulence [also: talk F. Jenko] 3. Alternative concepts of magnetic confinement  Equilibrium and stability 4. Burning plasma physics. RF waves  Heating, current drive, fast particles [also: talk Y. Peysson] 5. HPC issues  massive parallelism, scalability 6. Outlook and conclusions

3. CCP 2007 Brussels, Sept 5-8, 2007 3 Outline 1. Introduction  Fusion; ITER; tokamak  Magnetised plasmas: time-, length- scales; anisotropy; collective effects; kinetic effects; geometry 2. Transport  Turbulence 3. Alternative concepts of magnetic confinement  Equilibrium and stability 4. Burning plasma physics. RF waves  Heating, current drive, fast particle driven instabilities 5. HPC issues  massive parallelism, scalability 6. Outlook and conclusions

4. CCP 2007 Brussels, Sept 5-8, 2007 4 Fusion … in space (a nursery of stars: NGC 6357) Credit: NASA, ESA and J. M. Apellániz (IAA, Spain)NASAESAIAA

5. CCP 2007 Brussels, Sept 5-8, 2007 5 Controlled fusion on earth: magnetic confinement Figure of merit: density x Temperature x energy confinement time (nT  E ) Magnetic confinement :  E ~1s, n~10 20 m -3, T~10keV. ITER: nT  E =3.4 atm.s Source: EFDA-JET Source: EFDA and J.B. Lister CRPP JET has produced 16MW of fusion power

6. CCP 2007 Brussels, Sept 5-8, 2007 6 ITER: the way to fusion EU, Japan, USA, China, India, South Korea, Russian Federation 5.3G€ construction cost (1/3 G € /y = 0.06% of world overall R&D) Virtually inexhaustible, environmentally benign source of energy:  from Deuterium and Lithium; gives Helium. (Tritium is recycled) Source:ITER The plasma will be there (see next p) You are here

7. CCP 2007 Brussels, Sept 5-8, 2007 7 Magnetic confinement: tokamak Larmor radius  L Trapped particle Passing particle particle trajectory

8. CCP 2007 Brussels, Sept 5-8, 2007 8 Complexity: many nonlinearities Geometry of magnetic configuration is an essential feature of fusion plasma physics Neutral Beams RF current drive heat fusion heat transport turbulence microscopic stability heat transformer coils induction equilibrium macroscopic stability operational limits bootstrap

9. CCP 2007 Brussels, Sept 5-8, 2007 9 Timescales in the ITER plasma machine lifetime 1 shot energy confinement turbulence ion cyclotron electron cyclotron Physics spans several orders of magnitude Direct Numerical Simulation (DNS) of “everything” is unthinkable  Need to separate timescales using approximations  How to integrate all phenomena in a consistent manner?

10. CCP 2007 Brussels, Sept 5-8, 2007 10 Net energy transfer from the wave to the particles if Collisionless Landau damping Kinetic effects: wave-particle interaction Surfers with velocity just below the phase velocity of the wave will be accelerated -> momentum and energy transfer Surfers with velocity too different from the phase velocity of the wave will not ride the wave General: distribution function in 6D phase space To be solved with consistent electromagnetic fields

11. CCP 2007 Brussels, Sept 5-8, 2007 11 Outline 1. Introduction  Fusion; ITER; tokamak 2. Transport  Turbulence [also talk by F. Jenko]  Numerical methods and related issues 3. Alternative concepts of magnetic confinement  Equilibrium and stability 4. Burning plasma physics. RF waves  Heating, current drive, fast particles [also talk by Y. Peysson] 5. HPC issues  massive parallelism, scalability 6. Outlook and conclusions

12. CCP 2007 Brussels, Sept 5-8, 2007 12 Transport Hot plasma core, colder edge: => grad T If grad T > (grad T) crit => instabilities  small scales, i.e. spatial scales ~  L Growth, then nonlinear saturation of modes  => turbulent state  => transport (heat, particles, momentum)  “anomalous”, essentially by collisionless processes Turbulent transport >> collisional transport In spite of this, typical grad T ~ 10 6 K / cm Need a theory … and numerical tools!

13. CCP 2007 Brussels, Sept 5-8, 2007 13 Turbulence – gyrokinetic theory (GK) Assume Average out the fast motion of the particle around the guiding center Fast parallel motion, slow perpendicular motion (drifts) Strong anisotropy of turbulent perturbations (// vs perp to B) ion-drivenelectron-drivenZonal Flows phase space dimension reduction 6D -> 5D

14. CCP 2007 Brussels, Sept 5-8, 2007 14 Core fusion plasmas are very weakly collisional ( mfp,// >> device size) How can we obtain a turbulence-induced heat flux in a system without explicit dissipative term? Issue for computations: numerical dissipation must be small enough in order to make correct predictions  in the collisionless limit  and to isolate the physical collisional effects from numerical dissipation effects Collisionless gyrokinetics ? Spatial scales bounded by Larmor radius and device size => Direct Numerical Simulations (DNS) are possible, but: Collisionless (Landau) wave-particle interactions lead to filamentation in phase space An infinitesimal dissipation leads to finite power transfer x v

15. CCP 2007 Brussels, Sept 5-8, 2007 15 Solving GK: 3 classes of numerical methods Lagrangian: Particle In Cell, Monte Carlo  Sample phase space (markers) and follow their orbits  Noise accumulation is difficult to control in long simulations Semi-Lagrangian  Fixed grid in phase space, trace orbits back in time  Multi-dimensional interpolation is difficult Eulerian  Fixed grid in phase space, finite differences for operators  CFL condition; overshoot versus dissipation Common to all three: Poisson (+Ampere) field solver  Various methods: finite differences, finite elements, FFT,… Developing several approaches is not a luxury. The complexity of the system implies that all codes are run near the limits of their capability. Therefore cross-comparisons are a necessity.

16. CCP 2007 Brussels, Sept 5-8, 2007 16 Ion Temperature Gradient driven turbulence Resolution: 128x448x320, N = 83 million,  i  t=40 ~6 h on 1024 processors IBM BG/L@EPFL  f :solve for the perturbed distribution function only (control variates) Lagrange: PIC,  f, finite elements: ORB5 large scale Zonal Flows => breakup of turbulent eddies => self-regulation of turbulence turbulent eddies => radial transport hot core cold edge [S. Jolliet, et al., CPC 2007] Color contours of perturbed electrostatic potential

17. CCP 2007 Brussels, Sept 5-8, 2007 17 Gyrokinetic ordering High k // modes are unphysical => filter them out! Perturbations on a magnetic surface Small scales perpendicular to B Long scales parallel to B Next page Color contours of perturbed electrostatic potential

18. CCP 2007 Brussels, Sept 5-8, 2007 18 Fighting noise with field -aligned Fourier filter (FAFF) Turbulent perturbations. Detail on a magnetic surface. Without FAFF: unphysical small scales in the parallel direction develop With FAFF: unphysical small scales in the parallel direction are suppressed [S. Jolliet, et al., CPC 2007]

19. CCP 2007 Brussels, Sept 5-8, 2007 19 Fighting noise with field -aligned Fourier filter (FAFF) Heat flux versus time Without FAFF With FAFF Without FAFF, the noise-generated heat flux ends up dominating the simulation [S. Jolliet, et al., CPC 2007]

20. CCP 2007 Brussels, Sept 5-8, 2007 20 With sources – long simulation Numerical noise is controlled over very long time scales One should verify that the source term does not affect the physics Heat transport Signal / noise Without source: turbulence decay, grad T -> close to marginal stability

21. CCP 2007 Brussels, Sept 5-8, 2007 21 Semi-Lagrange approach Main motivation: get the better of two worlds  avoid the statistical noise problem inherent of Lagrange- PIC-MC methods  avoid the CFL condition inherent of Euler methods Interpolation of f @ foot of orbit Orbit, backward integrated from grid point GYSELA code:  5D gyrokinetic  Operator splitting  Leapfrog scheme, with averaging of integer and half-integer timesteps f=const along orbit

22. CCP 2007 Brussels, Sept 5-8, 2007 22 Lagrange (ORB5) vs ½-Lagrange (GYSELA) Heat diffusivity Alternative approaches bring useful cross-checks of numerical and physical results Fit from other codes (Dimits) Heat diffusivity Ion temperature gradient

23. CCP 2007 Brussels, Sept 5-8, 2007 23 Eulerian methods [also: GENE code, F. Jenko] Recent improvement: a skew-symmetric finite difference operator that conserves both L1 and L2 norms [Morinishi, et al., JCP 143, 90 (1998)]. G5D code [Idomura et al., JCP 2007]  Better stability against spurious oscillations  Better robustness for long-time simulations  Exact particle number conservation  Good energy conservation properties:  Lagrange-Euler benchmark: comparable cpu time, memory usage for Euler ~ 5 times larger. Euler L1&L2 Lagrange PIC  f

24. CCP 2007 Brussels, Sept 5-8, 2007 24 Eulerian, semi-Lagrangian and Lagrangian simulations confirm the role of zonal flows Idomura, et al., JCP 2007 Zonal flow velocity profile in a cylindrical plasma Zonal flow velocity profile of Jupiter’s high atmosphere NASA Cassini 2000

25. CCP 2007 Brussels, Sept 5-8, 2007 25 The plasma edge Occurrence of “transport barrier”: “pedestal”-like n and T profiles with steep gradients, leading to overall improved confinement.  An indispensable ingredient for the success of ITER A. Kendl, B.D. Scott, PoP 13, 012504 (2006) X-point separatrix The plasma cross- section is shaped: elongation, triangularity As we approach the edge, the geometry becomes more complex:

26. CCP 2007 Brussels, Sept 5-8, 2007 26 Edge turbulence Electron response departs significantly from Boltzmann Perturbations acquire an electromagnetic character Collisions play a non negligible role Gyrofluid ElectroMagnetic model: 6-moment of gyrokinetic equations. GEM code [B. Scott, PoP 2005]. Mechanisms linking geometry and turbulence Plasma cross-section elongation leads to reduced turbulent transport fluxes; small effect of triangularity . [Kendl Scott PoP 2006]

27. CCP 2007 Brussels, Sept 5-8, 2007 27 Outline 1. Introduction  Fusion; ITER; tokamak 2. Transport  Turbulence [see also talk by F. Jenko] 3. Alternative concepts of magnetic confinement  Equilibrium and stability 4. Burning plasma physics. RF waves  Heating, current drive, fast particle driven instabilities 5. HPC issues  massive parallelism, scalability 6. Outlook and conclusions

28. CCP 2007 Brussels, Sept 5-8, 2007 28 Beyond the tokamak, beyond ITER Inside the largest stellarator : LHD (NIFS, Japan) Unlike the tokamak, the stellarator is an intrisically steady-state magnetic confinement concept (does not need a net plasma current, does not need a transformer), but…

29. CCP 2007 Brussels, Sept 5-8, 2007 29 Stellarator Axial symmetry breaking implies several challenges: Does an equilibrium with nested magnetic surfaces exist? Are charged particles confined? What are the pressure stability limits? How to heat? RF? Neutral Beam Injection? Intensive numerical simulations are required The W7-X stellarator is now under construction in Greifswald (IPP) Provide rotational transform by external coil currents

30. CCP 2007 Brussels, Sept 5-8, 2007 30 Neutral Beam Injection heating Leads to anisotropic fast ion distributions  here an example of computation for LHD Mod-B Hot ion pressure HFS deposition Hot ion pressure LFS deposition  th =2.6%  hot =1.4%

31. CCP 2007 Brussels, Sept 5-8, 2007 31 Outline 1. Introduction  Fusion; ITER; tokamak 2. Transport  Turbulence [see also talk by F. Jenko] 3. Alternative concepts of magnetic confinement  Equilibrium and stability 4. Burning plasma physics. RF waves  Heating, current drive, fast particle driven instabilities 5. HPC issues  massive parallelism, scalability 6. Outlook and conclusions

32. CCP 2007 Brussels, Sept 5-8, 2007 32 Alfvén Eigenmode in W7-X Global wave code LEMAN: DNS of Maxwell’s equations (finite elements, Fourier)  Solves 4-potential ( ,A) in order to avoid spectral pollution Toroidicity-induced Alfvén Eigenmode  Could be destabilised by fast ions f=65.3kHz =4.2T

33. CCP 2007 Brussels, Sept 5-8, 2007 33 Ion Cyclotron RF heating in LHD # radial finite elements = 200, # Fourier modes = 630,  matrix storage (band) memory size = 490 GB Electric field normal component Power absorption density f=5.57MHz, =0.8T, Deuterium plasma, n 0 =4x10 19 m -3 f=f ci

34. CCP 2007 Brussels, Sept 5-8, 2007 34 Outline 1. Introduction  Fusion; ITER; tokamak 2. Transport  Turbulence [see also talk by F. Jenko] 3. Alternative concepts of magnetic confinement  Equilibrium and stability 4. RF waves  Heating, current drive, fast particle driven instabilities 5. HPC issues  massive parallelism, scalability 6. Outlook and conclusions

35. CCP 2007 Brussels, Sept 5-8, 2007 35 Parallelization scheme – ORB5 + Transpose operations (2D Fourier transforms of fields) Domain decomposition (toroidal direction) Domain Cloning: field quantities (density, potential) are replicated MPI parmove MPI global sum N pes = N d *N c particles fields

36. CCP 2007 Brussels, Sept 5-8, 2007 36 Massive parallelism Lagrange ORB5 code. Strong scaling with 8x10 8 particles (left). Weak scaling with 8x10 5 particles/proc (right).  IBM BG/L in co-processor mode  6.4 G particles: enough to simulate the whole core of ITER Euler GENE code: parallel efficiency 89% in strong scaling, 99% in weak scaling from 2k to 32k procs. 6.4 G particles

37. CCP 2007 Brussels, Sept 5-8, 2007 37 Conclusion and outlook Fusion plasma physics is an extraordinary rich and diverse field This diversity is reflected in the various theoretical and numerical approaches Turbulence simulations [talk F. Jenko] and fast particles physics [talk Y. Pesson] remain very challenging topics Future developments will go in the direction of including more self-consistent nonlinear interactions in multiscale simulations Numerical codes for the simulation of fusion plasmas are mature for the next generation of HPC platforms

38. CCP 2007 Brussels, Sept 5-8, 2007 38 Multiscale turbulence with GENE: The role of sub-ion-scales motivated by experiments, in particular, “transport barriers” extremely resource demanding (millions of CPU-hours per run if using realistic mass ratios) sub-ion-scales (which have been neglected so far) can indeed be important GENE simulation (reduced mass ratio)

39. CCP 2007 Brussels, Sept 5-8, 2007 39 Simulation meets experiment ExperimentGENE simulation Gyrokinetic simulations are getting realistic enough to be compared directly with experiments…

40. CCP 2007 Brussels, Sept 5-8, 2007 40 Gyrokinetic theory Averaging over gyromotion. 6D->5D. Distribution function f(R,v //,  ) Perp nonlinearity (~  E x B) Parallel nonlinearity (~  E // ) Parallel motion Grad-B drifts Equilibrium E x B drifts If EM perturbations are considered, Ampere’s equation must be solved too If collisions and sources are included: df/dt=C(f)+S

41. CCP 2007 Brussels, Sept 5-8, 2007 41 Solving GK: 3 classes of numerical methods Lagrangian: Particle In Cell, Monte Carlo  Sample phase space (markers) and follow their orbits  Project to real space => charge and current densities Semi-Lagrangian  Fixed grid in phase space  Trace characteristics back in time Eulerian  Fixed grid in phase space  Finite difference discretization of operators Common to all three: Poisson (+Ampere) field solver  Various methods: finite differences, finite elements, FFT, … Having several codes with different approaches is not a luxury. The complexity of the system implies that all codes are run near the limits of their capability. Therefore cross-comparisons are a necessity.

42. CCP 2007 Brussels, Sept 5-8, 2007 42 Lagrange: numerical (statistical) noise  Can “drown” the physical fluxes  Noise accumulation difficult to control in long simulations Semi-Lagrange: multi-dimensional interpolation  Difficult to generalize when time-splitting is not applicable  Overshoot problem vs numerical dissipation / dispersion Euler: CFL condition  Brings constraint on  t  Overshoot problem vs numerical dissipation / dispersion Common to all: massive parallelization is desirable  … up to 1000’s processors Solving GK: 3 different types of issues

43. CCP 2007 Brussels, Sept 5-8, 2007 43 Lagrange approach. Particle In Cell.  f Define  f=f-f 0, with f 0 a time-independent part Sample phase space with “particles” (markers) (“Monte Carlo”) Integrate the markers orbits  Runge-Kutta 4 th order; Burlish-Stoer;... Project markers onto real space to obtain perturbed density and current Solve field equations (Poisson; Maxwell)  Finite elements; Fourier transform; filter;…

44. CCP 2007 Brussels, Sept 5-8, 2007 44 ORB5 - Fighting noise with FAFF: Energy conservation With FAFF the conservative properties are preserved with an accuracy increased by orders of magnitude Without FAFF With FAFF Violation of energy conservation [S. Jolliet, et al., CPC 2007]

45. CCP 2007 Brussels, Sept 5-8, 2007 45 ORB5 - Fighting noise with FAFF: avoid drowning Turbulent energy spectrum time evolution With FAFF, the spurious generation of turbulence and zonal flows is strongly reduced Without FAFF, there is an unphysical secular generation of turbulence and zonal flows caused by numerical noise [S. Jolliet, et al., CPC 2007]

46. CCP 2007 Brussels, Sept 5-8, 2007 46 Semi-Lagrangian 5D code GYSELA  c t=25000 (R- R 0 )/  i Z/  i (R-R 0 )/  i With ZF, ITG streamers are broken V. Grandgirard, et al., Varenna 06 X. Garbet et al., IAEA 2006 (r, , ,v//,  )=(256, 256,16,32,8)  i  t=5, 100hrs on 64 proc Without ZF, ITG streamers persist

47. CCP 2007 Brussels, Sept 5-8, 2007 47 Eulerian methods [also: GENE code, F. Jenko] Recent improvement: a skew-symmetric finite difference operator that conserves both L1 and L2 norms [Morinishi, et al., JCP 143, 90 (1998)]. G5D code [Idomura et al., JCP 2007]  Better stability against spurious oscillations  Better robustness for long-time simulations  Exact particle number conservation  Good energy conservation properties:  Lagrange-Euler benchmark: comparable cpu time, memory usage for Euler ~ 5 times larger. Euler L1&L2 Lagrange PIC  f

48. CCP 2007 Brussels, Sept 5-8, 2007 48 Burning Plasma Physics. RF waves RF macroscopic stability operational limits fusion Neutral Beams rotation transport turbulence microscopic stability current drive heat transformer coils induction equilibrium bootstrap

49. CCP 2007 Brussels, Sept 5-8, 2007 49 Burning Plasma Physics Fusion  -particles heat the bulk plasma  => pressure profile  => bootstrap current => current profile  => affect the transport (i.e. transport barriers) Fusion  -particles interact with waves  => affect internal relaxation phenomena (sawteeth, fishbones)  => affect the stability of eigenmodes of the system, i.e. Alfvén Eigenmodes, and/or create Energetic Particle Modes  => which in turn can provoke losses of  -particles Complex non-linear modelling challenge

50. CCP 2007 Brussels, Sept 5-8, 2007 50 Parallelization scheme – ORB5 + Transpose operations (2D Fourier transforms of fields) Domain decomposition (toroidal direction) Domain Cloning MPI parmove MPI global sum N pes = N d *N c (0,0)(0,1) (0,N d -1) (1,0)(1,1)(1,N d -1) (N c -1,0)(N c -1,1) (N c -1,N d -1) particles fields

51. CCP 2007 Brussels, Sept 5-8, 2007 51 Parallelization scheme – ORB5