1. Large scale simulations of astrophysical turbulence Axel Brandenburg (Nordita, Copenhagen) Wolfgang Dobler (Univ. Calgary) Anders Johansen (MPIA, Heidelberg) Antony Mee (Univ. Newcastle) Nils Haugen (NTNU, Trondheim) etc. (...just google for Pencil Code)

2. 2 Pencil Code Started in Sept. 2001 with Wolfgang Dobler High order (6 th order in space, 3 rd order in time) Cache & memory efficient MPI, can run PacxMPI (across countries!) Maintained/developed by many people (CVS!) Automatic validation (over night or any time) Max resolution so far 1024 3, 256 procs

3. 3 Pencil formulation In CRAY days: worked with full chunks f(nx,ny,nz,nvar) –Now, on SGI, nearly 100% cache misses Instead work with f(nx,nvar), i.e. one nx-pencil No cache misses, negligible work space, just 2N –Can keep all components of derivative tensors Communication before sub-timestep Then evaluate all derivatives, e.g. call curl(f,iA,B) –Vector potential A=f(:,:,:,iAx:iAz), B=B(nx,3)

4. 4 Switch modules magnetic or nomagnetic (e.g. just hydro) hydro or nohydro (e.g. kinematic dynamo) density or nodensity (burgulence) entropy or noentropy (e.g. isothermal) radiation or noradiation (solar convection, discs) dustvelocity or nodustvelocity (planetesimals) Coagulation, reaction equations Homochirality (reaction-diffusion-advection equations) Other physics modules: MHD, radiation, partial ionization, chemical reactions, selfgravity

5. 5 Pencil Code check-ins

6. 6 High-order schemes Alternative to spectral or compact schemes –Efficiently parallelized, no transpose necessary –No restriction on boundary conditions –Curvilinear coordinates possible (except for singularities) 6th order central differences in space Non-conservative scheme –Allows use of logarithmic density and entropy –Copes well with strong stratification and temperature contrasts

7. 7 (i) High-order spatial schemes Main advantage: low phase errors

8. 8 Wavenumber characteristics

9. 9 Higher order – less viscosity

10. 10 Less viscosity – also in shocks

11. 11 (ii) High-order temporal schemes Main advantage: low amplitude errors 3 rd order 2 nd order 1 st order 2N-RK3 scheme (Williamson 1980)

12. 12 Shock tube test

13. 13 Hyperviscous, Smagorinsky, normal Inertial range unaffected by artificial diffusion Haugen & Brandenburg (PRE, astro-ph/0402301) height of bottleneck increased onset of bottleneck at same position

14. 14 256 processor run at 1024 3

15. 15 MHD equations Induction Equation: Magn. Vector potential Momentum and Continuity eqns

16. 16 Vector potential B=curlA, advantage: divB=0 J=curlB=curl(curlA) =curl2A Not a disadvantage: consider Alfven waves B-formulation A-formulation 2 nd der once is better than 1 st der twice!

17. 17 Comparison of A and B methods

18. 18 Wallclock time versus processor # nearly linear Scaling 100 Mb/s shows limitations 1 - 10 Gb/s no limitation

19. 19 Sensitivity to layout on Linux clusters yprox x zproc 4 x 32  1 (speed) 8 x 16  3 times slower 16 x 8  17 times slower Gigabit uplink 100 Mbit link only 24 procs per hub

20. 20 Why this sensitivity to layout? 0123456789012345 678901234 All processors need to communicate with processors outside to group of 24

21. 21 Use exactly 4 columns 0123 4567 891011 12131415 16171819 20212223 0123 4567 891011 12131415 Only 2 x 4 = 8 processors need to communicate outside the group of 24  optimal use of speed ratio between 100 Mb ethernet switch and 1 Gb uplink

22. 22 Fragmentation over many switches

23. 23 Pre-processed data for animations

24. 24 Ma=10 supersonic turbulence

25. 25 Animation of B vectors

26. 26 Animation of energy spectra Very long run at 512 3 resolution

27. 27 MRI turbulence MRI = magnetorotational instability 256 3 w/o hypervisc. t = 600 = 20 orbits 512 3 w/o hypervisc.  t = 60 = 2 orbits

28. 28 Fully convective star

29. 29 Geodynamo simulation

30. 30 Homochirality: competition of left/right Reaction-diffusion equation

31. 31Conclusions Subgrid scale modeling can be unsafe (some problems) –shallower spectra, longer time scales, different saturation amplitudes (in helical dynamos) High order schemes –Low phase and amplitude errors –Need less viscosity 100 MB link close to bandwidth limit Comparable to and now faster than Origin 2x faster with GB switch 100 MB switches with GB uplink +/- optimal

32. 32 Transfer equation & parallelization Analytic Solution: Ray direction Intrinsic Calculation Processors

33. 33 The Transfer Equation & Parallelization Analytic Solution: Ray direction Communication Processors

34. 34 The Transfer Equation & Parallelization Analytic Solution: Ray direction Processors Intrinsic Calculation

35. 35 Current implementation Plasma composed of H and He Only hydrogen ionization Only H - opacity, calculated analytically No need for look-up tables Ray directions determined by grid geometry No interpolation is needed

36. 36 Convection with radiation