High performance astrophysical simulations using hybrid multiscale and multiphysics methods Simon Portegies Zwart University of Amsterdam Frontiers in Numerical Gravitational Astrophysics

A Simulation that includes all physics would be as hard to interpret as nature Simulations should capture the fundamental principles of the physical system

11.4 GigaFLOPs scores 4-2 against 1 PetaFLOPs Do FLOPSs matter?

Size of a neutron star/cluster size: Black hole orbital time scale / liftime of the Universe Why is it such a hard problem

Modeling Cluster Dynamics Continuum methods gas sphere (Bettwieser & Sugimoto, Spurzem) direct Fokker-Planck (Cohn, Takahashi) Particle methods N-body “brute force”(Aarseth; NBODY6++; Starlab) Tree code (Barnes & Hut, Gadget) direct Monte-Carlo (Henon, Spitzer; Freitag, Giersz, Heggie, Rasio) hybrid Monte-Carlo (Giersz & Spurzem)

Why Direct N-body? Cons very expensive: Monte-Carlo:O(N) per relaxation time fast multipole:O(N 2 / log N) tree code:O(N 2 ) direct N-body:O(N 3.5 / log N) pros no simplifying assumptions naturally allows inclusion of complex stellar physics direct summation may be necessary to model relaxation and near equilibrium systen with negative hear capacity

M80 Arches Quintplet R13 6 NGC pc ~ Msun Trapezium Westerlund1 MGG-11 5pc Pleiades

Evolving stellar cluster Newton's law of gravitation. No match for approximate methods

But they are not point masses...

Direct integration: 2.4Xona FLOPS 2,400,000,000,000,000,000,000,000,000FLOPS Plus a little overhead for stellar evolution

SL Coffee daily at 7.30 (SL-time) in SpacePort Bravo

A LLNL IBM BG/L Total ~10PFLOPs in TOP500 But only ~10% is academic 1976, Cray1s 110MFLOP

Junichiro Makino, 2000, 1.3TFLOPs 1995: GRAPE-4 with 115 GFlops 1996: GRAPE-4 with 333GFlops 1999: GRAPE-5 price-performance 2000: GRAPE-6 with 1.3TFlops 2001: GRAPE-6 with 11.6TFlops 2003: GRAPE-6 with 40TFlops 2008, 1.3TFLOPs, GRAPE- DR Look at me, look at me, look at me now You can have fun, but you have to know how.

Programming model for GPU Preprocessor Host code GPU code GPU compiler C++ compiler CUDA library CUDA runtime library CUDA driver User application GPU program GPUCPU PCI-Ex

A collision between a binary and asingle star (Gaburov, Lombardi & PZ 2008) Stellar masses: (34, 14), 41Ms Separation: 25Rsun

Registers Shared memory Global memory Acquire thread characteristics (identifiers) Read j-particles Read i-particles Save results Compute number or parts, offsets etc. Calculate a, j and potential All particles done?

Non-recursive stack based tree-walk while (stack.not_empty) { node = stack.pop ;; obtain next node from the stack one = fetch(children, cache) ;; get first four children from cache two = fetch(children, texture) ;; get last four children from texture memory test_cell (node, one.x, stack) ;; test sub-cell in the 1st octant test_cell (node, one.y, stack).... etc.... test_cell (node, two.y, stack)... etc.... }

GRAPE-6Af ~ 6000Euro nVidia 8800GTX ~ 300Euro Sapporo: Gaburov, Harfst, SPZ (In preparation) Kirin: SPZ, Belleman, Bedorf, Geldof (2007), Belleman, Bedorf, SPZ (2008)

Gaburov in prep 2008

Gaburov etal in prep N=3M Plummer run on GPU with BH-treecode Host: Multipole moments GPU: Force, treewalk, integration

Cost $100Million 10 $0.1M$10M ASCI-Q (LANL) $1M Personal computer Sub-PC performance Super-PC performance Earth simulator $0.001M $0.01M TFLOPs

Application range of GPUs in (astro)physical calculations ● Operational: – Gravitational stellar dynamics (~200GLOPS) – Smoothed Particles Hydrodynamics (~120GLOPS) ● In progress: – High-precision gravitational dynamics (E. Gaburov in prep ) – Cold dark matter cosmology ( D. Groen in prep ) – Stellar evolution ( PhD defense E Glebbeek Utrecht 25 July 2008 )

Will GPUs revolutionarize scientific computing? ● The low cost for high performance ( buck-to-FLOP ratio ) ● With CUDA now much easier to program ● For the environmentally concerned: much fewer CO2 per FLOP than supercomputer ● The limited (single) precision limits the applicability ● But GPUs are easily put in cheap Beowulf computers

Time N-body simulation on GRAPE compared with GPU N=8k equal mass Plummer sphere Gaburov etal in prep

Gemmeke etal 2006; Faber etal 2008 The dynamics in dense star clusters is highly irregular. The unrestricted 3-body problem is intrinsically chaotic. A star cluster, consisting of a million stars (like Omega Cen) shows therefore a wide complexity of orbital characteristics.

High-performance grid computer for multi-scale and multi-physics modeling

High-performance grid computer for multi-scale and multi-physics modeling Harfst etal Groen etal

Plummer sphere, direct N-body, Ring algorithm Single GRAPE local host Global GRAPE Grid Gualandris etal 2006 Groen etal 2007

Smoothed Particles Hydrodynamics Some particle properties are determined by taking the average over neighboring particles The fluid is represented by a particle system Fluid dynamics h Each particle has: mass, position, velocity, acceleration, density, pressure, temperature, chemical composition,...

Compute particle acceleration Define averages: Define acceleration:

Strengths of SPH: Method is Lagrangian; high resolution at high density Easy to interface to N-body codes (especially tree codes) Method is simple, easy to code Code always runs (robust) Weaknesses of SPH: Method is Lagrangian; poor resolution in low density Code always runs (may gives misleading results) Shocks may be unresolved Slow (require >100 particles within smoothing length) Very diffusive See Petros Koumoutsakos' talk for more weaknesses...

Conclusions Stellar systems must be modeled with a muti-physics implementation This includes, stellar dynamics, stellar evolution and hydrodynamics This can be run on a grid In particular since each of the applications has specific hardware-dependent requirements The GPU proves to have excellent specifications