Michael L. Norman, UC San Diego and SDSC

 A parallel AMR application for astrophysics and cosmology simulations  Hybrid physics: fluid + particle + gravity + radiation  Block structured AMR  MPI or hybrid parallelism  Under continuous development since 1994  Greg Bryan and Mike NCSA  Shared memory  distributed memory  hierarchical memory  C++/C/F, >185,000 LOC  Community code in widespread use worldwide  Hundreds of users, dozens of developers  Version

ASTROPHYSICAL FLUID DYNAMICSHYDRODYNAMIC COSMOLOGY Supersonic turbulence Large scale structure

PhysicsEquationsMath typeAlgorithm(s)Communication Dark matterNewtonian N-body Numerical integration Particle-meshGather-scatter GravityPoissonEllipticFFT multigrid Global Gas dynamicsEulerNonlinear hyperbolic Explicit finite volume Nearest neighbor Magnetic fields Ideal MHDNonlinear hyperbolic Explicit finite volume Nearest neighbor Radiation transport Flux-limited radiation diffusion Nonlinear parabolic Implicit finite difference Multigrid solves Global Multispecies chemistry Kinetic equations Coupled stiff ODEs Explicit BE, implicit None Inertial, tracer, source, and sink particles Newtonian N-body Numerical integration Particle-meshGather-scatter Physics modules can be used in any combination in 1D, 2D and 3D making ENZO a very powerful and versatile code

 Berger-Collela structured AMR  Cartesian base grid and subgrids  Hierarchical timetepping

Level 0 AMR = collection of grids (patches); each grid is a C++ object Level 1 Level 2

Unigrid = collection of Level 0 grid patches

 Shared memory (PowerC) parallel ( )  SMP and DSM architecture (SGI Origin 2000, Altix)  Parallel DO across grids at a given refinement level including block decomposed base grid  O(10,000) grids  Distributed memory (MPI) parallel ( )  MPP and SMP cluster architectures (e.g., IBM PowerN)  Level 0 grid partitioned across processors  Level >0 grids within a processor executed sequentially  Dynamic load balancing by messaging grids to underloaded processors (greedy load balancing)  O(100,000) grids

Projection of refinement levels 160,000 grid patches at 4 refinement levels

1 MPI task per processor Task = a Level 0 grid patch and all associated subgrids; processed sequentially across and within levels

 Hierarchical memory (MPI+OpenMP) parallel (2008-)  SMP and multicore cluster architectures (SUN Constellation, Cray XT4/5)  Level 0 grid partitioned across shared memory nodes/multicore processors  Parallel DO across grids at a given refinement level within a node  Dynamic load balancing less critical because of larger MPI task granularity (statistical load balancing)  O(1,000,000) grids

N MPI tasks per SMP M OpenMP threads per task Task = a Level 0 grid patch and all associated subgrids processed concurrently within levels and sequentially across levels Each grid is an OpenMP thread

ENZO ON CRAY XT51% OF THE SIMULATION  Non-AMR Mpc box  15,625 (25 3 ) MPI tasks, root grid tiles  6 OpenMP threads per task  93,750 cores  30 TB per checkpoint/re- start/data dump  >15 GB/sec read, >7 GB/sec write  Benefit of threading  reduce MPI overhead & improve disk I/O

ENZO ON CRAY XT510 5 SPATIAL DYNAMIC RANGE  AMR Mpc box, 7 levels of refinement  4096 (16 3 ) MPI tasks, 64 3 root grid tiles  1 to 6 OpenMP threads per task to 24,576 cores  Benefit of threading  Thread count increases with memory growth  reduce replication of grid hierarchy data

Using MPI+threads to access more RAM as the AMR calculation grows in size

ENZO-RHD ON CRAY XT5COSMIC REIONIZATION  Including radiation transport 10x more expensive  LLNL Hypre multigrid solver dominates run time  near ideal scaling to at least 32K MPI tasks  Non-AMR and 16 Mpc boxes  4096 (16 3 ) MPI tasks, 64 3 root grid tiles

 Cosmic Reionization is a weak-scaling problem  large volumes at a fixed resolution to span range of scales  Non-AMR with ENZO-RHD  Hybrid MPI and OpenMP  SMT and SIMD tuning  to root grid tiles  4-8 OpenMP threads per task  4-8 TBytes per checkpoint/re-start/data dump (HDF5)  In-core intermediate checkpoints (?)  64-bit arithmetic, 64-bit integers and pointers  Aiming for K cores  M hours (?)

 ENZO’s AMR infrastructure limits scalability to O(10 4 ) cores  We are developing a new, extremely scalable AMR infrastructure called Cello   ENZO-P will be implemented on top of Cello to scale to

 Hierarchical parallelism and load balancing to improve localization  Relax global synchronization to a minimum  Flexible mapping between data structures and concurrency  Object-oriented design  Build on best available software for fault- tolerant, dynamically scheduled concurrent objects (Charm++)

1. hybrid replicated/distributed octree-based AMR approach, with novel modifications to improve AMR scaling in terms of both size and depth; 2. patch-local adaptive time steps; 3. flexible hybrid parallelization strategies; 4. hierarchical load balancing approach based on actual performance measurements; 5. dynamical task scheduling and communication; 6. flexible reorganization of AMR data in memory to permit independent optimization of computation, communication, and storage; 7. variable AMR grid block sizes while keeping parallel task sizes fixed; 8. address numerical precision and range issues that arise in particularly deep AMR hierarchies; 9. detecting and handling hardware or software faults during run-time to improve software resilience and enable software self-management.

 Enzo website (code, documentation)   2010 Enzo User Workshop slides   yt website (analysis and vis.)   Jacques website (analysis and vis.) 

Level 0 xx x Level 1 Level 2 (0,0) (1,0) (2,0) (2,1)

(0) (1,0) (1,1) (2,0) (2,1) (2,2) (2,3) (2,4) (3,0)(3,1)(3,2)(3,4)(3,5)(3,6) (3,7) (4,0) (4,1) (4,3) (4,4) Depth (level) Breadth (# siblings) Scaling the AMR grid hierarchy in depth and breadth

LevelGridsMemory (MB)Work = Mem*(2^level) , ,275114,629229, ,52221,22684, ,4486,08548,680 47,2161,97531,600 53,3701,00632,192 61, , ,808 Total305,881324,860683,807

real grid object virtual grid object grid metadata physics data grid metadata Current MPI Implementation

 Flat MPI implementation is not scalable because grid hierarchy metadata is replicated in every processor  For very large grid counts, this dominates memory requirement (not physics data!)  Hybrid parallel implementation helps a lot!  Now hierarchy metadata is only replicated in every SMP node instead of every processor  We would prefer fewer SMP nodes ( ) with bigger core counts (32-64) (=262,144 cores)  Communication burden is partially shifted from MPI to intranode memory accesses

 Targeted at fluid, particle, or hybrid (fluid+particle) simulations on millions of cores  Generic AMR scaling issues:  Small AMR patches restrict available parallelism  Dynamic load balancing  Maintaining data locality for deep hierarchies  Re-meshing efficiency and scalability  Inherently global multilevel elliptic solves  Increased range and precision requirements for deep hierarchies