1. An Accelerated Strategic Computing Initiative (ASCI) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical Thermonuclear Flashes The Flash Code Bruce Fryxell Leader, Code Group Year 3 Site Review Argonne National Laboratory Oct. 30, 2000

2. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Outline Talk 1 – Bruce Fryxell Overview of Flash Adaptive Mesh Refinement Performance and Scaling Year 3 Integrated Calculation Talk 2 – Paul Ricker Current production version of Flash Flash Code architecture Flash physics modules Code verification Talk 3 – Andrew Siegel Development version of Flash The future of Flash

3. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago The Flash Code Group Bruce Fryxell Group Leader Andrew Siegel Code Architect Architecture Team Physics Modules Development, Maintenance, Testing Caceres, Ricker, Riley, Vladimirova, Young Calder, Dursi, Olson, Ricker, Timmes, Tufo, Zingale Calder, Linde, Mignone, Olson, Ricker, Timmes, Tufo, Weirs, Zingale

4. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Overview of Flash Mesh Hydro Nuclear Burning EOSGravityDiffusion Driver Time Dependent Steady Initialization Parallel I/O

5. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Year Three Upgrades  Evolution to object-oriented code architecture  P. Ricker, A. Siegel talks  PARAMESH  PARAMESH 1  “SHMEM” emulation replaced by native MPI  Unnecessary barriers removed  PARAMESH 2 (K. Olson poster)  Elimination of permanent guard cell storage  Capability to advance solution at all refinement levels instead of just at leaf blocks  Adaptivity in time  Guard filling in one direction at a time  New and upgraded physics modules  P. Ricker talk, many posters

6. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Other Accomplishments  Parallel I/O  HDF 5  10x improvement in I/O throughput  Documentation  Comprehensive user manual  http://flash.uchicago.edu/flashcode/doc  The physics and algorithms used in Flash  http://flash.uchicago.edu/flashcode/pubs  Code release  Friendly users – May 2000  Astrophysics Community – Oct. 2000

7. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Adaptive Mesh Refinement  Reduces time to solution and improves accuracy by concentrating grid points in regions which require high resolution  PARAMESH (NASA / GSFC)  Block structured refinement (8 x 8 x 8 blocks)  User-defined refinement criterion – currently using second derivatives of density and pressure

8. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Flash / PARAMESH Block Guard Cells Interior Cells

9. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago PARAMESH Tree Structure  Each block contains n d zones in d dimensions  Blocks stored in 2 d -tree data structure  Factor of 2 refinement per level  Blocks assigned indices via space-filling curve 1345 2 6 789 10 1112 13 14 1516 17 18 192021 10 11 6 18 12 2 8 7 9 1419 20 21 1 4 5 13 16 15 17 3 Refinement Level

10. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Load Balancing  Work weighted Morton space filling curve  Performance insensitive to choice of space filling curve  Refinement and redist- ribution of blocks every four time steps

11. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Example – X-ray Burst

12. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Performance Optimization  Single processor tuning  Reduction in number of square roots and divides  Loop fusion to eliminate unneeded arrays  Elimination of scratch arrays  Removal of unnecessary array copies and initializations  Replacement of string comparisons by integer comparisons  Use of vendor-supplied math libraries  Modification of often-used routines to permit in-lining on ASCI Red  Result  90 Mflop/s on 250 MHz R10000 (64 bit)

13. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Performance Optimization  Parallel optimization  Use of Jumpshot and to identify problem areas  Removal of unnecessary barriers  Packing of small messages in tree portion of code  Result  Good scaling to 1024+ processors  238 Gflop/s on 6420 processors of ASCI Red for the year 3 integrated calculation  2000 Gordon Bell prize finalist

14. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Scaling qConstant work per processor scaling q Shock tube simulation q Two-dimensional q Hydrodynamics, Adaptive Mesh Refinement, gamma-law equation of state q Relatively high communication to computation cost

15. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Scaling - Constant Work Per Processor Flash 1.6 – May 30, 2000

16. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Scaling - Constant Work Per Processor

17. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Scaling - Constant Work Per Processor

18. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Scaling qFixed problem size scaling q Cellular detonation q Three-dimensional q Uses most of the major physics modules in the code q Relatively low communication to computation cost

19. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Scaling – Fixed Problem Size

20. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Scaling – Fixed Problem Size

21. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Summary of Scaling  As number of blocks per processor decreases, a larger fraction of the blocks must get their guard cell information from off processor  This causes deviation from ideal scaling when the number of blocks per processor drops too low  Of the three ASCI machines, this effect is most noticeable on Red, due to its relatively small memory per processor  Significant variation in timings on Nirvana between identical simulations

22. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Summary of Scaling  Significant improvement in cross-box scaling on Nirvana can be achieved by tuning MPI environment variables  Scalability on Blue Pacific is highly dependent on operating system revisions  Parallel efficiency for memory bound jobs  > 90% on Blue Pacific and Red  > 75% on Nirvana  Typical performance – 10-15% of peak on 1024 processors

23. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Integrated Calculation Cellular Detonation In A Type Ia Supernova See also: J. Truran talk F. Timmes poster Evening demos

24. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Why a Cellular Detonation?  Two of our target astrophysics calculations (X-ray bursts and Type Ia Supernovae) involve detonations  We can not resolve the structure of the detonation front in a calculation which contains the entire star  Want to do a study of a small portion of the detonation front to see if a subgrid model is necessary to compute  The detonation speed  The nucleosynthesis  This problem exercises most of the major modules in the code and thus serves as a good test of the overall code performance

25. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Integrated Calculation  1000 processors on ASCI Blue Pacific  Effective grid size (if fully refined)  256 x 256 x 5120 = 335 million grid points  Actual grid size  6 million points at beginning of calculation  45 million points at end of calculation  Savings from using AMR  40-50x for first half of calculation  7x at end of calculation  Total wall clock time ~ 70 hours

26. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Integrated Calculation  Generated 1.2 Tbyte of data  Half of wall clock time required for I/O  0.2 Tbyte transferred to ANL by network for visualization  Used GridFTP to transfer files  7 parallel streams to 7 separate disks  Throughput ~ 4 Mbytes/s  Total transfer time < 1 day

27. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Integrated Calculation 6 level 5 level 10 5 0 2 4 6 8 10 12 Simulation Time (10 -8 s)

28. The ASCI/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Summary  Substantial progress made in Year 3 in improving and extending Flash  Flash is now being used to address many of our target astrophysics problems and is producing important scientific results  Flash achieves good performance on all three ASCI computers and scales to thousands of processors  Large 3D integrated calculation completed on ASCI Blue Pacific and data successfully transferred back to Chicago for analysis and visualization