1. PARAMESH: A PARALLEL, ADAPTIVE GRID TOOL FOR THE SPACE SCIENCES Kevin Olson (NASA/GSFC and GEST/Univ. of MD, Baltimore) Presented, AISRP PI Meeting April, 2005 NASA/AMES Research Center

2. COLLABORATORS Peter MacNeice (NASA/GSFC and Drexel U.) Joan Centrella (NASA/GSFC) Don Lamb (U. of Chicago) other developers include: C. Mobarry, R. DeFainchtein, M. Gehmeyer, M. Bhat, C. Packer, M. Rilee, J. VanMetre, D. Choi (NASA/GSFC) R. Devore (NRL), M. Zingale, J. Dursi, K. Riley, A. Siegel, T. Linde, D. Sheeler (U. Chicago) Initial funding provided by NASA/ESTO-CT

3. TALK OUTLINE I.AN OVERVIEW OF PARAMESH II.AISRP GOALS III.APPLICATIONS OF PARAMESH IV.PROGRESS TOWARD GOALS

4. An Overview of PARAMESH

5. PARAMESH: what is it ? A Package designed to ease the task of adding parallelization and dynamic, adaptive mesh refinement (AMR) to an already existing uniform mesh, serial code A library of subroutines and accessible data structures PARAMESH is the basic parallelization and AMR tool for several important space science applications Written in Fortran90 (NAG, Lahey, Intel, Portland Group, HP-Compaq, IBM, SGI) Interprocessor communication using MPI Version 3.3 (beta) released January 2005, version 3.3 (stable) and 3.4 (beta) to be released June of 2005. PARAMESH WEB site: http://ct.gsfc.nasa.gov/paramesh/Users_manual/amr.html

6. A subset of Berger-Oliger, block-adaptive scheme Computational Volume is recursively bisected into ‘blocks’, forming a tree data structure.

7. Blocks are ordered and distributed to processors using a space filling curve as in particle tree codes.

8. Each Block is a logically cartesian, uniform mesh of cells. Each cell in a block can store user specified data at cell centers, corners, edges or faces A 2-D Block of CellsA Single Mesh Cell in 3-D

9. Support for consistent fluxes and ensuring conservation for finite volume schemes Support for averaging data at cell edges to ensure consistent circulation integrals around cell faces

10. AISRP GOALS

11. AISRP Goals ● Extend and improve PARAMESH – Parallel I/O (HDF5, NetCDF), I/O formats for graphics packages such as ChomboVis and Techplot. – C Interface – Improved support for multigrid and linear systems solvers – Improve divergence of B control – Improve user interface for interpolation – Improve support for non-cartesian coordinate systems

12. AISRP Goals ● Effective Open Source Development – Develop coding standards – Improve automatic testing procedure – Self-documenting comments using ‘Robodoc’ – Developers’ Guide – System for bug and feature request tracking – More formal release and patching strategy ● Integrate new versions of PARAMESH into actual, working, space science applications

13. SOME SPACE SCIENCE APPLICATIONS USING PARAMESH

14. NRL AMRMHD3D R. DeVore (NRL) P. MacNeice, K. Olson (NASA/GSFC) Solves the equations of MHD (DeVore, 1991) Code for which PARAMESH was developed Used for solar physics applications Numerical schemes: FCT with constrained transport for MHD and multigrid for implicit formulation of non-linear thermal diffusion

15. CORONAL MASS EJECTIONS (P. MacNeice et al., Drexel U. and NASA/GSFC)

16. General Relativity, HAHNDOL J. Centrella, D. Choi, B. Imbiriba, J. Baker, D. Fiske, & J. Van Meter (NASA/GSFC), D. Brown, L. Lowe (N.C. State) Solves Einstein Equations Goal: To simulate gravitational waves resulting from the collision of super-massive black holes in order to help interpret data from LISA mission (to be launched 2011). Numerical Schemes: Multigrid, Finite Difference Major user of Columbia System

17. Gravitational Wave Propagation (J. Van Metre NASA/GSFC, movie by C. Henze, NASA/AMES)

18. FLASH ASTROPHYSICS CODE FLASH code team Fryxell et al., 2000, ApJS, 131, 273, www.flash.uchicago.edu Implements various CFD Schemes, MHD, Nuclear Reactions, Stellar Equations of State, and self-gravity using multigrid. Designed to model Astrophysical thermonuclear ‘flashes’ (X-ray bursts, Novae, and Type 1a Supernovae). Awarded 2000 Gordon-Bell Prize (0.25 Tflops on 6,420 processors, ASCI ‘Red’)

19. Simulation of an X-ray Burst due to a detonation in He atmosphere on a neutron star. (Mike Zingale, U.C. Santa Cruz)

20. Type Ia Supernova due to an off-center explosion in a white dwarf. (T. Plewa, A. Calder, and D. Lamb, U. of Chicago)

21. Other Space Science Applications ● CASIM (M. Benna and P. Mahaffy at GSFC) – MHD application for modeling comet-solar wind interaction ● YDFCT (D. Odstrcil at NOAA) – MHD application for modeling multiple interacting CME’s, integrated into CCMC ● ZeusAMR (W. Abbett et al. at U.C. Berkeley) – Combination of Zeus MHD code and PARAMESH for modeling magnetic flux emergence from the sun ● IBEAM (D. Swesty et al. at UIUC/NCSA and SUNY-SB) – Modern Astrophysics framework, radiation hydrodynamics for modeling gamma ray burst fireballs ● Plus others, the list continues to grow!

22. PROGRESS TOWARD GOALS

23. Progress Toward Goals ● Parallel I/O checkpointing capability added to PARAMESH using HDF5 (version 3.3) ● Capability added to write files which can be viewed using ChomboVis graphics and analysis package (version 3.3) ● C interface work begun ● Divergence of B control working and in currently released version, user interface being improved ● Non-cartesian coordinates are working in tests now, will be released in a beta version this coming Fall

24. Progress Toward Goals ● PARAMESH being developed as an open source project through sourceforge.net web site. We are actively seeking developers! ● Sourceforge.net used for bug and feature request tracking. ● Rododoc documentation partially complete and will form a portion of the developers’ guide ● Release policy established for patches and major releases ● Developers’ guide under development

25. Progress Toward Goals ● HAHNDOL is using latest version of PARAMESH ● FLASH 3.0 development begun and will incorporate the latest version of PARAMESH

26. CONCLUSIONS Parallel, Adaptive Mesh Refinement has wide applicability in the space sciences PARAMESH provides a useful and flexible tool for adding parallel AMR to a wide variety of applications, allowing the efficient solution of ‘real’ problems We are making good progress toward the goals we promised for the AISRP project.