PARAMESH: A PARALLEL, ADAPTIVE GRID TOOL FOR THE SPACE SCIENCES Kevin Olson (NASA/GSFC and GEST/Univ. of MD, Baltimore) Presented, AISRP PI Meeting April, 2005 NASA/AMES Research Center

COLLABORATORS Peter MacNeice (NASA/GSFC and Drexel U.) Joan Centrella (NASA/GSFC) Don Lamb (U. of Chicago) other developers include: C. Mobarry, R. DeFainchtein, M. Gehmeyer, M. Bhat, C. Packer, M. Rilee, J. VanMetre, D. Choi (NASA/GSFC) R. Devore (NRL), M. Zingale, J. Dursi, K. Riley, A. Siegel, T. Linde, D. Sheeler (U. Chicago) Initial funding provided by NASA/ESTO-CT

TALK OUTLINE I.AN OVERVIEW OF PARAMESH II.AISRP GOALS III.APPLICATIONS OF PARAMESH IV.PROGRESS TOWARD GOALS

An Overview of PARAMESH

PARAMESH: what is it ? A Package designed to ease the task of adding parallelization and dynamic, adaptive mesh refinement (AMR) to an already existing uniform mesh, serial code A library of subroutines and accessible data structures PARAMESH is the basic parallelization and AMR tool for several important space science applications Written in Fortran90 (NAG, Lahey, Intel, Portland Group, HP-Compaq, IBM, SGI) Interprocessor communication using MPI Version 3.3 (beta) released January 2005, version 3.3 (stable) and 3.4 (beta) to be released June of PARAMESH WEB site:

A subset of Berger-Oliger, block-adaptive scheme Computational Volume is recursively bisected into ‘blocks’, forming a tree data structure.

Blocks are ordered and distributed to processors using a space filling curve as in particle tree codes.

Each Block is a logically cartesian, uniform mesh of cells. Each cell in a block can store user specified data at cell centers, corners, edges or faces A 2-D Block of CellsA Single Mesh Cell in 3-D

Support for consistent fluxes and ensuring conservation for finite volume schemes Support for averaging data at cell edges to ensure consistent circulation integrals around cell faces

AISRP GOALS

AISRP Goals ● Extend and improve PARAMESH – Parallel I/O (HDF5, NetCDF), I/O formats for graphics packages such as ChomboVis and Techplot. – C Interface – Improved support for multigrid and linear systems solvers – Improve divergence of B control – Improve user interface for interpolation – Improve support for non-cartesian coordinate systems

AISRP Goals ● Effective Open Source Development – Develop coding standards – Improve automatic testing procedure – Self-documenting comments using ‘Robodoc’ – Developers’ Guide – System for bug and feature request tracking – More formal release and patching strategy ● Integrate new versions of PARAMESH into actual, working, space science applications

SOME SPACE SCIENCE APPLICATIONS USING PARAMESH

NRL AMRMHD3D R. DeVore (NRL) P. MacNeice, K. Olson (NASA/GSFC) Solves the equations of MHD (DeVore, 1991) Code for which PARAMESH was developed Used for solar physics applications Numerical schemes: FCT with constrained transport for MHD and multigrid for implicit formulation of non-linear thermal diffusion

CORONAL MASS EJECTIONS (P. MacNeice et al., Drexel U. and NASA/GSFC)

General Relativity, HAHNDOL J. Centrella, D. Choi, B. Imbiriba, J. Baker, D. Fiske, & J. Van Meter (NASA/GSFC), D. Brown, L. Lowe (N.C. State) Solves Einstein Equations Goal: To simulate gravitational waves resulting from the collision of super-massive black holes in order to help interpret data from LISA mission (to be launched 2011). Numerical Schemes: Multigrid, Finite Difference Major user of Columbia System

Gravitational Wave Propagation (J. Van Metre NASA/GSFC, movie by C. Henze, NASA/AMES)

FLASH ASTROPHYSICS CODE FLASH code team Fryxell et al., 2000, ApJS, 131, 273, Implements various CFD Schemes, MHD, Nuclear Reactions, Stellar Equations of State, and self-gravity using multigrid. Designed to model Astrophysical thermonuclear ‘flashes’ (X-ray bursts, Novae, and Type 1a Supernovae). Awarded 2000 Gordon-Bell Prize (0.25 Tflops on 6,420 processors, ASCI ‘Red’)

Simulation of an X-ray Burst due to a detonation in He atmosphere on a neutron star. (Mike Zingale, U.C. Santa Cruz)

Type Ia Supernova due to an off-center explosion in a white dwarf. (T. Plewa, A. Calder, and D. Lamb, U. of Chicago)

Other Space Science Applications ● CASIM (M. Benna and P. Mahaffy at GSFC) – MHD application for modeling comet-solar wind interaction ● YDFCT (D. Odstrcil at NOAA) – MHD application for modeling multiple interacting CME’s, integrated into CCMC ● ZeusAMR (W. Abbett et al. at U.C. Berkeley) – Combination of Zeus MHD code and PARAMESH for modeling magnetic flux emergence from the sun ● IBEAM (D. Swesty et al. at UIUC/NCSA and SUNY-SB) – Modern Astrophysics framework, radiation hydrodynamics for modeling gamma ray burst fireballs ● Plus others, the list continues to grow!

PROGRESS TOWARD GOALS

Progress Toward Goals ● Parallel I/O checkpointing capability added to PARAMESH using HDF5 (version 3.3) ● Capability added to write files which can be viewed using ChomboVis graphics and analysis package (version 3.3) ● C interface work begun ● Divergence of B control working and in currently released version, user interface being improved ● Non-cartesian coordinates are working in tests now, will be released in a beta version this coming Fall

Progress Toward Goals ● PARAMESH being developed as an open source project through sourceforge.net web site. We are actively seeking developers! ● Sourceforge.net used for bug and feature request tracking. ● Rododoc documentation partially complete and will form a portion of the developers’ guide ● Release policy established for patches and major releases ● Developers’ guide under development

Progress Toward Goals ● HAHNDOL is using latest version of PARAMESH ● FLASH 3.0 development begun and will incorporate the latest version of PARAMESH

CONCLUSIONS Parallel, Adaptive Mesh Refinement has wide applicability in the space sciences PARAMESH provides a useful and flexible tool for adding parallel AMR to a wide variety of applications, allowing the efficient solution of ‘real’ problems We are making good progress toward the goals we promised for the AISRP project.