1. An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical Thermonuclear Flashes What’s FLASH? How does FLASH do it? Katherine M Riley Code Group

2. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Overview qWhat is FLASH qBasics of FLASH2 Architecture qBasics behind a problem setup qWalk away with: q Enough of an idea of these concepts to be able to look at a sample setup and understand what is happening.

3. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH: The Application qTo simulate matter accreted onto the surfaces of compact stars, nuclear ignition of the accumulated (and possibly stellar) material, and subsequent evolution of the star’s interior, surface, and exterior. q Novae (on white dwarf surfaces) q Type 1a supernovae (in white dwarf interiors) q X-ray bursts (on neutron star surfaces)

4. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Results Cellular detonations Compressible turbulence Helium burning on neutron stars Rayleigh-Taylor instability Intracluster interactions Flame-vortex interactions Nova outbursts on white dwarfs White Dwarf deflagration Shocked cylinder qCompressible reactive flow qWide range of length of time scales qMany interacting physical processes qOnly indirect validation possible for the astrophysics qMany people in collaboration

5. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago What FLASH Provides qPhysics q Hydrodynamics q PPM q MHD q Relativistic PPM q Nuclear Physics q Gravity q Cosmology q Particles qInfrastructure q Setup q AMR: Paramesh q Regular testing q Parallel I/O q hdf5, pnetcdf, q Profiling q Runtime and post- processing visualization

6. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago A Little FLASH History qFLASH0 q Paramesh2, Prometheus and EOS/Burn qFLASH1 q Smoothing out the smash q First form of module architecture & inheritance qFLASH2 q Untangle modules from each other (Grid) q dBase q Concept of levels of users qFLASH3 q Stricter interface control & module architecture q Taming the database BAM

7. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH2 Audiences FLASH Application Developer Application Programmer End User Develop physics modules Talk to grid Module communication Initialize setup Boundary conditions Basic data structure (Might add a kernel) Works on just about everything Grid development Data access Architecture

8. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH2 Code Basics qAn application code, composed of units/modules. Particular modules are set up together to run different physics problems. qPerformance, Testing, Usability, Portability qFortran, C, Python, … q 560,000* lines of code q 75% code, 25% comment qVery portable qScaling to 1000’s of procs * Internal Release

9. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Basic Computational Unit : Block qThe adaptive grid is composed of blocks qAll blocks: same dimensions qCover different fraction of the physical domain. qKevin Olson will talk about this more.

10. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago First Look at FLASH q‘The tar-ball’ q ‘source’ directories q modules or groups of modules q Post-processing tools, docs, setups qSetup q FLASH architecture tool q Selects and sets up these modules q Collecting variables, runtime parameters, etc

11. Mesh mesh_init() mesh_guardcells() mesh_updateRef() mesh_fluxConserve() MaterialsHydroSource_termsGravity init() tstep() hydro3d() init() tstep() grav3d() init() tstep() src_terms() eos3d() eos1d() eos() Explicitmhd PPM Diffuse Unsplit WENOPPM constant point_mass Poisson Multigrid Gamma Helmholtz... burn cool heat iso13... Multipole Database dBase_init() dBaseGetData() dBasePutData() dBaseProperty() Particles init() advance() Visualization init() render() IO init() Checkpoint_write() Checkpoint_read() MHD init() Driver init() dBaseGetData() dBasePutData() dBaseProperty() Cosmology init() Structure of FLASH Modules (not exact!)

12. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago What’s a FLASH Module? qFLASH basic architecture unit: Modules q Component of the FLASH code providing a particular functionality q Different combinations of modules are used for particular problem setups q Ex: driver, hydro, mesh, dBase, I/O qFake inheritance by use of directory structure qModules communicate q Driver q Variable Database

13. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Abstract FLASH2 Module 1. Meta-data (Configuration Info) Interface with driver and setup Variable/parameter registration Variable attributes Module Requirements 2. Interface Wrapper Exchange with variable database Prep data for kernels 3. Physics Kernel(s) Single patch, single proc functions written in any language Can be sub-classed FLASH Component Database mesh FLASH Application driver Collection of Flash2 Modules

14. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Module Implementations qFLASH2 Modules are directory trees q source/hydro/explicit/split/ppm qEach level might have source q Source relevant for all directories/implementations below qPreserves interfaces qAllows flexible implementations

15. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Inheritance Through Directories: Hydro inithydrotstep Explicit implemtation shock tstep Split DeltaForm hydro An ‘empty’ hydro init, hydro, tstep are defaults on top of the directory tree. Hydro/Explicit Replaces tstep Introduces ‘shock’ No hydro Implemented yet! Hydro/Explicit/Split hydro implemented Uses general explicit tstep Uses general shock Replaces init init hydro implemtation tstep

16. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago The Module Config File qDeclare solution variables, fluxes qDeclare runtime parameters q Sets defaults qLists required, exclusive modules qConfig files are additive down the directory tree - no replacements

17. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Setup Building an Application MeshDatabase Gravity Source Terms Materials Hydro Particles I/OVis MHD Configuration Tool (Setup) Driver

18. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Setup: Implements Architecture qPython code links together needed physics and tools for a problem q object qTraverses modules to get implementations qDetermines solution data storage list qCreates list of parameters from modules qConfigures Makefiles properly

19. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Accessing the Mesh qPhysics and tools access some scope of mesh qDatabase q Unifies method for module to access data out of its scope q Mesh data q Solution Data, Grid information q Runtime parameters qMesh Interface q Wrappers to the public grid routines q initialize, guardcell, fluxconserve, updateRefinment

20. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Audiences FLASH Application End User Initialize setup Boundary conditions Basic data structure (Might add a kernel) Solidify information so far Modules and setup How do they combine to make create an application? That runs? Go through a little of that.

21. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Simple FLASH Flow Driver Init Evolve TimeStep Evolve Hydro SourceTerms … UpdateGrid Hydro Prepare data for kernels SourceTerms Burn Heat Cool Ioniz… Burn burnBlock dBase Mesh Variable Database Paramesh updateGrid ghostCellFill fluxConserve hydroBlock Function Data

22. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Pulling it All Together qChoose a problem setup qRun setup to configure that problem q Everything is in a new top-level directory q ‘object’ qMake qRun q Flash.par for runtime parameters q Defaults already set from particular modules

23. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Setups qA basic problem setup q Config file q Required physics modules q Flash.par q Default list runtime parameter configuration q Init_block q Initial conditions for the problem set block by block q Many other possible files: q Driver, Refinement algorithms, User defined boundary conditions qAny files in setup take precedence

24. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Provided Driver qProvided: q Second order, state form, strang split Initialize() Loop over timesteps evolvePhysics() timestep() output() visualize() End loop Finalize() flash.F90 qNew drivers q Put in setups q Welcome contributions set time step hydro sourceTerms cosmology radiation particles gravity set time step (repeat physics) Mesh_updateGrid evolve.F90

25. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago What’s Next qCovered the basics of what FLASH can do qNow - How does it do it? q Alan Calder