An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical Thermonuclear Flashes Architecture, Inheritance and Naming Conventions

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Basic Computational Unit : Block qThe grid is composed of blocks qFLASH 2 All blocks: same dimensions qFLASH3 : blocks can be different sized qCover different fraction of the physical domain.

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago First Look at FLASH q ‘source’ directories q Units or groups of units q Post-processing tools, bin, sites q Setup q FLASH architecture tool q Parses Config files in the units q Selects and sets up these units q Collecting variables, runtime parameters, etc

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago What’s a FLASH Unit? qFLASH basic architecture unit q Component of the FLASH code providing a particular functionality q Different combinations of units are used for particular problem setups q Publishes a public interface for other units’ use. q Ex: Driver, Grid, Hydro, IO etc qFake inheritance by use of directory structure qInteraction between units governed by the Driver qNot all units are included in all applications

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Units Driver I/O Runtime Params Grid Profiling Logfile Simulation Infrastructure monitoring Hydro BurnGravity MHD physics

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Unit Hierarchy Unit API stubs/localAPI Impl_1 Remaining API impl kernel UnitMain Common API implementation UnitSomething API implementation kernel Impl_2 Remaining API impl kernel Impl_3 Remaining API impl common More common impl

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Example of a Unit - Grid Grid GridSolversGridMainGridParticles UG Paramesh2Paramesh3 paramesh PM3_package UGparamesh MoveSieveMoveTree MultipolemultigridPfft Paramesh local API Why Local API ? Grid_init calls init functions for all subunits, if subunit is not included code won’t build.

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Inside a Unit The top level qThe first capitalized directory in a branch of the source tree is a unit qContains stubs for every public function (API) in the unit q Does not contain the data module (unit scope data) q Individual API functions may be implemented in different subunits q A unit has a minimum three functions in its API, no limit on the maximum q Unit_init, Unit_finalize and the “do-er” function for the unit qIf necessary, contains a directory for the local API qMay contain the unit test q Different Unit tests can reside at different levels in the unit hierarchy q The config file contains minimal information, no runtime parameters except “useUnit” defined qMakefile includes All the API functions.

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Subunits qEvery unit has a UnitMain subunit, which must be included in the simulation if the unit is included. q Has the implementations for the init, finalize and the main “do-er” function q Also contains the unit scope data module qThe API functions and private functions implemented in different subunits are mutually exclusive qSubunits other than UnitMain may have private Unit scope functions that can be called by other subunits. q un_suInit and un_suFinalize are the most common ones q (naming convention explained later) qSubunits can also have private data modules, strictly within the scope limited to the specific subunit qSubunits can have their own unit tests

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago More on Subunits qA subunit may have multiple alternative implementations qAlternative implementations of UnitMain also act as alternative implementations of the Unit. qSome subunits have multiple implementations that could be included in the same simulation q GridSolver is one possible example. q Alternative implementations are specified using the “EXCLUSIVE” directive qThe “KERNEL” keyword indicates that subdirectories below that level need not follow FLASH architecture, and the entire subtree should be included in the simulation

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Naming Convention qNamespace directories are capitalized, organizational directories are not qAll API functions of unit start with Unit_ (i.e.Grid_getBlkPtr, Driver_initFlash etc) qSubunits have composite names that include unit name followed by a capitalized word describing the subunit (i.e. ParticlesMain, ParticlesMapping, GridParticles etc) qPrivate unit functions and unit scope variables are named un_routineName (i.e. gr_createDomain, pt_numLocal etc) qPrivate functions in subunits other than UnitMain are encouraged to have names like un_suRoutineName, as are the variables in subunit scope data module qConstants are all uppercase, usually have preprocessor definition, multiple words are separated by an underscore. q There are two type of constants, permanent in “constants.h” or “Unit.h”, and the ones in “Flash.h” generated by the setup script. The significance of capitalizing unit names A new unit can be added without the need to modify the setup script. If the setup script encounters a top level capitalized directory without an API function to initialize the unit, it issues a warning.

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Functional Component in Multiple Units qExample Particles q Position initialization and time integration in Particles unit q Data movement in Grid unit q Mapping divided between Grid and Particles qSolve the problem by moving control back and forth between units Driver Init Evolve Particles Init Map Evolve Grid Init Map Move

The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Inheritance qInheritance implemented through directory structure and Config file directives understood by the setup script qA child directory inherits all functions from the parent directory q If the child directory has its own implementation of a function, it replaces the inherited one. q The implementation in the lowest level offspring replaces all implementations in higher level directories. q An implementation in the “Simulation/MyProblem” directory overrides all implemenations when running MyProblem qConfig files arbitrate on multiple implementations through “Default” keyword qRuntime environment is created by taking a union of all variables, fluxes, and runtime parameters in Config files of included directories. q Value given to a runtime parameter in the “Simulation/MyProblem/Config” overrides any value given to it in other Config files q Value in “flash.par” overrides any value given in any Config file Multiple Config file initial values of a runtime parameter in units other than the simulation unit can lead to non-deterministic behavior since there are no other precedence rules.