1. An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical Thermonuclear Flashes Hands-On Flash Setting up Your Own Simulations

2. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Outline  Setting up an out of the box simulation  Customizing Flash for a new simulation (i.e., how to set up your problem)  Simple visualization and analysis with idl using fidlr3.0

3. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Setting up an out of the box problem

4. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Flash Source to Executable  Flash comes as source code consisting of source code components known as units  Going from Flash source code to executable consists of  The setup step: specifying which source units to include in the application and linking together the source files  The compilation step: running make to compile the flash executable

5. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago The Setup Script The primary job of the setup script is to: qTraverse the FLASH source tree and link necessary files for a given application to the object directory qFind the target Makefile.h for a given machine. qGenerate the Makefile to make the FLASH executable. qGenerate files needed to add runtime parameters to a given simulation. qGenerate files needed to parse the runtime parameter file.

6. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Example Makefile.h Located in FLASH3/sites/ MPI_PATH = /usr/local/mpich-intel HDF5_PATH = /usr/local/hdf5-icc NCMPI_PATH = /usr/local/pnetcdf-icc FCOMP = ${MPI_PATH}/bin/mpif90 CCOMP = ${MPI_PATH}/bin/mpicc CPPCOMP = ${MPI_PATH}/bin/mpiCC LINK = ${MPI_PATH}/bin/mpif90 FFLAGS_OPT = -c -r8 -i4 -O3 -real_size 64# -unroll -align -prefetch -pad -ip FFLAGS_DEBUG = -c -g -r8 -i4 -check bounds -check format -check output_conversion -warn all -real_size 64 FFLAGS_TEST = -c -r8 -i4 -O2 -real_size 64 CFLAGS_OPT = -c -O3 -D_LARGEFILE64_SOURCE CFLAGS_DEBUG = -c -g -debug extended -D_LARGEFILE64_SOURCE CFLAGS_TEST = -c -O2 -D_LARGEFILE64_SOURCE CFLAGS_HDF5 = -I $(HDF5_PATH)/include CFLAGS_NCMPI = -I $(NCMPI_PATH)/include CFLAGS_MPI = -I$(MPI_PATH)/include LFLAGS_OPT = -r8 -i4 -Vaxlib -lsvml -Ur -o LFLAGS_DEBUG = -r8 -i4 -Vaxlib -g -o LFLAGS_TEST = -r8 -i4 -Vaxlib -o LIB_HDF5 = -L $(HDF5_PATH)/lib -lhdf5 -lz LIB_MPI = -L$(MPI_PATH)/lib -lfmpich -lmpich LIB_NCMPI = -L$(NCMPI_PATH)/lib -lpnetcdf

7. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago First Setup Example  From the main flash directory, FLASH3/, type./setup Sedov -2d -auto command problem name dimensionSetup script determines which units to include based on the problem name (rather than manually) To run setup you shouldn’t need to modify the setup script (FLASH3/bin/setup.py)

8. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Setup Options -autosetup script will automatically generate the Units file based on the problem you specify INCLUDE Driver/DriverMain/TimeDep INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_package/headers INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_package/mpi_source INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_package/source INCLUDE Grid/localAPI INCLUDE IO/IOMain/hdf5/serial/PM INCLUDE PhysicalConstants/PhysicalConstantsMain INCLUDE RuntimeParameters/RuntimeParametersMain INCLUDE Simulation/SimulationMain/Sedov INCLUDE flashUtilities/general INCLUDE physics/Eos/EosMain/Gamma INCLUDE physics/Hydro/HydroMain/split/PPM/PPMKernel INCLUDE physics/Hydro/HydroMain/utilities If you don’t use the -auto flag, you must have a valid Units file in the top FLASH directory (FLASH3/Units) Try manually changing IO/IOMain/hdf5/serial/PM to IO/IOMain/hdf5/parallel/PM - Then run setup without the - auto flag Sample Units File

9. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Setup Options -unit= : force a specific unit to be used in setup  Another way to use a specific unit in setup is with the -unit option./setup Sedov -2d -auto -unit=IO/IOMain/hdf5/parallel/PM

10. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago More Setup Options -autoAutomatically generates the Units file -unit= Forces a specific unit to be used -[123]dSpecifies the dimension, default is 2d -nxb= -nyb= -nzb= Specifies number of zones/block, default is 8 -maxblocks= Assigns maxblocks per processor. Defaults are in place but you may want to modify depending on problem and machine specs -site= | -ostype= Allows you to directly specify the host or ostype. Typically setup finds this info but on some machines it isn’t directly accessible. -site=sphere.asci.uchicago.edu or -ostype=Linux -debug | -test-debug: makes compiler put debugging symbols in executable, possibly checks for array out of bounds conditions, etc -test: compiles code with no debugging or optimization options -opt: compiles code for highest performance (default) -portableNormally setup links files from the source directory to the object directory. With -portable files are copied to object dir instead -objdir= By default setup script links all files needed in compilation to the object directory. The -objdir flag allows you to specify a different or new directory -noclobberTo setup a simulation over a previous compiled object directory, so that previously included units are not recompiled, saving compilation time -verboseMore wordy explanations during setup

11. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Even More Setup Options -parfile= This causes setup to copy the specified file in the simulation directory to the object directory as flash.par -datafiles= This causes setup to copy the specified files in the simulation directory to the object directory. This option is used to copy data files required by the application. -unitsfile= This causes setup to copy the specified file to the object directory as Units before setting up the problem. This option can be used when -auto is not used, to specify an alternate Units file.

12. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Setup Shortcuts qThe Flash3 setup script includes a “shortcuts” feature qAllows many setup options to be included with one keyword qDefined in Flash3/bin/setup_shortcuts.txt qTo use a shortcut, add +shortcut to your setup line qExample: The shortcut ug is defined as: ug:--with-unit=Grid/GridMain/:Grid=UG:./setup Sod -auto +ug this is equivalent to typing in unit options with -unit=Grid/GridMain/UG -unit=IO/IOMain/hdf5/serial/UG (because the appropriate IO is included by default) qLook in Flash3/bin/setup_shortcuts.txt for more examples and to define your own

13. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago setup Generated Files contains the list of variables, fluxes, species, particle properties, and mass scalars used in the current setup, together with their descriptions setup_vars contains the list of runtime parameters defined in the Config files processed by setup setup_params contains the exact compiler and linker flagssetup_flags contains a list of all pre-process symbols passed to the compiler invocation directly setup_defines contains the list of all units which were included in the current setup setup_units contains the list of libraries and their arguments (if any) which was linked in to generate the executable setup_libraries contains the options with which setup was called and the command line resulting after shortcut expansion setup_call

14. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Customizing Flash for a new simulation (i.e., how to set up your problem)

15. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Adding a New Simulation  New Simulations are added by creating a directory in FLASH3/source/Simulation/SimulationMain  The Simulation directory gives people working on a particular problem a place to put problem specific code that replaces the default functionality in the main body of the code  It’s also a place to tell the setup script which units this problem will need from the rest of the code  In the Simulation directory, one needs to have:  A Config file to at least describe the units and runtime parameters needed by this problem  One or more flash.par files which each give a set of runtime parameter values for running the problem in different ways  Code specifying initial conditions and code for the simulation as a unit (e.g., initialization of the unit)

16. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Config File Syntax DEFAULT sub-unitEvery unit and subunit designates one implementation to be the ``default''. If no specific implementation of the unit or its sub-units is selected by the application, the designated default implementation gets included EXCLUSIVE implementation… Specify a list of implementations that cannot be included together REQUIRES unit[/sub- unit[/implementation...]] [ OR unit[/sub-unit...]]... Specify a unit requirement. Unit requirements can be general, without asking for a specific implementation, so that unit dependencies are not tied to particular algorithms REQUESTS unit[/sub- unit[/implementation...]] Requests a unit to be added to the Simulation. All requests, are upgraded to a ``REQUIRES'' if they are not negated by a "- without-unit" option from the command line. If negated, the REQUEST is ignored. This can be used to turn off profilers and other ``optional'' units which are included by default. CONFLICTS unit1[/sub- unit[/implementation...]]... Specifies that the current unit, subunit or specific implementation is not compatible with the list of units, subunits or other implemenations list that follows. Setup issues an error if the user attempts to set up a conflicting unit configuration. PARAMETER name type [constant] default [range spec] Specify a runtime parameter. Parameter names are unique up to 20 characters and may not contain spaces. Admissible types include REAL, INTEGER, STRING, and BOOLEAN

17. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago More Config File Syntax VARIABLE name Register variable with the framework with name name. The setup script collects variables from all the included units, and creates a comprehensive list with no duplications. It then assigns defined constants to each variables and calculates the amount of storage required in the data strucures for storing the variables. The defined constants, and the calculates sizes are written to the file Flash.h. LIBRARY name Specify a library requirement. Different FLASH units require different libraries, and they must inform setup so it can link the libraries into the executable. Some valid library names are HDF5, MPI. FLUX name Register flux variable name with the framework. When using adaptive mesh, flux conservation is needed at fine-coarse boundaries. Paramesh uses a data structure for this purpose, the flux variables provide index into that data structure. DATAFILES wildcard Declare that all files matching the given wildcard in the unit directory should be copied over to the object directory. GRIDVAR name This keyword is used in connection with the grid scope scratch space supported by FLASH3 (This feature wasn't available in FLASH2). It lets you ask for scratch space for variables specified with this keyword.

18. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Even More Config File Syntax SPECIES name An application that uses multiple species uses this keyword to define them MASS_SCALAR name If a quantity is defined with keywork MASS_SCALAR, space is created for it in the ``unk'' data structure. It is treated like any other variable by Paramesh, but the hydrodynamic unit treats it differently. It is advected, but other physical characteristics don't apply to it.

19. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Required Code for a New Simulation  There are certain pieces of code that all simulations must implement:  Simulation_initBlock.F90: Sets initial conditions in a single block.  Simulation_init.F90: Reads the runtime parameters, and performs other necessary unit initializations.  Simulation_data.F90: A Fortran module which stores data and parameters specific to the Simulation.  Optionally, a sumulation could implement:  Simulation_initSpecies.F90: To give the properties of the species involved in a multispecies simulation

20. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Customized Code for a new Simulation  In a FLASH simulation directory, you can place code that overrides the functionality you would pick up from other code units  In the custom code you can modify:  Boundary conditions (Grid_applyBCEdge.F90)  Refinement criterion (Grid_markRefineDerefine.F90)  Diagnostic integrated quanties for output (in the flash.dat file), e.g., total mass (a default) or vorticity (IO_writeIntegralQuantities.F90)  Diagnostics to compute new grid scope variables (Grid_computeUserVars.F90)  In general, this is a place to hack the code in ways specific to your problem, and you can hack basically anything

21. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Example Non-trivial Setup: TwoGamma qTwoGamma is for a simple test of advecting two fluids having different gammas to investigate whether an instability develops at the interface between the two fluids. qThis Simulation implements: q Simulation_initSpecies.F90, because it has multiple fluids q and Grid_applyBCEdge.F90, because it needs custom boundary conditions on the lower x edge of the domain

22. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago TwoGamma Config File # configuration file for the TwoGamma target problem REQUIRES Driver REQUIRES physics/Hydro REQUIRES physics/Eos/EosMain/Multigamma REQUIRES Multispecies REQUESTS IO # Parameters D sim_p0 constant pressure PARAMETER sim_p0 REAL 2.5e-0 D sim_rho1 density of the first fluid PARAMETER sim_rho1 REAL 1.0e-0 D sim_rho2 density of the second fluid PARAMETER sim_rho2 REAL 1.0e-0 D sim_cvelx initial velocity PARAMETER sim_cvelx REAL 0.1e-0 SPECIES FLD1 SPECIES FLD2

23. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago TwoGamma flash.par # AMR parameters lrefine_max = 4 lrefine_min = 4 # simulation parameters basenm = "twogamma_" restart =.false. plotFileIntervalTime = 0.1 checkpointFileIntervalTime = 0.5 nend = 15000 tmax = 10.0 checkpointFileNumber = 0 plotFileNumber = 0 dtini = 1.e-10 dtmin = 1.e-10 cfl =.5 cvisc =.1

24. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago TwoGamma flash.par (Cont.) smlrho = 1.e-10 smallt = 1.e-10 xmin = 0.0e0 xmax = 1.0 ymin = 0.0e0 ymax = 1.0e0 geometry = "cartesian" # variables for plotting plot_var_1 = "dens" plot_var_2 = "temp" plot_var_3 = "ener" plot_var_4 = "pres" plot_var_5 = "velx" plot_var_6 = "fld1" plot_var_7 = "fld2" xl_boundary_type = "user" xr_boundary_type = "outflow" yl_boundary_type = "periodic" yr_boundary_type = "periodic"

25. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago TwoGamma Simulation_initSpecies.F90 subroutine Simulation_initSpecies() implicit none #include "Multispecies.h" #include "Flash.h" #include "Multispecies.h" #include "Multispecies_interface.h" call Multispecies_setProperty(FLD1_SPEC, A, 1.) call Multispecies_setProperty(FLD1_SPEC, Z, 1.) call Multispecies_setProperty(FLD1_SPEC, GAMMA, 1.66666666667e0) call Multispecies_setProperty(FLD2_SPEC, A, 4.0) call Multispecies_setProperty(FLD2_SPEC, Z, 2.0) call Multispecies_setProperty(FLD2_SPEC, GAMMA, 2.0) end subroutine Simulation_initSpecies

26. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago TwoGamma Grid_applyBCEdge.F90 if(face==LOW) then select case (bcType) case(OUTFLOW) do i = 1,guard dataRow(i)= dataRow(guard+1) end do case(USER_DEFINED) select case(var) case(GAMC_VAR) dataRow(1:guard)=sim_gammac1 case(DENS_VAR) dataRow(1:guard)=sim_rho1 case(PRES_VAR) dataRow(1:guard)=sim_p0 case(VELX_VAR) dataRow(1:guard)=sim_cvelx case(VELY_VAR) dataRow(1:guard)=0.0 case(VELZ_VAR) dataRow(1:guard)=0.0 case(ENER_VAR) dataRow(1:guard)=max(0.5*(sim_cvelx**2)+sim_int1,sim_small)

27. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Simple visualization and analysis with idl using fidlr3.0

28. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Fidlr (Flash idl Routines) 3.0 q1 and 2d visualization q2d slices of 3d data qHas idl routines to read the HDF5 and NetCDF Flash checkpoint and plotfiles. qxflash3: Gui frontend qCan call load routines from idl command line and manipulate checkpoint data interactively qLives in tools/fidlr3.0 in your source

29. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Setting up and Running idl for fidlr qSetting up environment correctly very important qTo get color, run idl with start_linux: q % idl start_linux q IDL> xflash3 setenv XFLASH3_DIR flash-root-path/tools/fidlr3 setenv IDL_DIR idl-root-path setenv IDL_PATH ${XFLASH3_DIR}:${IDL_DIR}:${IDL_DIR}/lib

30. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago xflash3 – the fidlr GUI

31. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago fidlr Interactive Examples Read in density from file, put it on a uniform grid, and make a contour plot: IDL> dens = loaddata(‘sedov_2d_6lev_chk_0001’, ‘dens’, XCOORDS = x, YCOORDS=y) IDL> contour, dens, x, y Read in a dataset, with full AMR details: IDL> read_amr, ‘sedov_2d_6lev_hdf5_chk_0001’, $ TREE=tree, PARAMETERS=params, DATA=data, STORED_VARS=vars