1. An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical Thermonuclear Flashes Transitioning from FLASH2 to FLASH3 Katie Antypas June 4, 2006

2. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Overview qFLASH3 Enhancements qTransitioning to FLASH3 qWhere to get help

3. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Motivation qMake FLASH capable of being a community developed and supported code by cleaning up code architecture and defining clear unit interfaces qBuild on positive aspects of FLASH2 qMake FLASH code everlasting even if FLASH Center isn’t. qFirst opportunity to focus on design rather than capabilities

4. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Goals qConfigurable q Flexibility to easily configure the code for new applications q A traditional strong point of FLASH qExtensible q Users can add functionality quickly qVerifiable q Users should be able to test their applications easily q The FLASH test suite was release as a stand alone application in FLASH3 qWell Documented q A necessity for any community code that hopes to gain broad acceptance q FLASH3 include: q Extensive online API documentation q Improved Users Guide q Examples of transitions between FLASH2 and FLASH3

5. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Release qIncludes all infrastructure units (Grid, IO, Driver, etc) qIncludes basic physics units (Hydro, gamma EOS) qExtensive Web-based documentation q User’s guide q Reference Tips q Howto’s q Developer’s guide q Public Unit API documentation q Runtime parameters documentation qTest Suite, including use of unit test framework

6. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Future Planned Releases q Closely spaced to make new functionality available to users as soon as possible q Alpha release September 2006 q Particles (Lagrangian and active) q Nuclear burning q Eos Helmholtz q Simple gravity q MHD q Beta release December 2006 q 3.0 release April 2007 q Full FLASH2 functionality with new FLASH3 capabilities

7. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements : Interchangeable Grids Paramesh Uniform Grid  One block per processor  No AMR related overhead Patch Based Mesh (planned)  Adaptive mesh with variable block size  Block Structured  Fixed sized blocks  Specified at compile time  Not more than one level jump at fine course boundaries

8. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements : API Interface qEach unit has a well defined API (in CS terms each unit has a set of public routines which other units can call) qExamples for Grid unit: q Grid_fillGuardCells q Grid_conserveFluxes q Grid_updateRefinement q Grid_moveParticles q Grid_get/putBlkData q Grid_getNumProcs qEach routine is fully documented with argument and algorithm descriptions!!

9. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Robodocs API Grid_conserveFluxes

10. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements : Decentralized Data qIn FLASH2 centralized database was the key way to access all data q Positives: easy for developers to get data they needed quickly and easily q Negatives: not clear who owned the data, when it was last modified or if it was safe to modify data qIn FLASH3 in lieu of a centralized database each unit contains a FORTRAN module to hold data specific to that unit q Still have accessor functions so developers can get data easily q Now more clear who owns the data q Much of the data stored in the FLASH2 database is located in the Grid_data module in FLASH3 or in the Flash.h file, (more later) q Holds myPE, numProcs, access to grid variables, coordinates etc.

11. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Example Flash.h File #define DENS_VAR 1 #define EINT_VAR 2 #define ENER_VAR 3 #define GAMC_VAR 4 #define GAME_VAR 5 #define PRES_VAR 6 #define TEMP_VAR 7 #define VELX_VAR 8 #define VELY_VAR 9 #define VELZ_VAR 10 #define MFRAC_SPEC 11 #define NPROP_VARS 10 #define NSPECIES 1 #define NMASS_SCALARS 0 #define NUNK_VARS (NPROP_VARS + NSPECIES + NMASS_SCALARS) #define E_FLUX 1 #define EINT_FLUX 2 #define P_FLUX 3 #define RHO_FLUX 4 #define U_FLUX 5 #define UT_FLUX 6 #define UTT_FLUX 7 #define NPROP_FLUX 7 #define NSPECIES_FLUX 1 #define NMASS_SCALARS_FLUX 0 #define NFLUXES (NPROP_FLUX + NSPECIES_FLUX + NMASS_SCALARS_FLUX) #define NDIM 2 #define MAXBLOCKS 1000 #define TEMP_SCRATCH_GRID_VAR 1 #define NSCRATCH_GRID_VARS 1 qThe Flash.h file is written by the setup script for your specific application qDeclares variable and flux indices in multidimensional datastructures qDefines, number of dimensions, maximum blocks on a proc and much more... variable indices flux indicies

12. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Example Putting Data onto Grid FLASH2 FLASH3 idens = dbaseKeyNumber(‘dens’) call dbasePutData(idens,....) use index directly call Grid_putBlkData(DENS_VAR,....)

13. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Grid Directory Structure Grid API level Stub File ie. No implementation All routines called from other Units belong in the API GridMain UGparamesh PM2PM3 Grid_fillGuardCells Grid_getCellVolume Grid_conserveFluxes Grid_getBlkNeighbors Grid_getGlobalNumBlks and many more... Grid_getGlobalNumBlks Implementation levels All other levels hold the implementation for the API routines Both the Uniform Grid and Paramesh have the same implementation for Grid_getGlobalNumBlks The Uniform Grid has no implementation for Grid_conserveFluxes and so when that routine is called from the Hydro Unit, it just returns the empty stub Grid_getBlkNeighbors Grid_conserveFluxes

14. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Unit Structure qDistinguish between public and private routines q Public Routines q Must have stub implementation q Naming convention, start with “Unit_verbWhatUnitDoes” ie “Grid_putBlkData” q Private Routines q No stubs unless necessary for inter-subunit communication q Naming convention, small case abrev. for unit ie “gr_createDataTypes” qEach unit has a ‘UnitName’_data module to hold unit scope data qEach unit has a ‘UnitName’_init routine to initialize data stored in ‘UnitName’_data module q All initializations that were in “if firstcall =.true.” blocks, now belong in the “UnitName_init” routine which is called once at initialization time In FLASH2, the user was free to organize a code unit however he or she wanted with very few constraints. In FLASH3 we implemented a number of ‘strong conventions’ (rules) for easier code maintenance, development and usability.

15. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Moving a FLASH2 module to FLASH3 qCreate a ‘UnitName’_data FORTRAN module to store data needed from other units in the code. qCreate storage in ‘UnitName’_data for all necessary variables storing runtime parameters or physical constants. qEliminate the “first call” constructs, move them to the UnitName_init. qUse wraper functions to fill up the remaining variables in Unit_data.

16. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Impact on user qWe’ve taken steps to assure the transition isn’t too painful q Lots of documentation of Flash code support website q If you are having any problems please email us. We can help. qMost kernels are the same as in FLASH2 qA user, who has modified source files in setup: q should be able to read Unit API documentation to clearly determine how to make the code compatible with FLASH3 or q Look at the new source file implementation for how to start on converting to FLASH3 qIf you’re in the process of developing a deep functional module for FLASH2 that you want to contribute, talk to us now

17. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements: Unit Testing qFLASH3 has a unit testing framework and comes with several unit tests for the Grid, RuntimeParmeters, EOS, Multispecies and PhysicalConstants qUsers can easily add in new unit tests qAttempt to test units independently, however some units can not be meaningfully tested alone q Most unit tests require the Grid unit q Multigamma Eos, requires Eos and Multispecies qSetup examples: q./setup unitTest/Eos -auto q./setup unitTest/Grid -auto qSimple Driver can be written to test unit In any large complex code it is often difficult pin point where coding problems exist. Unit testing allows small sections of code to be tested independently, targeting the debugging process and giving the user more confidence in the complete application.

18. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements: The New Test Suite qFlash test suite runs a variety of problems to validate the code on a daily basis qEssential for immediately identifying bugs unintentionally introduced into the code qNew test suite was released with FLASH3 so users can monitor their own research problems and performance q test parameters easily modified through GUI q handles unit tests, compilation tests, comparison tests q automatically uploads test suite data to benchmarks database Green light indicates all runs were successful Date of run Platform Floating statistics box gives immediate overview of results Red light indicates 1 or more tests failed

19. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Editing of “test.info” file done right in browser “test.info” file holds all necessary information for running a FLASH problem Expandable graphic allows user to easily modify all problems in test suite Control Panel for modification of “test.info” file tree Info File Manager

20. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Test Suite Performance History The new test suite adds the ability to upload selected test log files to the database to track daily code performance. History of 2D and 3D Sedov ProblemsDetails of 2D History

21. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements : Enhanced Setup Script qSeamless library management qAbility to retain object files from one setup to the next qSuggests units a user might have neglected qUsers can setup shortcuts More about setup script options in next talk

22. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Framework How do you get the Framework out of FLASH 3? q Include the source tree up to the top level of each Unit. q Stub implementations q Config files q Include source/Simulation/SimulationMain/Default directory q Include the setup tool q Include the Driver q Unit tests could be included, but have no meaning with stubs

23. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago … which brings us to Discussion and Questions