1. Overview for Application Programmer The Cactus Team Albert Einstein Institute cactus@cactuscode.org

2. Outline of Overview n Required software and supported architectures n Cactus Computational Toolkit n Application Thorn API n (Infrastructure Thorn API) n Getting the code n Configurations and compilation n Running n Testing

3. Required Software For Cactus Flesh and Computational Toolkit: n Perl 5.0 www.perl.org n GNU make www.gnu.org n C/C++e.g. GNU compilers www.gnu.org For default parallel driver thorn (CactusPUGH/PUGH): n MPIe.g. MPICH www-unix.mcs.anl.gov/mpi/ For Fortran thorns: n Fortran 77e.g. g77 www.gnu.org n Fortran 90e.g. Vast90 www.psrv.com/lnxf90.html Recommended for checkout: n CVSwww.gnu.org

4. Supported Architectures In principle, Cactus runs on any architecture: l Flesh written in ANSI C l Autoconf/Perl deal with machine specific features l Just add Known Architectures files for some new machines/compilers l Usual problems with portability of C++ thorns (e.g. templates) Supported on all architectures we have access to l Laptops l Workstations l Clusters l Supercomputers

5. Supported Architectures

6. Cactus Computational Toolkit CactusBase l Boundary, IOUtil, IOBasic, CartGrid3D, IOASCII, Time CactusBench l BenchADM CactusConnect l HTTPD, HTTPDExtra CactusExample l WaveToy1DF77, WaveToy2DF77 CactusElliptic l EllBase, EllPETSc, EllSOR, EllTest CactusPUGH l Interp, PUGH, PUGHReduce, PUGHSlab CactusPUGHIO l IOFlexIO, IOHDF5, IsoSurfacer CactusTest l TestArrays, TestCoordinates, TestInclude1, TestInclude2, TestComplex, TestInterp, TestReduce CactusWave l IDScalarWave, IDScalarWaveC, IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90 external l IEEEIO, RemoteIO, TCPXX,jpeg6b

7. In Development AlphaThorns n Cart3d n IOHDF5 n Renderer n SimpleWorm n isosurfacer n MDS n TestFunctions BetaThorns n IOASCIIStream n IOHDF5Util n IOJpeg n IOPanda n IOStreamedHDF5 n Socket

8. Application Thorn API Flesh-Thorn and Thorn-Thorn contracts: l Cactus data types l Configuration files for each thorn detailing their requirements and capabilities –parameters, variables, scheduling, functions l Cactus grid hierachy variables l (Parallel) Driver API l IO API l Interpolation/Reduction API l Errors, warnings and info API l Utilities

9. Cactus Data Types n Variables and parameters shared between thorns and Flesh declared using Cactus data types n Ensures portability and consistency between platforms n Size of basic types chosen at compile time (can easily run different precisions)

10. Thorn Configuration Files n Each thorn contains three configuration (ccl) files, which detail the thorn’s interface with the Flesh and other thorns n These files are parsed (Perl) during compilation by the CST (Cactus Specification Tool), which generates wrappers, argument lists (Fortran!), parameter lists, a run list etc.

11. interface.ccl Implementation: the name of this contract with the flesh Inherits, friends: access variables from other thorns Variable groups: my variables Scope: private, restricted, global Type: grid function, grid array, grid scalar Data type: int, real, complex, etc. and Timelevels, Dimension, Ghostzones, Size, Distribution,... What does my thorn “do” What are my thorns grid variables? What variables do I need from other thorns? What variables am I going to make available to other thorns?

12. Example: interface.ccl Implements: matrix_add Inherits: matrix public: real matrices type=GF dim=2 timelevels=2 { matrix1, matrix2 } “Matrices to add together” private: int tempvals type=ARRAY dim=1 distrib=CONSTANT size=arraysize { tta1,tta2 } “Temporary arrays for matrix addition”

13. param.ccl Uses, Extends: parameters from other thorns Scope: private, protected, public Data type: keyword, string, boolean, int, real, etc. Ranges and descriptions Default value Steerable What are the parameters for my thorn? What are their ranges? Are they steerable? What parameters do I need from other thorns? Which of my parameters should be available for other thorns?

14. Example: param.ccl Shares: wavetoy USES timestep EXTENDS keyword initial_data { “Cylindrical” :: “3D data with cylindrical symmetry” } Private: real wavespeed “Physical wave speed” { 0:1 :: “Only subsonic speeds allowed” } 0.1

15. Parameter Files !DESC “Example parameter file” ActiveThorns = “pugh pughreduce cartgrid3d wavetoyf77 ioutil ioascii” (only these thorns are used) #Parameters for my run driver::global_nx = 30 (restricted: implementation) wavetoy::initial_data = “gaussian” (restricted: implementation) io::out_every = 5 (restricted: implementation) ioascii::out1d_vars = “wavetoy::phi” (private: thorn)

16. Parameters at Runtime n Extensive runtime checking of parameters l demanded by users!! n Parameters must lie in designated ranges n Reports parameters which are not defined n Query parameters l cactus_exe -o l cactus_exe -O l cactus_exe -Ov n Used by parameter file GUIs, remote monitoring

17. Steerable Parameters n Parameters designated “steerable” in param.ccl can have their values changed during a simulation n Before designating a parameter as steerable, must think whether it really makes sense, and add the infrastructure for changing the parameter (for example, check it each timestep) n Examples: l Output: how often to output, what variables to output, output directory l Timestep ? n Steerable parameters can be changed using remote steering thorns, after restarting from a checkpoint file, or in the future from a command line interface

18. Program Flow Parameter File Thorn C Thorn B Thorn A Parameter Parser Thorn Activator Scheduler Driver Thorn Flesh

19. schedule.ccl Schedule at Schedule before or after other routines Schedule while some condition is true Schedule routine aliases for implementations Assign Storage for grid variables Triggers for analysis routines When and how should my thorns routines be run? How do my routines fit in with routines from other thorns? Which variables should be synchronized on exit?

20. Example: schedule.ccl STORAGE: mainvars, otherthorn::maxvals if (doit == 1) { schedule MyThorn_Init at INITIAL as InitGeneric before YourThorn_Init { LANG: C STORAGE: tempvars } “Initialization routine, sets up main variables ready for evolution” } while (keepgoing==1) { schedule MyThorn_EvolStep at EVOL as StepTwo after StepOne { LANG: Fortran SYNC: evolvemaxwell::mainvars } “Do a second evolution step if necessary” }

21. Scheduling Time Bins n Cactus provides a number of default time bins. Thorns can in addition define their own groups of routines for more control.

22. Analysis Time Bin n The time bin CCTK_ANALYSIS is special n Routines scheduled here are only called by the scheduler if output is required for one of their “triggers” n e.g. Constraint evaluation schedule ADMConstraints at CCTK_ANALYSIS { LANG: Fortran STORAGE: admconstraints TRIGGERS: admconstraints } “Calculate the constraints for the ADM equations”

23. Example Run List ----------------------------------------------------------- Startup routines CartGrid3D: Register GH Extension for GridSymmetry CartGrid3D: Register coordinates for the Cartesian grid PUGH: Startup routine. IOUtil: Startup routine IOASCII: Startup routine IOBasic: Startup routine WaveToyF77: Register banner Parameter checking routines CartGrid3D: Check coordinates IDScalarWave: Check parameters Initialisation CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH IOASCII: Choose 1D output lines IOASCII: Choose 2D output planes PUGH: Report on PUGH set up Time: Set timestep based on Courant condition WaveToyF77: Schedule symmetries IDScalarWave: Initial data for 3D wave equation do loop over timesteps WaveToyF77: Evolution of 3D wave equation WaveToyF77: Boundaries of 3D wave equation t = t+dt if (analysis) endif enddo Termination routines PUGH: Termination routine Shutdown routines -----------------------------------------------------------

24. Driver Thorn n The management of grid variables, including assigning storage, distribution among processors, and communication is handled by a driver thorn n Cactus is designed around a distributed memory model. Each thorn is passed a section of the global grid, and information about whether its boundaries are physical or just local internal grid boundaries. n The actual parallel driver can use whatever method it likes to decompose the grid across processors and exchange ghost zone information - each thorn is presented with a standard interface, independent of the driver. n Driver thorn can be chosen at run time … easy to compare different parallel paradigms.

25. Ghost Zones The grid on each processor has an extra layer of grid-points (marked here in blue) which are in fact copies of those on the neighbour in that direction (marked in red). After the calculation step the processors exchange these ghost zones to synchronise their states.

26. CactusPUGH/PUGH n The standard driver thorn is called PUGH (Parallel Unigrid Grid Hierachy), which uses MPI for communications. n PUGH can automatically choose the grid decomposition, or it can be manually specified, by the number of processors in each direction and the number of grid points on each processor. n PUGH can deal with all the supported Cactus variable types and data types, in 1,2 or 3 spatial dimensions. n CactusPUGH/PUGHReduce also contains a number of routines for global reductions: maximum, minimum and norms.

27. Cactus Driver API n Application thorns should not often need to make direct calls to the driver n CCTK_MyProc(cctkGH) l what processor am I? n CCTK_nProcs(cctkGH) l how many processors are there altogether? n CCTK_Barrier(cctkGH) l wait for all processors to catch up n ( CCTK_SyncGroup(cctkGH,“groupname”) ) l exchange ghostzones for a group of variables (better to specify in scheduler)

28. IO Capabilities n IO capabilities in Cactus are provided by thorns, which register the IO methods they provide with the flesh. n IO is usually selected through parameter file options, but can also be performed from direct calls using the Cactus IO API. l The scheduler calls the output routines n IO thorns also provide the capability for checkpointing l Save the state of the simulation (grid variables and parameters) for restarting, optionally on a different machine, different number of processors l Save initial data for parameter studies

29. IO Methods in Computational Toolkit

30. IO Parameters n Thorn CactusBase/IOUtil acts as a general utility thorn for all the Cactus IO methods, providing default parameters which can be overriden by the individual IO methods from other thorns. n IO from the Cactus thorns can be customised through parameters for a given architecture/number of processors. IO::outdir = “run1” IO::out_every = 5 IOASCII::out1d_vars = “wavetoy::phi grid::x” IOHDF5::out3d_every = 50 IOHDF5::out3d_vars = “wavetoy::phi”

31. Cactus IO API n If you need to output directly from a thorn... n CCTK_OutputGH(cctkGH) l Loop over all registered IO methods n CCTK_OutputVarAsByMethod(cctkGH,“varname”,“alias”, “method”) l Output varname using given IO method, using alias to create the filename n CCTK_OutputVarByMethod(cctkGH,“varname”,“method”) l Output varname using given IO method n CCTK_OutputVarAs(cctkGH,“varname”,“alias”) l Loop over all registered IO methods, outputting varname, using alias to create the filenames n CCTK_OutputVar(cctkGH, “varname”) l Loop over all registered IO methods, outputting varname

32. Cactus Interpolation/Reduction API n Thorns register interpolation, reduction operators with the Flesh n Thorns call Cactus API, indicating which operator they require (e.g. by parameter value) call CCTK_ReductionHandle(handle, “minimum”) call CCTK_Reduce(ierr,cctkGH,handle, …, )

33. n These are C pointers … they cannot be treated as Fortran strings n To compare use if (CCTK_EQUALS(bound,”gaussian”) then Note that CCTK_EQUALS returns a logical and CCTK_Equals returns an integer …the case is important!! Don’t use CCTK_Equals as a logical, or you’re code won’t be portable! String Parameters in Fortran

34. n Include files n Argument lists n Cactus variables n Parameters n Functions n Local variable data types Cactus Source Code

35. Include Files n For both Fortran and C l #include “cctk.h” –any source file using Cactus infrastructure l #include “cctk_Arguments.h” –any source file using Cactus argument lists l #include “cctk_Parameters.h” –any source file using Cactus parameters

36. n The scheduler passes in all the variables, through the argument list, that a thorn needs l Cactus variables describing the grid hierachy, such as the local grid size All the variables in the thorns interface.ccl All the variables needed from other thorns (inherited and friends) #include “cctk_Arguments.h” subroutine WaveToyF77_Evolution(CCTK_ARGUMENTS) implicit none DECLARE_CCTK_ARGUMENTS Argument Lists

37. n Pass argument list into other routines using: call WaveToyF77_Boundaries(CCTK_PASS_FTOF) … subroutine WaveToyF77_Boundaries(CCTK_ARGUMENTS) DECLARE_CCTK_ARGUMENTS n Can add extra arguments call WaveToyF77_Boundaries(CCTK_PASS_FTOF,myint) … subroutine WaveToyF77_Boundaries(CCTK_ARGUMENTS,myint) DECLARE_CCTK_ARGUMENTS integer myint Passing Argument Lists

38. All thorn routines called by the Scheduler, are passed the following variables, describing the grid layout: cctk_dim dimension of this grid cctk_lsh local grid size cctk_delta_time, cctk_time time step, current time cctk_delta_space grid spacing in each direction cctk_nghostzones number of ghostzones in each direction cctkGH pointer identifying grid n Plus a few others … Cactus Variables in Scheduled Routines

39. All parameters defined in a thorn’s param.ccl (bound), and all global parameters appear as local variables in a thorn n Do not change them!! l We don’t stop you! l (Because we can’t) #include “cctk_Parameters.h” subroutine MySubroutine(arg1,arg2) DECLARE_CCTK_PARAMETERS Cactus Parameters

40. n These are C pointers … they cannot be treated as Fortran strings n To compare use if (CCTK_EQUALS(bound,”gaussian”) then Note that CCTK_EQUALS returns a logical and CCTK_Equals returns an integer …the case is important!! Don’t use CCTK_Equals as a logical, or you’re code won’t be portable! String Parameters in Fortran

41. The function CCTK_WARN is used both to give warnings and to stop the code n Warnings are given a warning level. By default, warning level 0 will stop the code, and warning level 1 will be printed to standard error. n This can be changed at run time from the command line. CCTK_WARN(0,”Boundary conditions not applied - giving up!”) CCTK_WARN(1,”You probably don’t want to do this”) CCTK_WARN(4,”Probably harmless, but I’ll tell you”) Warnings and Errors

42. Use CCTK_INFO to write to standard output at runtime: CCTK_INFO(“Creating initial data”) n Use this rather than “print” statements l multiprocessor l can switch on and off Runtime Information

43. Infrastructure Thorn API n Cactus provides a richer API for infrastructure programmers to interact with the Flesh n Function and capability registration l any number of thorns can register any number of routines l e.g. IO, reduction, interpolation, elliptic solvers, coordinates n Function overloading l overloadable functions can only be provided by one routine l e.g. driver capabilities, CCTK_nProcs l e.g. main routines in Flesh are all overloadable n Many utility functions l linked lists, hash tables etc. l parameter querying

44. Getting the Code n Cactus is available for checkout via CVS (recommended) or from automatically generated tar files from the web site. n Full instructions at www.CactusCode.org n The easiest way to checkout Cactus is via a “Thorn List”, which allows you to checkout just the thorns you require for a given application (using GetCactus script from website). n Web site contains sample thorn lists to get you started. l Example WaveToy application l Benchmark application

45. Making Cactus Cactus Executable Configure Build CSTCompile Make

46. Configurations n Configurations are used to contain details of different compiled versions of Cactus l Shared file spaces used by different machines l Different thorn lists l Different compiler options l Repository of (not-)working versions

47. Configuring Determine Architecture Find Compilers Determine Flags Configure Fortran Names Known Architectures Command Line Options Options File config-data

48. Cactus Specification Tool (CST) Parse CCL Files Validate Output CST bindings ThornList interface.ccl param.ccl schedule.ccl Flesh interface.ccl param.ccl schedule.ccl Thorn A interface.ccl param.ccl schedule.ccl Thorn B

49. Compiling Dependencies Preprocess Files Compile Files Postprocess Files libthorn.a Thorn config-data + make.code.defn

50. Making Cactus n Configure: gmake -config gmake -config OPTIONS= Many options, e.g. gmake MyCactus-config gmake MyCactus-config MPI=NATIVE DEBUG=yes gmake MyCactus-config MPI=CUSTOM MPI_LIBS=mpi MPI_LIB_DIRS=/usr/lib MPI_INC_DIRS=/usr/include F90=pgf90 OPTIMISE=yes DEBUG=yes HDF5=yes THORNLIST=mythorns.th

51. Making Cactus (2) n Build gmake -build Many options, e.g. gmake MyCactus-build gmake MyCactus-build WARN=yes TJOBS=4 n Executable: exe/cactus_

52. n The make system uses source file extensions to indicate the coding language, these include: Source File Names

53. To include source files in the compilation process, add them to the make.code.defn file in the thorn src directory n Use the right file extensions!! n (Can completely configure … add your own Makefile etc) # Main make.code.defn file for thorn WaveToyF77 # Source files in this directory SRCS = InitSymBound.F77 WaveToy.F77 Startup.c # Subdirectories containing source files SUBDIRS = Compiling Thorn Source Files

54. Running Cactus n Cactus is executed from a parameter file n l e.g../exe/cactus_wave wavetoy.par l e.g. mpirun -np 4./exe/cactus_wave wavetoy.par -w 9 n Command line options for warning level, information querying etc. n Designed to provide information for GUIs, portals, etc.

55. Run Time Output ----------------------------------------------------------------------------- 10 1 0101 ************************ 01 1010 10 The Cactus Code V4.0 1010 1101 011 www.cactuscode.org 1001 100101 ************************ 00010101 100011 (c) Copyright The Authors 0100 GNU Licensed. No Warranty 0101 ----------------------------------------------------------------------------- Activating thorn Cactus...Success -> active implementation Cactus Activating thorn iobasic...Success -> active implementation IOBasic Activating thorn idscalarwave...Success -> active implementation idscalarwave Activating thorn time...Success -> active implementation time Activating thorn wavetoyf77...Success -> active implementation wavetoy Activating thorn pugh...Success -> active implementation driver Activating thorn cartgrid3d...Success -> active implementation grid Activating thorn ioutil...Success -> active implementation IO Activating thorn ioascii...Success -> active implementation IOASCII -----------------------------------------------------------------------------

56. Run Time Output (2) Startup routines CartGrid3D: Register GH Extension for GridSymmetry CartGrid3D: Register coordinates for the Cartesian grid PUGH: Startup routine. IOUtil: Startup routine IOASCII: Startup routine IOBasic: Startup routine WaveToyF77: Register banner Parameter checking routines CartGrid3D: Check coordinates for CartGrid3D IDScalarWave: Check parameters Initialisation CartGrid3D: Set up spatial 3D Cartesian coordinates on the GH IOASCII: Choose 1D output lines IOASCII: Choose 2D output planes PUGH: Report on PUGH set up Time: Set timestep based on Courant condition WaveToyF77: Schedule symmetries IDScalarWave: Initial data for 3D wave equation

57. Run Time Output (3) do loop over timesteps WaveToyF77: Evolution of 3D wave equation WaveToyF77: Boundaries of 3D wave equation t = t+dt if (analysis) endif enddo Termination routines PUGH: Termination routine Shutdown routines ----------------------------------------------------------------------------- Driver provided by PUGH ----------------------------------------------------------------------------- WaveToyF77: Evolutions of a Scalar Field ----------------------------------------------------------------------------- INFO (CartGrid3D): dx=>3.0000000e-01 dy=>3.0000000e-01 dz=>3.0000000e-01 INFO (CartGrid3D): x=>[-0.150, 8.550] y=>[-0.150, 8.550] z=>[-0.150, 8.550] INFO (PUGH): MPI Evolution on 4 processors INFO (PUGH): 3-dimensional grid functions INFO (PUGH): Size: 30 30 30 INFO (PUGH): Processor topology: 1 x 2 x 2 INFO (PUGH): Local load: 7680 [30 x 16 x 16] INFO (PUGH): Maximum load skew: 0.000000

58. Run Time Output (4) -------------------------------------------- it | | phi | | t | Min Max | -------------------------------------------- 0 | 0.000| 0.00000000 | 0.99142726 | 10 | 1.500| -0.14944313 | 0.00005206 | 20 | 3.000| -1.16992203 | 0.00000000 | 30 | 4.500| -0.72347868 | -0.00000002 | 40 | 6.000| -0.41127922 | -0.00000094 | 50 | 7.500| -0.29834817 | -0.00002427 | 60 | 9.000| -0.23577373 | -0.00035044 | 70 | 10.500| -0.19532474 | 0.00000311 | 80 | 12.000| -0.16688911 | 0.00010812 | 90 | 13.500| -0.14582895 | 0.00022987 | 100 | 15.000| -0.12957000 | 0.00033802 | 110 | 16.500| -0.11631732 | 0.00025752 | 120 | 18.000| -0.06945640 | 0.00019346 | ----------------------------------------------------------------------------- Done.

59. Command Line Arguments -O, -describe-all-parametersfull list of all parameters from all compiled thorns -o, -describe-parameter full description of a single parameter -T, -list-thornslist all compiled thorns -t, -test-thorn-compiled test if a thorn is included in executable -h, -?, -helpprovide help message -v, -versionversion information for this executable -W, -warning-level set the warning level, default value if 1 -E, -error-level set the warning level at which Cactus terminates, default value is 1 -r, -redirect-outputredirect standard output from each processor to a file

60. Command Line Arguments (2) >./exe/cactus_tut -T ---------------Compiled Thorns------------- Boundary Cactus CartGrid3D Hyperslab IDBinarySource IDScalarWave IDScalarWaveC IDScalarWaveCXX IOASCII IOBasic IOUtil PUGH Time WaveToyC WaveToyCXX WaveToyF77 WaveToyF90 WaveToyFreeF90 -------------------------------------------

61. Command Line Arguments (3) >./exe/cactus_tut -t PUGH Thorn 'PUGH' available. >./exe/cactus_tut -t PAGH Thorn 'PAGH' unavailable. >./exe/cactus_tut -v /home/allen/CACTUS/Cactus/exe/cactus_tut: Version 4.0.b8. Compiled on Jul 12 2000 at 09:53:07

62. Command Line Arguments (4) >./exe/cactus_tut -help /home/allen/CACTUS/Cactus/exe/cactus_tut, compiled on Jul 12 2000 at 09:53:07 Usage: /home/allen/CACTUS/Cactus/exe/cactus_tut [-h] [-O] [-o paramname] [-x [nprocs]] [-W n] [-E n] [-r] [-T] [-t name] [-v] Valid options: -h, -help : gets this help. -O, -describe-all-parameters : describes all the parameters. -o, -describe-parameter : describe the given parameter. -x, -test-parameters [nprocs] : does a quick test of the parameter file pretending to be on nprocs processors, or 1 if not given. -W, -warning-level : Sets the warning level to n. -E, -error-level : Sets the error level to n. -r, -redirect-stdout : Redirects standard output to files. -T, -list-thorns : Lists the compiled-in thorns. -t, -test-thorn-compiled : Tests for the presence of thorn. -v, -version : Prints the version.

63. Command Line Arguments (5) >./exe/cactus_exe -O... Parameters of thorn 'IDScalarWaveC' providing implementation 'idscalarwave': IDScalarWaveC::amplitude idscalarwave::initial_data IDScalarWaveC::kx IDScalarWaveC::ky IDScalarWaveC::kz IDScalarWaveC::radius IDScalarWaveC::sigma Parameters of thorn 'WaveToyC' providing implementation 'wavetoy': WaveToyC::bound Parameters of thorn 'WaveToyCXX' providing implementation 'wavetoy': WaveToyCXX::bound...

64. Command Line Arguments (6) >./exe/cactus_tut -Ov Parameter: IDScalarWaveC::sigma Description: "The sigma for the gaussian wave" Type: REAL Default: 0.1 Scope: PRIVATE Range: 0:* Origin: IDScalarWaveC Description: Parameters of thorn 'WaveToyC' providing implementation 'wavetoy': Parameter: WaveToyC::bound Description: "Type of boundary condition to use" Type: KEYWORD Default: none Scope: PRIVATE Range: radiation Origin: WaveToyC Description: Range: flat Origin: WaveToyC Description: Range: none Origin: WaveToyC Description:...

65. Command Line Arguments (7) >./exe/cactus_tut -o PUGH::global_nx Parameter: PUGH::global_nx, driver::global_nx Description: "The size of the grid in the x direction" Type: INTEGER Default: 10 Scope: RESTRICTED Range: 0:* Origin: PUGH Description:

66. Testing Cactus n Cactus provides a testsuite checking mechanism which can be used to compare output from different architectures, different numbers of processors, different code versions etc. against previously saved “correct” output n gmake -testsuite l runs all tests applicable for this configuration n It is recommended that testsuites are added for each thorn to test each aspect of its functionality, and are checked each time that changes are made to any part of the code.