1. Grads Meeting - San Diego Feb 2000 The Cactus Code Gabrielle Allen Albert Einstein Institute Max Planck Institute for Gravitational Physics allen@cactuscode.org

2. 2 Cactus Code Versions 1,2,3 n A code for Numerical Relativity l collaborative, portable, parallel,... n Model of “Flesh” + “Thorns” l Flesh: core code, provides parallelisation, IO. l Thorns: plug in modules, written in Fortran or C, which provide the application code. n Successful for numerical relativity, but problems as number of thorns and number of users increased n Redesign, incorporating lessons learnt from previous versions l Cactus 4.0

3. 3 Current Version Cactus 4.0 n Cactus 4.0 beta 1 released September 1999 n Flesh and many thorns distributed under GNU GPL n Wednesday: Cactus 4.0 beta 6 n Supported Architectures: l SGI Origin l SGI 32/64 l Cray T3E (142GF on 1024 nodes) l Dec Alpha l Intel Linux l Windows NT l (HP Exemplar) l (IBM SP2) l (Sun Solaris)

4. 4 Cactus 4 Design Goals n Generalization l meta-code that can be applied to any system of PDEs –mainly 3D cartesian finite differencing codes, but not only... n Abstraction l Identify key concepts that can be abstracted –Evolution skeleton. Reduction operators. I/O. Etc... n Encapsulation l Protect the developers of thorns from other thorns... n Extension l Prepare for new concepts in future thorns l Overloading, Inheritance, etc... n In some way, make it a little Object Oriented

5. 5 Design issues n Modular and Object Oriented l Keep the concept of thorns l Encapsulation, Polymorphism, Inheritance,... n Fortran l influences most design issues n Portable Parallelism n Support for FMR and AMR as well as Unigrid n Powerful Make system n Tools such as “Testsuite checking technology”

6. 6 Realization n Perl l The final code is created from Thorn configuration files by perl scripts that are some sort of seed for a new language: –The Cactus Configuration Language (CCL): variables, (functions), parameters, scheduling l Perl scripts also take care of testsuite checking n Flesh written in ANSI C n Thorns written in C, C++, Fortran77, Fortran90

7. 7 Cactus Flesh : interface between Application Thorns and Computational Infrastructure Thorns

8. 8 The Flesh l Abstract API –evolve the same PDE with unigrid, AMR (MPI or shared memory, etc) without having to change any of the application code. l Interfaces –set of data structures that a thorn exports to the world (global), to its friends (protected) and to nobody (private) and how these are inherited. l Implementations –Different thorns may implement e.g. the evolution of the same PDE and we select the one we want at runtime. l Scheduling –call in a certain order the routines of every thorn and how to handle their interdependencies. l Parameters –many types of parameters and all of their essential consistency checked before running

9. 9 Cactus Computational Toolkit n Parallel Evolution Drivers l PUGH –MPI domain decomposition based unigrid driver –Distributed using globus l GrACE/PAGH –Adaptive Mesh Refinement driver n Elliptic Solvers l PETSc l BAM n Interpolators n I/O l FlexIO, ASCII, HDF5, Panda, etc... n Visualization, etc...

10. 10 Data Structures n Grid Arrays l An multidimensional and arbitrarily sized array distributed among processors n Grid Functions l A field distributed on the multidimensional computational grid (a Grid Array sized to the grid) –Every point in a grid may hold a different value “f(x,y,z)” n Grid Scalars l Values common to all the grid points n Parameters l Values/Keywords that affect the behavior of the code (initialization, evolution, output, etc..) –parameter checking, steerable parameters

11. 11 Data Types n Cactus data types to provide portability across platforms n CCTK_REAL l CCTK_REAL4, CCTK_REAL8, CCTK_REAL16 n CCTK_INT l CCTK_INT2, CCTK_INT4, CCTK_INT8 n CCTK_CHAR n CCTK_COMPLEX l CCTK_COMPLEX8, CCTK_COMPLEX16, CCTK_COMPLEX32

12. 12 Scheduling n Thorns schedule when their routines should be executed n Basic evolution skeleton idea l standard scheduling points INITIAL, EVOL, ANALYSIS l fine control: run this routine BEFORE/AFTER that routine n Extend/customise with scheduling groups l Add my routine to this group of routines l Run the group WHILE some condition is met n Future redesign l The scheduler is really a runtime selector of the computation flow. l We can add much more power to this concept

13. 13 Interface n The concept: contract with the rest of the code l Now it is only for the data structures : variables and parameters l adding routines and arguments n Private l The variables that you want the flesh to allocate/communicate but no other thorn to see. n Public l The variables that you want everybody to see (that means that everybody can modify them too!) l Inheritance n Protected l Variables that you want only your friends to see! l [Watch out for the change of meaning from C++ names]

14. 14 Implementation n Why l Two or more thorns that provide the same functionality but different internal implementation –Interchangeable pieces that allow easy comparison and evolution in the development process –They are compiled together and only one is activated at runtime n How l If all the other thorns need to see the same contract, then thorns implementing a certain functionality must –Have the same public variables –and their protected ones!! –The same concept applies to parameters and scheduling n Example l Wildly different evolution approaches for the same equations, so all the analysis and initial data thorns remain the same.

15. 15 Parallelism in Cactus n Cactus is designed around a distributed memory model. Each thorn is passed a section of the global grid. n The actual parallel driver (implemented in a thorn) can use whatever method it likes to decompose the grid across processors and exchange ghost zone information - each thorn is presented with a standard interface, independent of the driver. driver:nghostzones = 1

16. 16 PUGH n The standard parallel driver supplied with Cactus is supplied by thorn PUGH n Driver thorn: Sets up grid variables, handles processor decomposition, deals with processor communications n 1,2,3D Grid Arrays and Grid Functions (beta6) n Uses MPI n Custom processor decomposition n Otherwise decomposes in z, then y, then x directions

17. 17 Parallizing an Application Thorn n CCTK_SyncGroup –synchronise ghostzones for a group of grid variables –just added Synchronization to Scheduler configuration file as well n CCTK_Reduce –call any registered reduction operator, e.g. maximum value over the grid n CCTK_Interpolate –call any registered interpolation operator n CCTK_MyProc –unique processor number within the computation n CCTK_nProcs –total number of processors n CCTK_Barrier –waits for all processors to reach this point

18. 18 Building an executable n Compiling Cactus involves two stages l creating a configuration l compiling the source files to an executable n Configuration: l Cactus can be compiled with different compilers, different compilation options, with different lists of thorns, with different external libraries (e.g. MPICH or LAM), and on different architectures. l To facilitate this Cactus uses configurations, which store all the distinct information used to build a particular executable (Cactus/configs) l Each configuration is given a unique name.

19. 19 Configuration Options gmake MyConfig-config (or options file) n Default options decided by autoconf n Compiler and tool specification e.g. l F77=/weirdplace/pgf90 n Compilation and tool flags e.g. l CFLAGS=save-temps l DEBUG=ALL n Library and include file specification n Precision options e.g. l REAL_PRECISION=16

20. 20 Configuring with External Packages n Cactus currently knows about the external packages: l MPI (NATIVE, MPICH, LAM, WMPI,CUSTOM) l HDF5 l GRACE and will search standard locations for them l gmake MyConfig MPI=NATIVE l gmake MyConfig MPI=MPICH MPICH_DEVICE=globus GLOBUS_LIB_DIR=/usr/local/globus/lib l gmake MyConfig MPI=CUSTOM MPI_LIBS=mpi MPI_LIB_DIRS=/usr/lib MPI_INC_DIRS=/usr/include

21. 21 Compile Options gmake MyConfig n Parallel build l FJOBS= l TJOBS= n Compilation debugging l SILENT=no n [Compiler warnings] l WARN=yes

22. 22 Running Cactus./exe/cactus_MyConfig MyParameterFile.par Additional command line options -hhelp -O[v]details about all parameters -o details about one parameter -vversion number, compile date -Tlist all thorns -t is thorn compiled -rredirect stdout -W reset warning level -E reset error level

23. 23 Parameter Files n Cactus runs from a user’s parameter file l chooses the thorns to be used for the run (so that inactive thorns can’t do any damage) l sets parameters which are different from default values !desc = “Demonstrates my new application” ActiveThorns = “PUGH WaveToyF77 Boundary CartGrid3D” driver::global_size = 30# Change the grid size wavetoy:: amplitude = 1.56# Initial wave amplitude

24. 24 Coming up (Cactus 4.2) n Cactus communication layer l Parallel driver thorn (e.g. PUGH) currently provides both variable management and communication … l abstract send and receives etc n Abstract communication from driver thorn l easily implement different parallel paradigms l shared memory, threads, Corba, OpenMP, PVM,... n Compact groups (different layout in memory for improved Cache performance) n Unstructured Meshes/Finite Elements/Spectral Methods n Unstructured Multigrid Solver n Convergence/Multiple Coordinate Patches n Capability browsing mechanism n Command line interface … connect directly to Cactus, scheduling n GUIs, Documentation, GUIs, Documentation ….

25. 25 www.CactusCode.org n Documentation l IEEE Computer December 1999 l Users Guide l Maintainers Guide n Download l CVS distribution (stable and development versions) n Development l Bugs and feature requests l Mailing lists (e.g. cactususers@cactuscode.org, flesh@cactuscode.org ) n Showcase l Presentations, publications, movies... n News, and Links to related institutions, software