2. Cactus Framework I Overview, Design principles and Architecture The Cactus Team Albert Einstein Institute cactus@cactuscode.org

3. Plan for today … n Why I’m talking here n Programming frameworks and environments n Software requirements from computational physicists n Introduction to Cactus n Cactus Code design goals and architecture n Configuration files n Drivers and parallellization n Cactus user and developer communities

4. and the next days … n Writing code using Cactus framework and toolkits n Simple examples: Hello World, Scalar Wave Equation n I/O and Visualization n Numerical Relativity Toolkit, what is there and how to use it n Support and help in the community

5. Why I’m talking here … n Scientific computing is now a key part of just about every area of physics (computer algebra, numerical relativity, astrophysics, cosmology, QCD, gravitational wave analysis…) n Existing frameworks and toolkits such as Cactus make it much easier to get into the different fields and to collaborate with the different communities, and allow scientists to concentrate on physics instead of computer science. n Everything (except the supercomputers) which is described here is freely available and open source n Just about everything Ed is talking about you can download and try for yourself on any (Linux) laptop or PC.

6. What is Cactus? CACTUS is a freely available, modular, portable and manageable environment for collaboratively developing parallel, high- performance multi-dimensional simulations

7. Programming Frameworks

8. Why use a Framework? n Separate computational physics from infrastructure headaches: l Make system, Parallellization, I/O, Interpolators, Elliptic Solvers n Collaboration, modular frameworks enable sharing code n Community building n Project continuity and portability n Code reuse n Leverage of other peoples developments: grid, viz, etc as well as physics

9. Deciding to use a Framework n Does it support the languages/machines you want to use? n Is it already used in your community? n Is it supported and developed? n Will it be supported and developed next year? n Is there documentation? n Try it out with a simple problem … find out if it can handle the most complicated thing you want to do. n Is it open source? n How does it interact with other packages and tools (visualization, adaptive mesh refinement, linear algebra packages).

10. Requirements for Computational Physics

11. What Computational Physicists Need From Their Software … n Collaborative n Portable n Large scale ! n High throughput n Easy to understand and interpret results n Supported and developed n Produce believed results n Flexible n Reproducible n Have generic computational toolkits n Incorporate other packages/technologies n Easy to use/program Primarily, it should enable the physics they want to do, and that means it must be:

12. Numerical Relativity n Basic 3+1 approach n Initial data for 3-metric & extrinsic curvature (black holes, waves, …) n Choose lapse and shift n Evolution method for 3- metric and extrinsic curvature n Analysis routines n Output data n Visualize and interpret results

13. Collaborative n Many different sub-parts of complex physics problems, e.g. NR: initial data, evolution, horizons, waves, elliptic solvers, AMR, excision, shift conditions n Need a “modular” framework which connects (geographically distributed) experts in each of these areas n Just one code version !! n Also: l Need not know about the whole code l Short startup times for new people l Code reuse after people leave Community: Share common code and experiences, validate results, compare methods. Example ThornList ADM_BSSN (Miguel) FishEye (John B) Zorro (Manuela/Carlos) AHFinder (Miguel) PUGH (Tom) FlexIO (John S) IOFlexIO (Thomas R) BAM_Elliptic (Bernd) PsiKadelia (Steve) Time (Gabrielle) EllBase (Gerd) Einstein (All)

14. Large Scale n Typical BH run (but we want bigger!) needs 45GB of memory: l 171 Grid Functions l 400x400x200 grid n Typical run makes 3000 iterations with 6000 Flops per grid point: l 600 TeraFlops !! n Output of just one Grid Function at just one time step l 256 MB l (320 GB for 10GF every 50 time steps) n One simulation takes longer than queue times l Need 10-50 hours n Computing time is a valuable resource l One simulation: 2500 to 12500 SUs l Need to make each simulation count Parallelism Parallel/Fast IO, Data Management, Visualization Optimization Requirements Interactive monitoring, steering, visualization, portals Checkpointing

15. Produce Believable Results n Continually test with known/validated solutions l Code changes l Using new thorns l Different machines l Different numbers of processors n Open community: l The more people using your code, the better tested it will be l Open Source … not black boxes l Source code validates physical results which anyone can reproduce n Diverse applications: l Modular structure helps construct generic thorns for Black Holes, Neutron Stars, Waves, … l Other applications, …

16. Generic Toolkits Computational Toolkit: Drivers (Unigrid/FMR/AMR) Coordinates Boundary conditions I/O Methods Interpolation operators Reduction operators Elliptic solvers Web server Steering Notification Example applications Einstein Toolkit: Evolution (ADM) Maximal slicing Analysis Wave extraction Scalar invariants Apparent horizons Constraints Initial data Black holes Gravitational waves Provide common functionality: quicker to develop applications, well tested, written by experts in an area. Continually added to and improved

17. Portability n Develop and run on many different architectures (laptop, workstations, supercomputers) n Set up and get going quickly (new computer, visits, new job, wherever you get SUs) n Use/Buy the most economical resource (e.g. our new supercomputer) n Make immediate use of free resources n Tools and infrastructure also important n Portability crucial for “Grid Computing” RECENT GROUP RESOURCES Origin 2000 (NCSA) Linux Cluster (NCSA) Compaq Alpha (PSC) Linux Cluster (AHPCC) Origin 2000 (AEI) Hitachi SR-8000 (LRZ) Cray T3E (Garching) Institute Workstations Linux Laptops Ed’s Mac Very different architectures, operating systems, compilers and MPI implementations

18. Easy to Use and Program n Program in favorite language (C,C++,F90,F77) n Hidden parallelism n Computational Toolkits n Good error, warning, info reporting n Modularity !! Transparent interfaces with other modules n Extensive parameter checking n Work in the same way on different machines n Interface with favorite visualization package n Documentation

19. Requirements: Developers n And for computer scientists supplying infrastructure … l support development and incorporation of new technologies l allow different programming languages and paradigms l provide rich low level APIs and services l have fixed and extensible interfaces l allow modules to take control of or replace all key default functionality (“main” routine, scheduling tree, …) l make new infrastructure modules immediately available to user communities

20. Introduction to Cactus

21. Cactus History n Cactus originated in 1997 as a code for numerical relativity, following a long line of codes developed at the NCSA and then the AEI. n Numerical Relativity: complicated 3D hyperbolic/elliptic PDEs, dozens of equations, thousands of terms, many people from very different disciplines working together, needing a fast, portable, flexible, easy-to-use, code which can incorporate new technologies without disrupting users. n Cactus 1,2,3: Paul Walker, Joan Masso, John Shalf, Ed Seidel. n Cactus 4.0 beta 1, August 1999: Total rewrite and redesign of code, learning from experiences with previous versions (Tom Goodale)

22. Cactus in a Nutshell n Cactus acts a the “main” routine of your code, it takes care of e.g. parallelism, IO, checkpointing, parameter file parsing for you (if you want), and provides different computational infrastructure such as reduction operators, interpolators, coordinates, elliptic solvers, … n Everything Cactus “does” is contained in thorns (modules), which you need to compile-in. If you need to use interpolation, you need to find and add a thorn which does interpolation. n It is very extensible, you can add you own interpolators, IO methods etc. n Not all the computational infrastructure you need is necessarily there, but hopefully all of the APIs etc are there to allow you to add anything which is missing. n We’re trying to provide a easy-to-use environment for collaborative, high-performance computing, from easy compilation on any machine, to easy visualization of your output data.

23. Cactus Today Cactus is a framework designed by/for numerical relativity and astrophysics n Cactus 4.0 Beta 12 Released last Friday l Reorganised modules for Numerical Relativity !! n Current core team: Gabrielle Allen, David Rideout, Tom Goodale, Thomas Radke, Kashif Rasul, John Shalf, Ian Kelley

24. Cactus Core “Flesh” Plug-In “Thorns” (modules) driver input/output interpolation SOR solver coordinates boundaryconditions boundary conditions black holes equations of state remote steering wave evolvers multigrid parameters gridvariables grid variables errorhandling error handling scheduling extensibleAPIs extensible APIs makesystem make system ANSI C Fortran/C/C++ Your Physics !! Your Computational Tools !!

25. Modularity: “Plug-and-play” Executables Zorro AHFinder PsiKadelia Extract BAM_Elliptic IDLinearWaves IDAnalyticBH ADM_BSSN ConfHyp MaximalADM SimpleExcision FishEye IDBrillWaves Cartoon2D HLL PUGHPAGH Carpet CartGrid3D TimeBoundary IOASCII IOUtilIOBasic IOFlexIO IOHDF5IOJpeg EllBase HTTPDExtraHTTPD ADMConstraint EllSOR LegoExcision ISCO Run PUGH CartGrid3D Time EllBase IOASCII IOHDF5 IOUtil HTTPD IOBasic ADMConstraint Zorro AHFinder Boundary ADM_BSSN FishEye BAM_Elliptic ISCO with Excision ISCO with AMR ?? PUGHPAGH SimpleExcision Numerical Relativity ThornsComputational Thorns IDAxiBrillBH TGRPETSc Faster elliptic solver ?? TGRPETSc BAM_Elliptic

26. Cactus Computational Toolkit CactusBase l Boundary, IOUtil, IOBasic, CartGrid3D, LocalInterp, IOASCII, Time CactusBench l BenchADM CactusConnect l HTTPD, HTTPDExtra, Socket CactusExamples l HelloWorld, FleshInfo, DemoInterp, TimerInfo, SampleIO, WaveToy1DF77, WaveToy2DF77 CactusElliptic l EllBase, EllPETSc, EllSOR CactusPUGH l PUGH, PUGHInterp, PUGHReduce, PUGHSlab CactusPUGHIO l IOFlexIO, IOHDF5, IOHDF5Util, IOPanda, IOStreamedHDF5, IsoSurfacer CactusTest l TestArrays, TestCoordinates, TestInclude1, TestInclude2, TestComplex, TestInterp, TestReduce CactusWave l IDScalarWave, IDScalarWaveC, IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90 CactusExternal l FlexIO,jpeg6b

27. Einstein Toolkit: CactusEinstein Infrastructure: ADMBase, StaticConformal, SpaceMask, ADMCoupling, ADMMacros, CoordGauge Initial Data: IDSimple, IDAnalyticBH, IDAxiBrillBH, IDBrillData, IDLinearWaves Evolution: ADM, EvolSimple, Maximal Analysis: ADMConstraints, ADMAnalysis, Extract, AHFinder, TimeGeodesic, PsiKadelia, IOAHFinderHDF Other thorns available from other groups/individuals E.g. a few from AEI … Excision LegoExcision, SimpleExcision AEIThorns ADM_BSSN, BAM_Elliptic, BAM_VecLap PerturbedBH DistortedBHIVP, IDAxiOddBrillBH, RotatingDBHIVP

28. Supported Architectures

29. Design Goals and Architecture

30. Abstract Design Goals n Generalization l Meta-code that can be applied to any numerical system n Abstraction l Identify key concepts that can be abstracted n Encapsulation l Modularize, protect thorn developers from each other n Extension l Prepare for future concepts and technologies

31. Generalization Meta-code for any numerical system Finite differences Finite elements N-body Monte-Carlo Cartesian Unstructured grids Regular/irregular grids Polar Adaptive mesh refinement Hyperbolic Parabolic Elliptic Multi-patch Finite volumes Ray tracing

32. Abstraction Abstract Key Concepts Evolution skeleton Grid variables Reduction operators Interpolation operators Parallel operators Checkpointing Input/output Elliptic solvers Coordinates Data types Convergence

33. Flesh API n Abstract Flesh API for l Driver functions (storage, communication) l Interpolation l Reduction l IO, Checkpointing l Coordinates l etc, etc n In general, thorns “overload” or “register” their capabilities with the Flesh, agreeing to provide a function with the correct interface n e.g. CCTK_Interp(…,”second order interpolator”) n e.g. CCTK_OutputVar(“variable”,“IOASCII”) n e.g. CCTK_CoordRange(lower,upper,“x”,”cart3d”)

34. Encapsulation Protection for thorn developers CVS Versioning Parameter ranges Parameter checking Implementations Private parameters ActiveThorns Inheritance Private grid variables Thorns and Thorn Arrangements

35. Encapsulation n Thorns l A thorn usually implements a single specific task n Object Orientated Features l Grid Variables, Functions –private, protected, public l Parameters –private, restricted, global l Implements –what does this thorn do? Provides relationships between thorns. l Inherits –get all public variables, functions from an implementation and its ancestors

36. Extension Be ready for new concepts GUIs and Portals Adaptive meshes Simulation steering New hardware architectures Scripting language Java/Perl thorns Interactive scheduling Unstructured grids Communication layer Capability browsing New things with haven’t been named yet

37. Extension n Thorns provide all the real functionality, to add something new just write a new thorn n Everything in the flesh can be overloaded and/or extended by thorns l Scheduling mechanism l Main evolution routine l Make system

38. Cactus Architecture Configure CST Flesh Computational Toolkit Operating Systems AIX NT Linux Unicos Solaris HP-UX Thorns SuperUX Irix OSF Make

39. Thorn Architecture Make Information Source Code Documentation! Configuration Files Parameter Files and Testsuites ???? Fortran Routines C++ Routines C Routines Thorn

40. Thorn Collections n For organizational convenience, thorns are grouped into arrangements l may have related functionality (e.g. IO or Maxwell solvers) l may have the same author l may contain everything needed for one problem n We call a collection of arrangements, a toolkit e.g. l Cactus Computational Toolkit l Cactus Relativity Toolkit CactusBaseCactusPUGH CactusPUGHIOCactusElliptic +... Cactus Computational Toolkit

41. Application View Flesh CCTK_(…)CST Application Toolkit Computational Toolkit

42. Portability n Flesh written in ANSI C n Cactus data types: l CCTK_REAL, CCTK_INT, CCTK_COMPLEX, … l Used for Grid Variables and Parameters n GNU Autoconf n Known Architectures files n Thorns call Cactus API rather than system/software dependent packages l CCTK_Abort =>MPI_Abort(or something else) n Can checkpoint on one machine and restart on another n Can run distributed jobs across a T3E and Windows

43. Compilation n Autoconf based configuration process for different architectures, compilers and libraries n CST (Cactus Specification Tool): Perl based parsing of files written in Cactus Configuration Language (CCL) generates thorn wrappers, argument lists (Fortran!), parameters, variables, scheduling, etc. n Cactus creates different configurations from the same source code l different architectures on common file system l use different compiler options, debug-modes, communication protocols l different thorn lists

44. Testing n Test Suite Checking Technology l Thorns can contain parameter files and their output l Running Cactus “test suite” program re-executes these parameter files checking against the original output. l RemoteTest l Advantages –Checks development work on your own thorns –Protects against changes in other thorns –Compares e.g multiprocesser runs against single processor runs, or the use of new thorns –Compares different architectures n Warning Levels l CCTK_Warn l Warning level can be chosen at run time

45. Drivers and Parallelization

46. Driver Modules/Thorns n Driver thorns: l Decompose computational grid/tasks across processors l Set up grid variables (grid functions, grid arrays, grid scalars) l Handle storage l Provide communication/synchronization of grid variables n Associated thorns provide hyperslabbing, interpolation, reduction … anything requiring communication across processors n Different drivers needed for different methodologies: l Finite differencing l Unstructured grids l Adaptive mesh refinement l Particle methods l Spectral methods l SPH

47. Parallelisation n Consider a finite difference scheme with a stencil width 1 n The standard way to use multiple processors with Cactus is to decompose the computational grid between processors n This is actually done by a driver thorn … n The default driver thorn is called PUGH which uses MPI for communication between processors Proc 0

48. Parallelisation n PUGH splits the grid in the optimal way between different processors, e.g. in this example two processors. n If the grid is simply split between two processors there is no longer enough data to update the blue point Proc 0 Proc 1

49. Parallelisation n To overcome this ghostzones are used. n Ghostzones are additional (duplicated) grid points which are added to each processor containing the required data to advance to the next iteration Proc 0 Proc 1

50. Parallelisation n The data on each grid must be periodically synchronised n The ghostzones are updated with current data Proc 0 Proc 1

51. How is this done? (or why you might want to use a framework) Use MPI (e.g. thorn PUGH)PUGH n On each processor n Fill up buffers with ghostzone data n Figure out which other processors to exchange data with n Send them your data n Wait for the data they are sending n Optimize for optimal message sizes, blocking or nonblocking sends and receives etc. Use a framework like Cactus n Anytime you want to synchronise a group of variables … CALL CCTK_SyncGroup(“group”)

52. Parallelization n Make use of different communication protocols l MPI, PVM, SHMEM, pthreads, OpenMP, Corba, RDMA, … l portability across platforms l choose best option for a given architecture l compare and different protocols n Thorns don’t call communication functions directly, but through the Cactus API l CCTK_SyncGroup, CCTK_Barrier, CCTK_Reduce, … l Thorns overload or register functions to provide these capabilities l e.g. thorn CactusPUGH/PUGH (default parallel MPI driver) Overloads CCTK_Barrier with MPI_Barrier

53. Parallel IO n In this example just want to output fields from 2 processors, but it could be 2000 n Each processor could write it’s own data to disk IO::outmode = “proc” n Then the data usually is moved to one place and “recombined” to produce a single coherent file Proc 0 Proc 1

54. Parallel IO n Alternatively processor 0 can gather data from the other processors and write it all to disk IO::out_mode = onefile n Usually a combination of these works best … let every nth processor gather data and write to disk IO::out_mode = np IO::out_proc_every = 8 Proc 0 Proc 1

55. Interpolation n For example, processor 0 needs to know the value of a field at x=5, y=7 … could be on any processor CCTK_Interp(value, x, y, field, “interpolator”) Proc 0 Proc 1

56. Configuration Files

57. Configuration files n Each thorn provides 3 configuration files, detailing its interface with the Flesh and with other thorns n CCL: Cactus Configuration Language n interface.ccl l implementation, this thorn’s variables and variables used from other thorns, functions provided or used from other thorns n param.ccl l this thorn’s parameters, parameters used and extended from other thorns n schedule.ccl l when and how this thorn’s routines should be executed, optionally with respect to routines from other thorns

58. Grid Variables n Essentially these are just variables which your thorn tells the Cactus infrastructure about, to get parallelisation, IO, interpolation, communication, checkpointing etc. n Public, restricted or private, depending on who should see them. n Can be arbitrarily sized grid arrays, or grid functions fixed to the size of the computational domain, or grid scalars. n Many other available features: any dimension, distribution type etc.

59. Scheduling n The Cactus Flesh contains a flexible rule based scheduler which controls the program flow. n The scheduler calls routines from thorns, the order in which the routines are called is prescribed in a thorn configuration file. n (The main calling routines in the Flesh are also overloadable, providing a flexible and configurable mechanism for customising program flow)

60. Implements n Each thorn declares what it “does” or what it implements l interpolate l maxwell_initial_data l extract_multipoles n Defines relationships between thorns n Thorns providing same implementation must have common public and protected variables n Can compile many different thorns all providing same implementation l pick which one to use in the parameter file at run time!

61. Parameters n Range checking and validation l combats old problem of setting parameters to values which didn’t exist: evolution_method = “superstable fast 10th order shock capturing” l thorn writers must now specify ranges and descriptions for all parameters l checked at run time n Steerable l new implementation of steerable/changeable parameters for remote steering l must define steerable … only if it makes sense to do so

62. User and Developer Communities

63. Cactus User Community Using and Developing Physics Thorns GoddardPenn State Wash UAEI TACTuebingen Southampton SISSA Thessaloniki Climate Modeling (NASA,+) Chemical Engineering (U.Kansas) Bio-Informatics (Canada) Geophysics (Stanford) Astrophysics (Zeus) Crack Prop. (Cornell) EU Astrophysics Network NASA Neutron Star Grand Challenge Early Universe (LBL) Numerical Relativity Other Applications Portsmouth RIKEN Plasma Physics (Princeton) Pittsburg

64. Numerical Relativity Community n Recent workshop in Mexico comparing different evolution systems for Einstein’s Equations. l Hyperbolic, Conformal, Standard ADM, … n Everyone used the same initial data sets and the same analysis routines. n Concentrate on just one aspect of the total problem. n Comparisons much easier, reproducible, absolute. n Easy for even individuals to take part. n Very successful, first in a series, work continuing via CVS and mail lists. n Web pages, mailing list etc being set up specifically for numerical relativity community.

65. Cactus Developer Community Developing Computational Infrastructure Argonne National Laboratory EGrid Wash U AEI Cactus Group Lawrence Berkeley Laboratory Konrad-Zuse Zentrum NCSA Clemson Global Grid Forum SGI Compaq Intel Microsoft U. Chicago U. Kansas The Users Grants and Projects DFN TiKSL DFN GriKSL EU GridLab NSF KDI ASC NSF GrADS Direct Benefits Visualization Parallel I/O Remote Computing Portal Optimization Experts TAC Sun

66. What is the Grid? … … infrastructure enabling the integrated, collaborative use of high-end computers, networks, databases, and scientific instruments owned and managed by multiple organizations … … applications often involve large amounts of data and/or computing, secure resource sharing across organizational boundaries, not easily handled by today’s Internet and Web infrastructures …

67. Remote Monitoring/Steering: n Thorn which allows simulation to any to act as its own web server n Connect to simulation from any browser anywhere … collaborate n Monitor run: parameters, basic visualization,... n Change steerable parameters n Running example at www.CactusCode.org www.CactusCode.org n Wireless remote viz, monitoring and steering

68. Cactus ASC Portal www.ascportal.org n Part of NSF KDI Project n Thin Client !! Any Web Browser n Portal (will) provides: l Single access to all resources l Locate/build executables l Central/collaborative parameter files, thorn lists etc l Job submission/tracking l Access to new Grid Technologies Astrophysics Simulation Collaboratory

69. Physicist has new idea ! S1S1 S2S2 P1P1 P2P2 S1S1 S2S2 P2P2 P1P1 S Brill Wave Dynamic Grid Computing Found a horizon, try out excision Look for horizon Calculate/Output Grav. Waves Calculate/Output Invariants Find best resources Free CPUs!! NCSA SDSC RZG LRZ Archive data SDSC Add more resources Clone job with steered parameter Queue time over, find new machine Archive to LIGO public database