EU Network Meeting June 2001 Cactus Gabrielle Allen, Tom Goodale Max Planck Institute for Gravitational Physics, (Albert Einstein Institute)

Outline n Brief what is Cactus? n Review of developments since last meeting l Current release Beta 10 l New Viz Tools, thorns, architectures l Current work n General Relativity thorns l Released thorns in CactusEinstein l Other relevant thorns l Matter thorns from the WashU Relativity Group l Matter in Cactus: current status and plans n Rest of the meeting l Talk with me and Tom about what you want to use Cactus for l Talk with me if you want to work through some examples, make sure your laptop has everything installed right, get some advice on how to start, etc n Any time left … demos of new viz/control stuff

Cactus in a Nutshell n Cactus acts a the “main” routine of your code, it takes care of e.g. parallelism, IO, checkpointing, parameter file parsing for you (if you want), and provides different computational infrastructure such as reduction operators, interpolators, coordinates, elliptic solvers, … n Everything Cactus “does” is contained in thorns (modules), which you need to compile-in. If you need to use interpolation, you need to add in a thorn which does interpolation. n It is very extensible, you can add you own interpolators, IO methods etc. n Not all the computational infrastructure you need will be there, for example we don’t have a thorn for 2D coordinates yet, you may need to add your own (then hopefully make it available for others to use). n We’re trying to provide a easy-to-use environment for collaborative, high-performance computing, from easy compilation on any machine, to easy visualization of your output data.

Porting Your Code If your existing code has this kind of structure n Split into subroutines n Clear argument lists it should be straightforward to put into Cactus. Cactus will take care of the parameter file, and (hopefully) the coord system and IO, and if you’re lucky you can take someone elses Analysis, Evolution, Initial Data, … modules. program YourCode call ReadParameterFile call SetUpCoordSystem call SetUpInitialData call OutputInitialData do it=1,niterations call EvolveMyData call AnalyseData call OutputData end do end

Thorn EvolveMyData n Probably you will make a thorn to do the evolution. This thorn will simply contain the source code of your existing evolution routines (with hopefully not to many changes). n You will need to add configuration files which detail how your thorn interacts with the Flesh and other thorns. Make Information Source Code Documentation! Configuration Files Parameter Files and Test Suites ???? Fortran Routines C++ Routines C Routines Thorn EvolveMyData

Black Hole Run Extract Exact PsiKadelia ADM AHFinder ADMConstraints IDAnalyticBH PUGHInterp PUGHReduce PUG H IOASCII CartGrid3D IOBasi c IOUtil PhysicsComputational Einstein MyThorn FLESH

Parameter File ActiveThorns = "Time CartGrid3D PUGH pughslab pughreduce Einstein IDAnalyticBH ADM IOASCII IOUtil iobasic" cactus::cctk_itlast = 160 grid::type = "byspacing" grid::dx = 0.15 grid::dy = 0.15 grid::dz = 0.15 grid::domain = "octant" driver::global_nx = 20; driver::global_ny = 20; driver::global_nz = 20; einstein::initial_lapse = "one" einstein::evolution_system = "ADM" einstein::initial_data = "schwarzschild" einstein::slicing = "1+log" adm::bound = "radiative" adm::method = "stagleap" IO::outdir = "einstein1" IOASCII::out1D_vars = "einstein::grr einstein::alp" IOASCII::out1D_every = 5 IOBasic::outInfo_vars = "einstein::grr" IOBasic::outInfo_every = 20

How To Find Out About Thorns n Look at web pages l l AEI and WashU thorns ?? n Ask a colleague for a sample parameter file l New checkout script to make it easy to check out n Parameters? l Look in param.ccl in any thorn l Use command line options from executable e.g../exe/cactus_ -O n Thorn documentation: l gmake -ThornGuide l gmake ThornGuide n No documentation? Complain to authors.

Current Release: Beta 10 n New thorns: l CactusUtils/NaNChecker l CactusPUGH/PUGHInterp (replaces CactusPUGH/Interp) n Flesh: l Changes to time levels, interpolation API, parameters (more error checking for parameter file) n Thorns: l Boundary conditions, initialize memory to NaN/Zero/uninitialized n Tools: l Recombiner for HDF5 data n New architectures: l MacOS X (no Fortran or MPI) n Thorn documentation l gmake -ThornGuide

Planned Development n Easier scripts for checking out and compiling l GetCactus works now from a ThornList l Adding GetCactus from a parameter file, and also automatically getting configuration file for a know machine n Einstein: l Improve spacetime – matter interface so that any thorns can be used together for spacetime, matter evolution l Add timelevels (gxx, gxx_p, gxx_p_p), should improve performance, and be ready for AMR l Easier viz, set up networks for OpenDX, Amira n Release Cactus 4.0 !! l Mainly documentation and odd and ends n Elliptic Solvers: l Get what is there (SOR, PETSc, ) fully working and better documented using the elliptic interfaces n Coordinates: standard 2D, 1D, spherical n Cool stuff: web interface, /SMS thorn, lots more … n Ask for what you need, what is difficult:

From Other Projects n AMR l Good progress with FMR and AMR now n Portal l Web interface from anywhere to all your machines, parameter files, collaborative and other awesome stuff n Grid Computing l For us this really means not needing to know anything at all about the machines you have access to, codes automatically seek out and exploit your resources n Visualization l Remote visualization very important for big simulations l Also trying to make it easier, scripts to automatically find the right viz clients and networks for data (VizLauncher), networks specifically for GR n Data description and management l Again important for big simulations being run on many different machines l Archiving data also really important, understanding and being able to reproduce old data sets

Users View

Computational Toolkit CactusBase l Boundary, IOUtil, IOBasic, CartGrid3D, IOASCII, Time CactusBench l BenchADM CactusConnect l HTTPD, HTTPDExtra CactusExample l WaveToy1DF77, WaveToy2DF77 CactusElliptic l EllBase, EllPETSc, EllSOR, EllTest CactusPUGH l PUGHInterp, PUGH, PUGHSlab, PUGHReduce CactusPUGHIO l IOFlexIO, IOHDF5, IsoSurfacer CactusIO l IOJpeg CactusTest l TestArrays, TestCoordinates, TestInclude1, TestInclude2, TestComplex, TestInterp, TestReduce CactusWave l IDScalarWave, IDScalarWaveC, IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90 CactusExternal l FlexIO, jpeg6b CactusUtils l NaNChecker

Einstein Toolkit n Thorn CactusEinstein/Einstein l Sets up standard GR variables for 3D problems: –gxx, gxy, gxz, gyy, gyz, gzz, –kxx,kxy,kxz,kyy, kyz, kzz, –alp, betax, betay, betaz l Other tools –Mixed slicing methods (us maximal every 10 timesteps or if ????) –Courant timestep condition –Calculate spherical components –Generic mask function l You need to use these variables if you want to use any of the other Einstein thorns or thorns from AEI/WashU

Einstein Toolkit n Spacetime evolution: l ADM: staggered leapfrog, iterative Crank-Nicholson n Analysis: l AHFinder: locates apparent horizons using either minimization or flow algorithm. l ADMConstraints: violation of constraint equations. l PsiKadelia: calculate gauge invariant quantities l Extract: waveforms from nearly axisymmetric, nearly linear regions n Initial Data: l IDAnalyticBH, IDBrillData,IDLinearWaves,IDAxiBrillBH

Other Thorns n Links for more information on these will be added to the EU web pages n AEI: l Web Page? l ADM_BSSN: BS evolution with different shift conditions l Lazarus: NP quantities, perturbative evolution l Excision: masks out region inside a black hole n AMR: l Carpet l PAGH n Zeus: l Newtonian MHD n Alpha/BetaThorns l Maximal, Cartoon2D

Now Tom … n Tom will talk about what’s available at WashU, how Matter is currently treated with the CactusEinstein thorns, and how it is going to be improved so that we can use different spacetime and matter evolvers together.

More Information... n Look at news on Cactus home page n Also lots more information on Cactus pages: l Worked example of WaveToy showing how to use web server thorn, viz clients n Mailing lists: news, users, developers n CVS updates of the thorns you are using n cactusmaint.cactuscode.org (or me, Tom, Ed,..)

Go! Dynamic Grid Computing Clone job with steered parameter Queue time over, find new machine Add more resources Found a horizon, try out excision Look for horizon Calculate/Output Grav. Waves Calculate/Output Invariants Find best resources Free CPUs!! NCSA SDSC RZG SDSC LRZArchive data