1. Cactus Tools for the Grid
Gabrielle Allen
Max Planck Institute for Gravitational Physics,
(Albert Einstein Institute)

2. www.CactusCode.org Cactus CACTUS is a freely available, modular,
portable and manageable environment
for collaboratively developing parallel, high-
performance multi-dimensional simulations
THE GRID: Dependable, consistent, pervasive access to high-end resources

3. What is Cactus
Flesh (ANSI C) provides code infrastructure (parameter, variable, scheduling databases, error handling, APIs, make, parameter parsing, )
Thorns (F77/F90/C/C++/Java/Perl/Python) are plug-in and swappable modules or collections of subroutines providing both the computational instructructure and the physical application. Well-defined interface through 3 config files
Just about anything can be implemented as a thorn: Driver layer (MPI, PVM, SHMEM, …), Black Hole evolvers, elliptic solvers, reduction operators, interpolators, web servers, grid tools, IO, …
User driven: easy parallelism, no new paradigms, flexible
Collaborative: thorns borrow concepts from OOP, thorns can be shared, lots of collaborative tools
Computational Toolkit: existing thorns for (Parallel) IO, elliptic, MPI unigrid driver,
Integrate other common packages and tools: HDF5, Globus, PETSc, PAPI, Panda, FlexIO, GrACE, Autopilot, LCAVision, OpenDX, Amira, ...
Trivially Grid enabled!

4. Current Version Cactus 4.0
Cactus 4.0 beta 1 released September 1999
Community code: Distributed under GNU GPL
Currently: Cactus 4.0 beta 8
Supported Architectures:
SGI Origin
SGI 32/64
Cray T3E
Dec Alpha
Intel Linux IA32/IA64
Windows NT
HP Exemplar
IBM SP2
Sun Solaris
Hitachi SR8000-F
NEC SX-5
Mac Linux
...

5. Cactus Computational Toolkit: Parallel utilities (thorns) for computational scientist
CactusBase
Boundary, IOUtil, IOBasic, CartGrid3D, IOASCII, Time
CactusBench
BenchADM
CactusConnect
HTTPD, HTTPDExtra
CactusExample
WaveToy1DF77, WaveToy2DF77
CactusElliptic
EllBase, EllPETSc, EllSOR, EllTest
CactusPUGH
Interp, PUGH, PUGHSlab, PUGHReduce
CactusPUGHIO
IOFlexIO, IOHDF5, IsoSurfacer
CactusIO
IOJpeg
CactusTest
TestArrays, TestCoordinates, TestInclude1, TestInclude2, TestComplex, TestInterp, TestReduce
CactusWave
IDScalarWave, IDScalarWaveC, IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90
CactusExternal
FlexIO, jpeg6b
BetaThorns (In Development)
IOStreamedHDF5, Renderer, IOHDF5Util,…, many more

6. Computational Science
Cactus Community
DLR
Geophysics
(Bosl)
Numerical
Relativity Community
Cornell
Crack prop.
NASA NS GC
Livermore
SDSS
(Szalay)
Intel
Microsoft
Clemson
“Egrid”
NCSA, ANL, SDSC
AEI Cactus Group
(Allen)
NSF KDI
(Suen)
EU Network
(Seidel)
Applications
Astrophysics
(Zeus)
US Grid Forum
DFN Gigabit
Computational Science
“GRADS”
(Kennedy, Foster,
Dongarra, et al)
ChemEng
(Bishop)
San Diego, GMD,
Berkeley

7. Grid Computing
AEI Numerical Relativity Group has access to high-end resources in over ten centers in Europe/USA
They want:
Bigger simulations, more simulations and faster throughput
Intuitive IO at local workstation
No new systems/techniques to master!!
How to make best use of these resources?
Provide easier access … noone can remember ten usernames, passwords, batch systems, file systems, … great start!!!
Combine resources for larger productions runs (more resolution badly needed!)
Dynamic scenarios … automatically use what is available
Remote/collaborative visualization, steering, monitoring
Many other motivations for Grid computing ...

8. Grand Picture Viz of data from previous simulations in SF café
Remote Viz in St Louis
Remote steering and monitoring from airport
Remote Viz and steering from Berlin
DataGrid/DPSS
Downsampling
IsoSurfaces
http
HDF5
T3E: Garching
Origin: NCSA
Globus
Simulations launched from Cactus Portal
Grid enabled Cactus runs on distributed machines

9. IsoSurfaces and Geodesics Grid FunctionsStreaming HDF5
Remote Visualization
OpenDX
OpenDX
Amira
Contour plots
(download)
IsoSurfaces and Geodesics
LCA Vision
Grid FunctionsStreaming HDF5
Amira

10. Remote Steering Any Viz Client Amira XML HTTP Remote Viz data HDF5

11. Remote Offline Visualization
Viz Client (Amira)
HDF5 VFD
DataGrid (Globus)
DPSS
FTP
HTTP
Viz in Berlin
VisualizationClient
Downsampling, hyperslabs
Only what is needed
DPSS Server
FTP Server
Web Server
Remote Data Server
4TB at NCSA

12. Remote Monitoring: Thorn HTTPD
Thorn which allows simulation any to act as its own web server
Connect to simulation from any browser anywhere … collaborate
Monitor run: parameters, basic visualization, ...
Change steerable parameters
See running example at
Wireless remote viz, monitoring and steering

13. Remote Viz: OpenDX
Open source, (free), multiplatform, large active development community, easy to program
Reads HDF5 (Cactus) data from file or remotely streamed from Cactus
Simple GUI, select different hyperslabs from 3D data
Also support for streamed ASCII data from Cactus

14. Remote Viz: IsoView Lightweight, free, isosurface viewer
Distributed from Cactus website
Displays isosurfaces streamed from Cactus (thorn IsoSurfacer calculates isosurfaces inline with the parallel simulation)
Connect to running simulation on any remote machine
Steer isosurface value from client

15. Cactus Portal KDI ASC Project
Technology: Globus, GSI, Java Beans, DHTML, Java CoG, MyProxy, GPDK, TomCat, Stronghold
Locates resources, builds/finds executables, central management of parameter files/job output, submit jobs to local batch queues, tracks active jobs
Submission of distributed runs
Adding more collaborative features (eg group accounting, …)
Accesses the ASC Grid Testbed (SDSC, NCSA, Argonne, ZIB, LRZ, AEI)

16. New Grid Applications
Dynamic Staging: move to faster/cheaper/bigger machine
“Cactus Worm”
Multiple Universe
create clone to investigate steered parameter (“Cactus Virus”)
Automatic Convergence Testing
from intitial data or initiated during simulation
Look Ahead
spawn off and run coarser resolution to predict likely future
Spawn Independent/Asynchronous Tasks
send to cheaper machine, main simulation carries on
Thorn Profiling
best machine/queue
choose resolution parameters based on queue ….

17. New Grid Applications (2)
Dynamic Load Balancing
inhomogeneous loads
multiple grids
Portal
resource choosing
simulation launching
management
Intelligent Parameter Surveys
farm out to different machines
Make use of
Running with management tools such as Condor, Entropia, etc.
Scripting thorns (management, launching new jobs, etc)
Dynamic use of eg MDS for finding available resources

18. Dynamic Grid Computing
Add more
resources
SDSC
Queue time over,
find new machine
Free CPUs!!
RZG
SDSC
Clone job with
steered parameter
Calculate/Output
Invariants
LRZ
Archive data
Found a horizon,
try out excision
Calculate/Output
Grav. Waves
Look for
horizon
Find best
resources
Go!
NCSA

19. Users View

20. Cactus Grid Development
Projects: GrADs, KDI Astrophysics Collaboratory, TIKSL/GriKSL, EGrid/GGF, Globus team, AEI
Grid Application Development Toolkit: Application developer should be able to build simulations with tools that easily enable dynamic grid capabilities e.g. Query information server, Network Monitoring, Simulation instrumenation/Contract specification, Spawning and Migrating, Authentification Server, Data handling, Data transfer, Visualization, Steering, …
Remote visualization, steering, data management
User Portal
Dynamics Applications

21. More Information ... Cactus: Global Grid Forum (Egrid) ASC Portal
Web Site: (Documentation/Tutorials etc)
Cactus Worm:
Global Grid Forum (Egrid)
ASC Portal
TIKSL Gigabit Computing
Black Holes and Neutron Star: Pictures and Movies
jean-luc.aei.mpg.de
Any questions: