Metacomputing Within the Cactus Framework What and why is Cactus? What has Cactus got to do with Globus? Gabrielle Allen, Thomas Radke, Ed Seidel. Albert-Einstein-Institut MPI-Gravitationsphysik Gabrielle Allen, Thomas Radke, Ed Seidel. Albert-Einstein-Institut MPI-Gravitationsphysik

Cactus 4.0

Why Cactus? Numerical relativity has very high computational requirements. Not every group has the resources or desire to develop a 3D code. (Especially IO, elliptic solvers, AMR) Previous experiences show that even a few people using one code is problematic. Need a structure that is maintainable and collaborative Scientists like to program in Fortran Want the ability to make new computational advances instantly and transparently available, without users modifying code

What Is Cactus? Cactus was developed as a general, computational framework for solving PDEs (originally in numerical relativity and astrophysics) Modular … for easy development, maintenance and collaborations. Users supply “thorns” which plug-into compact core “flesh” Configurable … thorns register parameter, variable and scheduling information with “runtime function registry” (RFR). Object-orientated inspired features Scientist friendly … thorns written in F77, F90, C or C++ Accessible parallelism … driver layer (thorn) is hidden from physics thorns by a fixed flesh interface

What Is Cactus? Standard interfaces … interpolation, reduction, IO, coordinates. Actual routines supplied by thorns Portable … Cray T3E, Origin, NT/Win9*, Linux, O2, Dec Alpha, Exemplar, SP2 Free … distributed under the GNU GPL. Uses as much free software as possible Up-to-date … new computational developments and/or thorns immediately available to users (optimisations, AMR, Globus, IO) Collaborative … thorn structure makes it possible for large number of people to use and development toolkits New version … Cactus beta-4.0 released 30th August

Cactus 4.0 Credits Cactus flesh and design –Gabrielle allen –Tom goodale –Joan massó –Paul walker Computational toolkit –Flesh authors –Gerd lanferman –Thomas radke –John shalf Development toolkit –Bernd bruegmann –Manish parashar –Many others Relativity and astrophysics –Flesh authors –Miguel alcubierre – Toni Arbona – Carles Bona – Steve Brandt – Bernd Bruegmann – Thomas Dramlitsch – Ed Evans – Carsten Gundlach – Gerd Lanferman – Lars Nerger – Mark Miller – Hisaaki Shinkai – Ryoji Takahashi – Malcolm Tobias Vision and Motivation – Bernard Schutz – Ed Seidel "the Evangelist" – Wai-Mo Suen

Full GR Neutron Star Collision With Cactus

Thorn Arrangements

Cactus 4.0 IOFlexIO FLESH (Parameters, Variables, Scheduling) IOHDF5 PUGH WaveToyF90 CartGrid3D GrACE Boundary WaveToyF77

DAGH/AMR (UTexas) AEI NASA NCSA Valencia ZIB Panda I/O (UIUC CS) Globus (Foster) Petsc (Argonne) SGI Cactus: Many Developers Wash. U HDF5 FlexIO

What Has It Got to Do With Globus? Easy access to available resources Access to more resources –Einstein equations require extreme memory, speed –Largest supercomputers too small! –Networks very fast! DFN gigabit testbed: 622 mbits potsdam-berlin-garching, connect multiple supercomputers Gigabit networking to US possible Connect workstations to make supercomputer Acquire resources dynamically during simulation! Interactive visualization and steering from anywhere Metacomputing experiments in progress with Cactus+Globus

TIKSL Tele Immersion: Collision of Black Holes German research project aimed to exploit the newly installed gigabit testbed Süd+Berlin Project partners Albert-Einstein-Institut Potsdam Konrad-Zuse-Institut Berlin Rechenzentrum Garching Main project goals Distributed simulations of black hole collisions with Cactus Remote visualization and application steering with Amira AEI

Running Cactus in a Distributed Environment Using the Globus services to Locate computing resources via MDS Authenticate the cactus users (GSS) Transfer necessary files to remote sites (executable, parameter files) via GASS Start the Cactus job via GRAM Do parallel communication and file I/O using Nexus MPI and MPI-IO extensions Access output data via GASS

Computational Needs for 3D Numerical Relativity Explicit finite difference codes –~ 10 4 flops/zone/time step –~ 100 3D arrays Require zones or more –~1000 gbytes –Double resolution: 8x memory, 16x flops Tflop, tbyte machine required Parallel AMR, I/O essential Etc InitialData: 4 coupled nonlinear elliptics Time step update explicit hyperbolic update also solve elliptics t=0 t=100

(A Single) Such Large Scale Computation Requires Incredible Mix of Varied Technologies and Expertise! Many scientific/engineering components –Formulation of ee’s, “gauge conditions”, equation of state, astrophysics, hydrodynamics, etc Many numerical algorithm components –Finite differences? Finite elements? Structured meshes? –Hyperbolic equations: implicit vs implicit, shock treatments, dozens of methods (and presently nothing is fully satisfactory!) –Elliptic equations: multigrid, krylov subspace, spectral, preconditioners (elliptics currently require most of the time…) –Mesh refinement? Many different computational components –Parallelism (HPF, MPI, PVM, ???) –Architecture efficiency (MPP, DSM, vector, NOW, ???) –I/O bottlenecks (generate gigabytes per simulation, checkpointing…) –Visualization of all that comes out!

Distributing Spacetime: SC’97 Intercontinental Metacomputing at Aei/Argonne/Garching/NCSA Immersadesk 512 Node T3E

Metacomputing the Einstein Equations: Connecting T3e’s in Berlin, Garching, San Diego

The Dream: not far away... Physics Module 1 Garching T3E Globus Resource Manager Cactus/Einstein solver MPI, MG, AMR, DAGH, Viz, I/O,... NCSA Origin 2000 array Mass storage BH Initial Data Budding Einstein in Berlin... Ultra 3000: Whatever-Wherever