Cactus in GrADS (HFA) Ian Foster Dave Angulo, Matei Ripeanu, Michael Russell

Presentation Outline  Introduction to Cactus  Cactus Applications  Cactus Architecture  Cactus Worm Thorn  Tequila Thorn (Cactus in GrADs)  Tequila Architectur  Issues

What is Cactus? Cactus is a freely available, modular, portable and manageable environment for collaboratively developing parallel, high-performance multidimensional simulations

Example Cactus Output Example output from Numerical Relativity Simulations

Cactus Applications  Application Thorns are Astrophysics applications  Calculate Schwartzchild Event Horizons for colliding black holes

Cactus Applications (cont.)  Candidate apps are Elliptical Solver or BenchADM  Abstract Topologies are simple 3D Grid

Cactus Applications (cont.)  Applications can easily be “linked” in with the other thorns used as tools.  Application Thorns are just selected and run with the other selected thorns

Cactus Architecture Configure CST Flesh Computational Toolkit Operating Systems AIX NT Linux Unicos Solaris HP-UX Thorns Cactus SuperUX Irix OSF Make

Cactus Model (cont.) Building an executable Cactus Source Flesh IOBasic IOASCII WaveToy LDAP Worm … Thorns Configuration Compiler options Tool options MPI options HDF5 options

Running Cactus Parameter File Specify which thorns to activate Specify global parameters Specify restricted parameters Specify private parameters

Cactus model  This is the currently working Cactus application framework that we will modify Cactus “flesh” internals Cactus Application Thorn(s) Other Cactus Thorn(s) Cactus “Worm” Thorn

Worm Thorn Functions  Initiates moving to new resource when scheduled time is exhausted  Contacts IS to get a new node to run on  Checkpoints application  Restarts application on the new node  Runs on single node only

GrADS Cactus Model  We will start with “Worm” thorn code to make new “Tequila” thorn (Apotheosized Cactus Worm). Cactus “flesh” internals Cactus Application Thorn(s) Other Cactus Thorn(s) Cactus “Tequila” Thorn

Tequila thorn functions  Receives event (generated by user) to initiate adapting resources.  Contacts ResourceSelector to get new bag of resources  Checkpoints application  Restarts application on the new resources

Events  Events that cause the user to want to adapt resources: –User changes parameters during runtime that requires additional resources Example: starting an analysis routine Example: running an event horizon finder –User specifies that performance is not meeting expectations

Future Events  Possible Future Plans for automatic resource adapting: –User changes parameters during runtime that requires additional resources –Contract violations fire similar events we were wrong first time resources get overloaded more (or fewer) (or different) processors appear distribution changed resolution changed

Tequila thorn contacts ResourceSelector  ResourceSelector must be set up as service  Tequila thorn sends request for new bag of resources  ResourceSelector responds with the new bag

Request and Response  The Request to the ResourceSelector will be stored in the InformationService  Only the pointer to the data in the IS will be passed to the ResourceSelector  The Response from the ResourceSelector will also be stored in the IS  Only the pointer to the data in the IS will be passed back.

Tequila communication overview Cactus Tequila Thorn Resource Selector Information Service

Cactus Architecture in GrADS Configure CST Flesh Computational Toolkit Operating Systems AIX NT Linux Unicos Solaris HP-UX Thorns Cactus SuperUX Irix OSF Make Toolkit Grads Communi- cation library

Open Issues  How does Contract Monitor fit into architecture?  How does PPS fit into architecture?  How does COP and Aplication Launcher fit into architecture (Cactus has its own launcher and compiles its own code)?  How does Pablo fit into architecture (Which thorns are monitored, is flesh monitored)?

End of Presentation

Slides Explaining Communication Details  ********************

Communication Details step 1  Event sent to Tequila thorn requesting restart Cactus Tqeuila Thorn Resource Selector Information Service

Communication Details step 2  Tequila store AART in IS Cactus Tqeuila Thorn Resource Selector Information Service

Communication Details step 3  Tequila sends request to ResourceSelector passing pointer to data in IS Cactus Tqeuila Thorn Resource Selector Information Service

Communication Details step 4  ResourceSelector retrieves AART from IS Cactus Tqeuila Thorn Resource Selector Information Service

Communication Details step 5  ResourceSelector stores bag of resources (in AART) in IS Cactus Tqeuila Thorn Resource Selector Information Service

Communication Details step 6  ResourceSelector responds to Tequila passing pointer to data in IS Cactus Tqeuila Thorn Resource Selector Information Service

Communication Details step 7  Tequila retrieves AART with new bag of resources from IS Cactus Tqeuila Thorn Resource Selector Information Service

Requirements  Using the IS for communication adds overhead.  Why do this?  GrADS requirement 1: do some things (e.g. compile) at one time and have the results stored in a persistent storage area. Pick these stored results up later and complete other phases.

Requirements (cont.)  GrADS requirement 2: Application people want to be able to allow users to manually interact in any of the "module interfaces." Tequila allows this to be done with a web client.

Slide Explaining Parallelism in Cactus  ***************

Parallelism in Cactus  Cactus is designed around a distributed memory model. Each thorn is passed a section of the global grid.  The actual parallel driver (implemented in a thorn) can use whatever method it likes to decompose the grid across processors and exchange ghost zone information - each thorn is presented with a standard interface, independent of the driver.  Standard driver distributed with Cactus (PUGH) is for a parallel unigrid and uses MPI for the communication layer  PUGH can do custom processor decomposition and static load balancing

Slide with Alternate Tequila Architecture  ***************

Sample Tequila Scenario  User asks to run an ADM simulation 400x400x400 for 1000 timesteps in 10s.  Resource selector contacted to obtain virtual machines  Best virtual machine selected based on performance model.  AM starts Cactus on that virtual machine (and monitors execution Contracts?)  User (or application manager) decides that computation advances too slow and decides to search for a better virtual machine  AM finds a better machine, commands the Cactus run to Checkpoint, transfers files and restart Cactus

Slides Explaining Different Tequila Architectures  *********************

Tequila Architecture Choices  Main presentation explained the short term Tequila Architecture  Open issues covered not-yet-resolved architectural choices for longer term integration

Worm Spawning Cactus Flesh Worm Thorn GIIS 1. Resources Obtained Cactus Flesh Worm Thorn Application Manager 2. Application Manager instructed to spawn new instance 3. New instance spawned

Tequila Spawning  The short-term plan is to simply replace the GIIS with the UCSD Resource Selector  Tequila would make the request for new resources to the RS instead of the GIIS

Tequila Spawning 1. Resources Obtained Cactus Flesh Tequila Thorn UCSD Resource Selector Cactus Flesh Tequila Thorn Application Manager 2. Application Manager instructed to spawn new instance 3. New instance spawned

Tequila Spawning  Longer term plan is not yet resolved.  One possibility is to put all grads pieces into Application Manager

Application Manager Cactus Flesh Tequila Thorn UCSD Resource Selector Cactus Flesh Tequila Thorn Application Manager 1. Application Manager instructed to spawn new instance 3. New instance spawned 2. Resources Obtained