The ACTS Toolkit (What can it do for you?) Osni Marques and Tony Drummond ( LBNL/NERSC )

06/06/2000NUG/ORNL2 What is the ACTS Toolkit? Advanced Computational Testing and Simulation Tools for development of parallel applications 21 tools developed (primarily) at DOE labs originally conceived as autonomous tools ACTS is an “umbrella” project collect tools leverage numerous independently funded projects

06/06/2000NUG/ORNL3 ACTS Project Goals Bring software packages together into a “toolkit” Make the software interoperable Provide consistent application interfaces Promote general solutions to parallel programming needs Promote code reusability Enable large scale applications Impact DOE science

06/06/2000NUG/ORNL4 Recent Successful Cases Scattering in a quantum system of three charged particles (Rescigno, Baertschy, Isaacs and McCurdy, Dec. 24, 1999). Cosmic Microwave Background Analysis, BOOMERanG collaboration, MADCAP code (Apr. 27, 2000).

06/06/2000NUG/ORNL5 NERSC Activities Make ACTS tools available on NERSC platforms Provide technical support Perform independent evaluation of tools Maintain online ACTS information center Identify new users who can benefit from toolkit Work with users to integrate tools into applications

06/06/2000NUG/ORNL6 ACTS Information Online Information center coordinated and integrated with developers resources general information and pointers to documentation User-oriented information Which tool should I use to solve my problem? Which tools are ready for prime time? Reviews: strengths, weaknesses, caveats, etc

06/06/2000NUG/ORNL7 ACTS Support Support at different levels applications code optimization tool selection tool utilization tool installation Leverage with developers Minimize risk to users

06/06/2000NUG/ORNL8 Tools Categorization Numerical  software that implements numerical algorithms Structural (“frameworks”)  software that manages data, communication Infra-structural  runtime, support tools, developer’s bag

06/06/2000NUG/ORNL9 Numerical Tools Aztec: iterative methods for solving sparse linear systems Hypre: collection of advanced preconditioners Opt++: solution of nonlinear optimization problems PETSc: methods for the solution of PDE related problems PVODE: solvers for large systems of ODE’s ScaLAPACK: dense linear algebra computations SuperLU: direct methods for sparse linear systems

06/06/2000NUG/ORNL10 Structural (Frameworks) Global Arrays: portable, distributed array library, shared memory style of programming Overture: library of grid functions which derives from P++ arrays POET (Parallel Object-oriented Environment and Toolkit): allows for “mixing and matching” of components POOMA (Parallel Object-Oriented Methods and Applications): C++ abstraction layer between algorithm and platform (similar to HPF)

06/06/2000NUG/ORNL11 Infra-structural CUMULVS (Collaborative User Migration User Library for Visualization and Steering), PAWS (Parallel Application WorkSpace): computational steering, data post-processing, interactive visualization Globus: infrastructure for high performance distributed computing (computational grids) SILOON (Scripting Interface Languages for Object-Oriented Numerics): scripting features TAU (Tuning and Analysis Utilities): advanced performance analysis and tuning

06/06/2000NUG/ORNL12 Infra-structural (cont.) Tulip: C++ applications with threads, global pointers and other parallel operations ATLAS (Automatically Tuned Linear Algebra Software), PHiPAC (Portable High Performance ANSI C) : automatic generation of optimized numerical software (mainly BLAS) Nexus: multithreading, communication and resource management facilities PADRE (Parallel Asynchronous Data and Routing Engine) : abstracts the details of representing and managing distributed data PETE (Portable Expression Template Engine) : efficient C++ operator overloading through expression templates

06/06/2000NUG/ORNL13 Tool Status at NERSC Installed and supported (modules,T3E and SP) Aztec PETSc ScaLAPACK SuperLU TAU Known to have been used (T3E) CUMULVS Global Arrays Globus POOMA PVODE

06/06/2000NUG/ORNL14 Aztec Solves large sparse linear systems on distributed memory machines Implements Krylov iterative methods (CG, CGS, Bi- CG-Stab, GMRES, TFQMR) Suite of preconditioners (Jacobi, Gauss-Seidel, overlapping domain decomposition with sparse LU, ILU, BILU within domains) Highly efficient, scalable (1000 processors on the “ASCI Red” machine)

06/06/2000NUG/ORNL15 Aztec (applications) TOUGH2 (Transport Of Unsaturated Groundwater and Heat) code, transport simu- lation in porous and fractured media (LBNL). Co-flowing Annular Jet Combuster, a parallel 3D pseudo-transient simulation to steady state operation ; MPSalsa code (SNL).

06/06/2000NUG/ORNL16 Aztec (basic steps) Prepare your linear system distribute the matrix call AZ_transform set up right-hand side and initial guess call AZ_reorder_vec on initial guess and right-hand side selective an iterative solver and a preconditioner call AZ_solve call AZ_invorder_vec on solution

06/06/2000NUG/ORNL17 PETSc Linear and nonlinear solvers Data structures for distributed vectors and sparse matrices 5 matrix storage formats It has been integrated with Aztec 10+ Krylov iterative subspace methods Preconditioners include Jacobi, additive overlapping Schwartz, parallel ILU(0), etc Modular, object design methodology C and Fortran interfaces Can be used as a front end to other ACTS tools

06/06/2000NUG/ORNL18 PETSc (applications) Multiphase flow, 4 million cell blocks, 32 million DOF, over 10.6 Gflops on an IBM SP (128 nodes), entire simulation runs in less than 30 minutes (Pope, Gropp, Morgan, Seperhrnoori, Smith and Wheeler). Prometheus code (unstructured meshes in solid mechanics), 26 million DOF, 640 nodes on NERSC’s Cray T3E (Adams and Demmel).

06/06/2000NUG/ORNL19 ScalaPACK Parallel version of LAPACK Based on BLACS (Basic Linear Algebra Communication Subroutines) Dense linear algebra (BLAS) Direct solution of linear systems Dense matrix eigensolvers

06/06/2000NUG/ORNL20 ScaLAPACK (applications) Induced current (white arrows) and charge density (colored plane and gray surface) in crystallized glycine due to an external field (Louie, Yoon, Pfrommer and Canning). Cosmic Microwave Background Analysis, BOOMERanG collaboration, MADCAP code (Apr. 27, 2000).

06/06/2000NUG/ORNL21 SuperLU Direct solution of large sparse linear systems Shared and distributed memory implementations Attained 8.3 Gflops on 512 nodes of the T3E

06/06/2000NUG/ORNL22 SuperLU (applications) Scattering in a quantum system of three charged particles (Rescigno, Baertschy, Isaacs and McCurdy, Dec. 24, 1999). SuperLU speedup (matrices dimension varying from to ).

06/06/2000NUG/ORNL23 TAU Profiling of Fortran 90, C, C++, HPF, and HPC++ codes Detailed information (much more than prof/gprof) C++: per-class and per-instance profiling Graphical display of profiling results (built-in viewers, interface to Vampir)

06/06/2000NUG/ORNL24 TAU (Main Control Window) COSY: COmpile manager Status displaY FANCY: File ANd Class displaY CAGEY: CAll Graph Extended displaY CLASSY: CLASS hierarchY browser RACY: Routine and data ACcess profile displaY SPEEDY: Speedup and Parallel Execution Extrapolation DisplaY

06/06/2000NUG/ORNL25 TAU (SPEEDY)

06/06/2000NUG/ORNL26 Coming up Hypre PVODE POOMA Globus PETE (summer student) Survey of iterative linear solver packages

06/06/2000NUG/ORNL27 PVODE and POOMA PVODE is being used in the ParFlow ground- water flow project to solve three-dimensional models of fluid flow and chemical transport through heterogeneous porous media (LNNL). MC++ Monte Carlo neutron transport simulation written using the Tecolote framework, which is built atop POOMA, (ASCI Blanca project, LANL).

06/06/2000NUG/ORNL28 Globus Provides means for creating a large scale computational grid A toolkit of core services with which applications can be developed to access the grid. Operates with other ACTS tools (like CUMULVS, PAWS and Nexus)

06/06/2000NUG/ORNL29 Globus (cont.) NASA’s Information Power Grid (IPG) joins supercomputers and storage devices owned by participating organizations into a single, seamless computing environment. With a Globus-enabled X-ray microto- mography program a fully 3-D reconstruction of an ant head (2mm in diameter) was obtained in less than 10 minutes using the acquisition hardware at a beamline at ANL and an SGI Origin 2000 at NPACI.

06/06/2000NUG/ORNL30 Future Directions CCA (Common Component Architecture) Developing standardized ways of managing numerical components to allow mixing-and-matching Frameworks for gluing components together Similarities to CORBA, DCOM, Java Beans Scientific interface description language (allowing Fortran) ESI (Equation Solver Interface) Developing standardized interfaces for scalable linear solvers Specific test case for CCA component design

acts-