1. Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu acts-support@nersc.gov

2. http://acts.nersc.gov/ What is the ACTS Toolkit? Advanced Computational Testing and Simulation Part of the DOE 2000 Project Will likely shift to base DOE funding Tools (roughly 20) for developing parallel applications Developed (primarily) at DOE Labs Separate projects originally ACTS is an “umbrella” project Collecting tools Leveraging numerous independently funded projects

3. http://acts.nersc.gov/ ACTS Project Goals Bringing tools together into a “Toolkit” Making the tools interoperable Providing consistent application interfaces Promoting general (not application specific) solutions to parallel programming needs Encouraging code re-use Impacting DOE science ASCI, IT2(SSP) Enabling large scale applications example follows: Oil Reservoir Simulation

4. http://acts.nersc.gov/ Prometheus Multigrid solver Unstructured meshes in solid mechanics Modified matrix graphs 26 million DOF, 640 PEs Use PETSc http://www.cs.berkeley.edu/~madams/prometheus

5. http://acts.nersc.gov/ NERSC Activities Make ACTS tools available on NERSC platforms Provide technical support Find new ER users who can benefit from the tools Work with users to integrate tools into applications Evaluate the tools Create ACTS information center (online)

6. ACTS Information Online One-stop Shopping But coordinated and integrated with developer resources. User-oriented Information Which tool should I use to solve my problem? Which tools are ready for prime time? Reviews: strengths, weaknesses, caveats, etc. http://acts.nersc.gov/

7. ACTS Support at NERSC Support for application development Technical support Tool installation (by request) Leverage with developers Support is also available through traditional NERSC channels (e.g. consult@nersc.gov). acts-support@nersc.gov

8. http://acts.nersc.gov/ Risk ACTS Tools are still research projects Can’t guarantee indefinite support Can’t guarantee fixed interface But commercial software doesn’t either A NERSC goal is to minimize risk to users Provide unbiased advice Promote tools that work Support tools with the best chance of surviving Be an advocate for users

9. http://acts.nersc.gov/ Tool Catagorization Numerical Libraries/data structures that implement numerical algorithms I.e. your “classic” numerical packages PETSc, Aztec, Hypre, ScaLAPACK, SuperLU, PVODE, Opt++ Structural (Frameworks) Libraries/data structures that manage data, communicate Global Arrays, Overture, POOMA, Cumulvs, PAWS, InDEPS Infrastructural TAU (performance analysis) Globus (meta-computing) PADRE, Nexus, Tulip (middleware -- mostly used by other tools)

10. http://acts.nersc.gov/ Tool Status at NERSC Installed on the T3E and fully supported by NERSC: Aztec PETSc ScaLAPACK TAU Known to have been successfully used on NERSC’s T3E: CUMULVS Global Arrays Globus POOMA PVODE

11. Tools available on T3E

12. http://acts.nersc.gov/ ScaLAPACK Installed on the T3E Parallel version of LAPACK Well known, trusted Basic dense linear algebra (BLAS) Direct solution of linear systems / Factorization General, tri-diagonal, banded Dense matrix eigensolvers Sustained 605 GFLOPS on ASCI Red in materials simulation Get started: “man scalapack” or “module help scalapack”

13. http://acts.nersc.gov/ PETSc Installed on the T3E Functionality for solving PDEs in parallel The most widely used and well supported member of the ACTS toolkit Runs on serial and parallel machines (even NT) C and Fortran Programming Interfaces Modular, objected design methodology Methodology extensible to User code Get started: “module help petsc” or “man petsc”

14. http://acts.nersc.gov/ Aztec Installed on T3E Solving sparse linear systems on distributed machines Highly efficient and scalable Some Apps have successfully scaled to 1000s of processors Krylov iterative methods: CG, CGS, Bi-CG-Stab, GMRES, TFQMR Large preconditioning suite including Jacobi, Gauss-Seidel, overlapping domain decomps (ILU et al) Get started: “module help aztec” or “man aztec”

15. http://acts.nersc.gov/ Aztec User input matrix in DMSR or DVBR format Aztec sets up its own data structure for solving linear systems Used by groups of dedicated users Salinas MPSalsa COYOTE II TOUGH... http://endo.sandia.gov/BB/comp_str_dyn/mp-fema.html

16. http://acts.nersc.gov/ Using Aztec: basic steps Prepare your linear system distribute the matrix call AZ_transform set up right-hand side and initial guess call AZ_reorder_vec on initial guess and right-hand side selective an iterative solver and a preconditioner call AZ_solve call AZ_invorder_vec on solution

17. http://acts.nersc.gov/ Using Aztec: DMSR PE 0 PE 1 PE 2 local index to global index map ja: ja[0:nloc] - starting positions ja[nloc+1:] - global column indices val val[0:nloc-1] diagonal elements val[nloc] not used val[nloc+1:] off-diagonal nonzeros

18. http://acts.nersc.gov/ Using Aztec Two re-ordering: global re-ordering to distribute the matrix (done by user) local re-ordering to allow maximum overlapping of communication and computation (done by AZ_transform) Part of the right-hand side (elements beyond nloc) is used as scratch space internally more information at http://acts.nersc.gov/aztec/evaluation.html examples to look at /usr/local/pkg/Aztec/Aztec-2.0/app /u1/kewu/azmm.tar

19. http://acts.nersc.gov/ TAU Currently installed on the T3E Profiling of C++/C and Fortran programs Detailed information - much more than prof/gprof For C++: per-class and per-instance profiling Graphical display of profiling results Built-in viewers, interface to VAMPIR Automatic instrumentation in the future Get started: “module help tau” or “man tau”

20. http://acts.nersc.gov/ TAU Screen Shot

21. http://acts.nersc.gov/ Using TAU: instrumentation Template class Templated { private: T Data; public: T SetData (T d) { Data = d; return Data; } TAU_TYPE_STRING(str, CT(Data)+“(“+CT(d)+“)”); TAU_PROFILE(“Templated::SetData()”, str, TAU_DEFAULT);

22. http://acts.nersc.gov/ Using TAU: compilation Load module tau to define TAUROOTDIR include one of the Makefiles from ${TAUROOTDIR}/t3e/lib to define TAU_DEFS and TAULIBS Add ${TAULIBS} to link command Examples at ${TAUROOTDIR}/examples More help at http://acts.nersc.gov/tau/at-nersc.html

23. http://acts.nersc.gov/ Using TAU: view traces Must use VAMPIR on T3E To turn TAU trace files to VAMPIR trace file merge all TAU trace files into one (tau_merge) convert merged file with tau_convert -vampir see Makefile of examples in ${TAUROOTDIR}/examples To use VAMPIR module load vampir vampir tracefile.pv

24. Tools not on T3E or under development

25. http://acts.nersc.gov/ SuperLU Direct solution of large sparse linear systems Single processor and multithread versions are available http://www.nersc.gov/~xiaoye Distributed memory (MPI) version under development Attained 10 GFLOPS (speedup 100) on 512 node T3E 2-D matrix distribution static pivoting (replace small pivot with larger value) iterative refinement to ensure accuracy

26. http://acts.nersc.gov/ SuperLU http://www.nersc.gov/~xiaoye/SuperLU

27. http://acts.nersc.gov/ Hypre Family of problem specific preconditioners Physics and grid based Algebraic and structured multigrids Sparse approximate inverse Will be becoming available for use soon Should be useful stand-alone …and in the context of other ACTS tools http://www.llnl.gov/CASC/linear_solvers/

28. http://acts.nersc.gov/ PVODE Solves ODE & DAE Adams multi-step for non-stiff problems BDF multi-step for stiff problems Can use linear solvers from ACTS Integration with PETSc planned Now includes non-linear solvers (KINSOL) Newton based solvers Globalization strategies (line search, trust region) Http://www.llnl.gov/CASC/PVODE

29. http://acts.nersc.gov/ ATLAS and PHiPAC Automatic generation of machine optimized BLAS Can beat hand-tuned performance

30. http://acts.nersc.gov/ Global Arrays “Shared memory” programming interface on distributed memory computers But doesn’t hide remote access characteristics Global array object abstraction One-sided communication Used as infrastructure for several computational chemistry packages (e.g. NWChem on the NERSC T3E)

31. http://acts.nersc.gov/ Global Arrays http://www.emsl.pnl.gov/pub/proj/tms/hpcc_actinides

32. http://acts.nersc.gov/ Overture Complex geometry C++ Class Library Overlapping composite grids

33. http://acts.nersc.gov/ POOMA C++ framework (class library) for high performance parallel computing Abstractions for arrays, grids, particles Allows application to be written with high level “data- parallel” style T3E specific difficulties have apparently been overcome Extensive expression templates overwhelmed compilers

34. http://acts.nersc.gov/ CUMULVS Simple mechanisms for Computational steering Interactive visualization Fault tolerance Handles, for multiple attached viewers, Data collection, distribution Computational parameter control (“steering”) Can be used in heterogeneous environment (layered on PVM)

35. http://acts.nersc.gov/ Globus Provides many services to enable development of meta- computing applications: Resource discovery, monitoring, allocation Process (and IO) staging, management Communication - varied protocols - “Nexus” User and resource authentication, security (RSA) Hides implementation details from application Provides common “middleware” layer of interfaces Transparent remote access to resources User authentication, File staging, Batch queuing, etc., through common interfaces Perhaps of greatest immediate interest to application programmer

36. http://acts.nersc.gov/ PAWS Distributing data between separate parallel applications Coupling parallel simulation to parallel visualization tools that require different parallel data distributions run on same machine...or run across heterogeneous machines Minimal application coding requirements PAWS “Controller” handles interactions, data mapping

37. http://acts.nersc.gov/ PAWS Screen Shot Controller Sim. Code Viz. Code

38. http://acts.nersc.gov/ Future Directions Comparisons of parallel linear system solvers AZTEC BLOCKSOLVE ISIS++ PETSc PSPARSLIB Test problems, anyone ?

39. http://acts.nersc.gov/ Future Directions CCA (Common Component Architecture) Developing standardized ways of managing numerical “components” to allow mixing-and-matching Frameworks for gluing components together Similarities to CORBA, DCOM, Java Beans But: Scientific Interface Description Language (allowing Fortran) Designed for zero-copy interconnections ESI (Equation Solver Interface) Developing standardized interfaces for scalable linear algebra Specific test case for CCA component design

40. http://acts.nersc.gov/ Conclusion DOE has funded a nice set of tools We are here to maintain them for you Any questions? http://acts.nersc.gov/ acts-support@nersc.gov