1. NERSC User Group Meeting The DOE ACTS Collection Osni Marques Lawrence Berkeley National Laboratory OAMarques@lbl.gov

2. 09/19/2007NUG Meeting2 What is the ACTS Collection? Advanced CompuTational Software Collection Tools for developing parallel applications ACTS started as an “umbrella” project http://acts.nersc.gov Goals  Extended support for experimental software  Make ACTS tools available on DOE computers  Provide technical support (acts-support@nersc.gov)  Maintain ACTS information center (http://acts.nersc.gov)  Coordinate efforts with other supercomputing centers  Enable large scale scientific applications  Educate and train

3. 09/19/2007NUG Meeting3 ACTS Timeline 199920032007 ACTS Toolkit User Community ACTS Challenge Codes Computing Systems Interoperability Pool of Software Tools Testing and Acceptance Phase Workshops and Training Scientific Computing Centers Computer Vendors Numerical Simulations Physics Chemistry Biology Medicine Mathematics Bioinformatics Computer Sciences Engineering Software Collection Software Sustainability Center Software Tool Box

4. 09/19/2007NUG Meeting4 Challenges in the Development of Scientific Codes Research in computational sciences is fundamentally interdisciplinary The development of complex simulation codes on high-end computers is not a trivial task Productivity Time to the first solution (prototype) Time to solution (production) Other requirements Complexity Increasingly sophisticated models Model coupling Interdisciplinarity Performance Increasingly complex algorithms Increasingly complex architectures Increasingly demanding applications Research in computational sciences is fundamentally interdisciplinary The development of complex simulation codes on high-end computers is not a trivial task Productivity Time to the first solution (prototype) Time to solution (production) Other requirements Complexity Increasingly sophisticated models Model coupling Interdisciplinarity Performance Increasingly complex algorithms Increasingly complex architectures Increasingly demanding applications Libraries written in different languages Discussions about standardizing interfaces are often sidetracked into implementation issues Difficulties managing multiple libraries developed by third-parties Need to use more than one language in one application The code is long-lived and different pieces evolve at different rates Swapping competing implementations of the same idea and testing without modifying the code Need to compose an application with some other(s) that were not originally designed to be combined Libraries written in different languages Discussions about standardizing interfaces are often sidetracked into implementation issues Difficulties managing multiple libraries developed by third-parties Need to use more than one language in one application The code is long-lived and different pieces evolve at different rates Swapping competing implementations of the same idea and testing without modifying the code Need to compose an application with some other(s) that were not originally designed to be combined

5. 09/19/2007NUG Meeting5 Current ACTS Tools and their Functionalities CategoryToolFunctionalities Numerical Trilinos Algorithms for the iterative solution of large sparse linear systems (includes AZTEC00) Hypre Algorithms for the iterative solution of large sparse linear systems, intuitive grid-centric interfaces, and dynamic configuration of parameters. PETSc Tools for the solution of PDEs that require solving large-scale, sparse linear and nonlinear systems of equations. OPT++ Object-oriented nonlinear optimization package. SUNDIALS Solvers for the solution of systems of ordinary differential equations, nonlinear algebraic equations, and differential-algebraic equations. ScaLAPACK Library of high performance dense linear algebra routines for distributed-memory message- passing. SLEPc Eigensolver package built on top of PETSc SuperLU General-purpose library for the direct solution of large, sparse, nonsymmetric systems of linear equations. TAO Large-scale optimization software, including nonlinear least squares, unconstrained minimization, bound constrained optimization, and general nonlinear optimization. Code Development Global Arrays Library for writing parallel programs that use large arrays distributed across processing nodes and that offers a shared-memory view of distributed arrays. Overture Object-Oriented tools for solving computational fluid dynamics and combustion problems in complex geometries. Code Execution TAU Set of tools for analyzing the performance of C, C++, Fortran and Java programs. Library Development ATLAS Tools for the automatic generation of optimized numerical software for modern computer architectures and compilers. Availability To be installed Installed Upon request Installed* To be installed Installed* To be installed Installed** Upon request Under test Upon request * Also in LibSci ** USG

6. 09/19/2007NUG Meeting6 ACTS Tools: numerical functionalities Computational ProblemMethodologyAlgorithmsLibrary Linear Equations Direct Methods LU factorizationScaLAPACK (dense), SuperLU (sparse) Cholesky factorizationScaLAPACK LDL T factorization (tridiagonal matrices) QR factorization QR factorization with column pivoting LQ factorization Full orthogonal factorization Generalized QR factorization Iterative Methods Conjugate gradient (CG)AztecOO (Trilinos), PETSc GMRESAztecOO, Hypre, PETSc CG SquaredAztecOO, PETSc Bi-CG-Stab QMRAztecOO Transpose free QMRAztecOO, PETSc SYMMLQPETSc Richardson Block Jacobi preconditionerAztecOO, Hypre, PETSc Point Jacobi preconditionerAztecOO Least-squares polynomials SOR preconditionerPETSc Overlapping additive Schwarz Approximate inverse Hypre Sparse LU preconditionerAztecOO, Hypre, PETSc Incomplete LU (ILU) preconditioner Multigrid MG preconditionerHypre, PETSc Algebraic multigridML (Trilinos), Hypre SemicoarseningHypre

7. 09/19/2007NUG Meeting7 ACTS Tools: numerical functionalities Computational ProblemMethodologyAlgorithmsLibrary Linear least squaresleast squares min x  b  Ax  2 ScaLAPACK minimum norm min x  x  2 minimum norm least squares min x  x  2 and min x  b  Ax  2 Standard eigenvalue problemsiterative, direct Az= z for A=A T or A=A H ScaLAPACK (dense), SLEPc (sparse) Generalized eigenvalue problems Az= Bz, ABz= z, BAz= z Singular value decomposition A=U  V T, A=U  V H Non-linear equations problemsNewton-based Line searchPETSc, KINSOL (SUNDIALS) Trust regionsPETSc Pseudotransient continuationPETSc Matrix-freePETSc Nonlinear optimizationNewton-based NewtonOPT++, TAO Finite differencesOPT++ Quasi-NewtonOPT++, TAO (LMVM) Nonlinear interior pointOPT++, TAO CG Standard nonlinear CGOPT++, TAO Limited memory BFGSOPT++ Gradient projectionTAO Direct Search Without derivative informationOPT++ Semismooth Infeasible semismoothTAO Feasible semismooth

8. 09/19/2007NUG Meeting8 ACTS Tools: numerical functionalities Computational ProblemMethodologyAlgorithmsLibrary ODEsIntegration Variable coefficient Adams-MoultonCVODE (SUNDIALS) Backward differential Direct and iterative solvers ODEs with sensitivity analysisIntegration Variable coefficient Adams-Moulton Backward differential Direct and iterative solvers Differential-algebraic equationsBackward differential formula Direct and iterative solversIDA (SUNDIALS) Nonlinear equations with sensitivity analysis Inexact Newton line searchSensKINSOL (SUNDIALS) Tuning and optimizationAutomatic code generator BLAS and some LAPACK routinesATLAS

9. 09/19/2007NUG Meeting9 Software Interfaces CALL BLACS_GET( -1, 0, ICTXT ) CALL BLACS_GRIDINIT( ICTXT, 'Row-major', NPROW, NPCOL ) : CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL ) : CALL PDGESV( N, NRHS, A, IA, JA, DESCA, IPIV, B, IB, JB, DESCB, INFO ) Data Layout structuredcompositeblockstrcunstrucCSR Linear Solvers GMGFACHybrid,...AMGeILU,... Linear System Interfaces -ksp_type [cg,gmres,bcgs,tfqmr,…] -pc_type [lu,ilu,jacobi,sor,asm,…] More advanced: -ksp_max_it -ksp_gmres_restart -pc_asm_overlap -pc_asm_type [basic,restrict,interpolate,none] command line function call problem domain (ScaLAPACK) (PETSc) (Hypre)

10. 09/19/2007NUG Meeting10 Use of ACTS Tools Electronic structure optimization performed with TAO, (UO 2 ) 3 (CO 3 ) 6 (courtesy of deJong). Molecular dynamics and thermal flow simulation using codes based on Global Arrays. GA have been employed in large simulation codes such as NWChem, GAMESS-UK, Columbus, Molpro, Molcas, MWPhys/Grid, etc. Problems (different grid types) solved with Hypre. Micro-FE bone modeling using ParMetis, Prometheus and PETSc; models up to 537 million DOF (Adams, Bayraktar, Keaveny, and Papadopoulos). Model of the heart mechanics (blood- muscle-valve) by an adaptive and parallel version of the immersed boundary method, using PETSc, Hypre and SAMRAI (courtesy of Boyce Griffith, New York University). 3D incompressible Euler,tetrahedral grid, up to 11 million unknowns, based on a legacy NASA code, FUN3d (W. K. Anderson), fully implicit steady-state, parallelized with PETSc (courtesy of Kaushik and Keyes).

11. 09/19/2007NUG Meeting11 Use of ACTS Tools Induced current (white arrows) and charge density (colored plane and gray surface) in crystallized glycine due to an external field (Louie, Yoon, Pfrommer and Canning), eigenvalue problems solved with ScaLAPACK. OPT++ is used in protein energy minimization problems (shown here is protein T162 from CASP5, courtesy of Meza, Oliva et al.) Omega3P is a parallel distributed-memory code intended for the modeling and analysis of accelerator cavities, which requires the solution of generalized eigenvalue problems. A parallel exact shift-invert eigensolver based on PARPACK and SuperLU has allowed for the solution of a problem of order 7.5 million with 304 million nonzeros. Finding 10 eigenvalues requires about 2.5 hours on 24 processors of an IBM SP. Two ScaLAPACK routines, PZGETRF and PZGETRS, are used for solution of linear systems in the spectral algorithms based AORSA code (Batchelor et al.), which is intended for the study of electromagnetic wave-plasma interactions. The code reaches 68% of peak performance on 1936 processors of an IBM SP.

12. 09/19/2007NUG Meeting12 ScaLAPACK BLAS LAPACK BLACS MPI/PVM/... PBLAS Global Local platform specific Clarity,modularity, performance and portability. Atlas can be used here for automatic tuning. Linear systems, least squares, singular value decomposition, eigenvalues. Communication routines targeting linear algebra operations. Parallel BLAS. Communication layer (message passing). http://acts.nersc.gov/scalapack Version 1.7.5 released in January 2007; NSF funding for further development. UTK, UCB …

13. 09/19/2007NUG Meeting13 ScaLAPACK: understanding performance 60 processors, Dual AMD Opteron 1.4GHz Cluster with Myrinet Interconnect, 2GB memory LU on 2.2 GHz AMD Opteron (4.4 GFlop/s peak performance)

14. 09/19/2007NUG Meeting14 ScaLAPACK: understanding the 2D block-cyclic distribution http://acts.nersc.gov/scalapack/hands-on/datadist.html

15. 09/19/2007NUG Meeting15 PETSc Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK Profiling Interface PETSc PDE Application Codes Object-Oriented Matrices, Vectors, Indices Grid Management Linear Solvers Preconditioners + Krylov Methods Nonlinear Solvers, Unconstrained Minimization ODE Integrators Visualization Interface ANL Portable, Extensible Toolkit for Scientific computation

16. 09/19/2007NUG Meeting16 PETSc: Linear Solvers (SLES) PETSc Application Initialization Evaluation of A and b Post- Processing Solve Ax = b PCKSP Linear Solvers (SLES) PETSc codeUser code Main Routine

17. 09/19/2007NUG Meeting17 PETSc: setting SLES parameters at run time -ksp_type [cg,gmres,bcgs,tfqmr,…] -pc_type [lu,ilu,jacobi,sor,asm,…] more advanced: -ksp_max_it -ksp_gmres_restart -pc_asm_overlap -pc_asm_type [basic,restrict,interpolate,none] many more (see manual)

18. 09/19/2007NUG Meeting18 Important Questions for Application Developers How does performance vary with different compilers? Is poor performance correlated with certain OS features? Has a recent change caused unanticipated performance? How does performance vary with MPI variants? Why is one application version faster than another? What is the reason for the observed scaling behavior? Did two runs exhibit similar performance? How are performance data related to application events? Which machines will run my code the fastest and why? Which benchmarks predict my code performance best? From http://acts.nersc.gov/events/Workshop2005/slides/Shende.pdf

19. 09/19/2007NUG Meeting19 TAU Multi-level performance instrumentation Multi-language automatic source instrumentation Flexible and configurable performance measurement Widely-ported parallel performance profiling system Computer system architectures and operating systems Different programming languages and compilers Support for multiple parallel programming paradigms Multi-threading, message passing, mixed-mode, hybrid Support for performance mapping Support for object-oriented and generic programming Integration in complex software systems and applications U Oregon Tuning and Analysis Utilities

20. 09/19/2007NUG Meeting20 Definitions: profiling and tracing Profiling Recording of summary information during execution (inclusive and exclusive time, number of calls, hardware statistics, etc) Reflects performance behavior of program entities (functions, loops, basic blocks, user-defined “semantic” entities) Very good for low-cost performance assessment Helps to expose performance bottlenecks and hotspots Implemented through sampling: periodic OS interrupts or hardware counter traps instrumentation: direct insertion of measurement code Tracing Recording of information about significant points (events) during program execution entering/exiting code region (function, loop, block, etc) thread/process interactions (send/receive message, etc) Save information in event record timestamp CPU identifier, thread identifier Event type and event-specific information Event trace is a time-sequenced stream of event records Can be used to reconstruct dynamic program behavior Typically requires code instrumentation

21. 09/19/2007NUG Meeting21 TAU: Example 1 (1/2) http://acts.nersc.gov/tau/programs/pdgssvx set the C compiler Ex. tau-multiplecounters-mpi-papi-pdt

22. 09/19/2007NUG Meeting22 TAU: Example 1 (2/2) PAPI provides access to hardware performance counters (see http://icl.cs.utk.edu/papi for details and contact acts-support@nersc.gov for the corresponding TAU events). In this example we are just measuring FLOPS. PARAPROF

23. 09/19/2007NUG Meeting23 TAU: Example 2 (1/2) PESCAN is a code that uses the folded spectrum method for nonselfconsistent nanoscale calculations. It uses a planewave basis, and conventional Kleinman-Bylander nonlocal pseudopotetials in real space. It is parallelized using MPI and can calculate million atom systems. # Makefile for PESCAN include $(TAULIBDIR)/Makefile.tau-multiplecounters-mpi-papi-pdt #include $(TAULIBDIR)/Makefile.tau-callpath-mpi-pdt FC = $(TAU_COMPILER) mpxlf90_r CC = $(TAU_COMPILER) mpcc_r ⋮

24. 09/19/2007NUG Meeting24 TAU: Example 2 (2/2)

25. 09/19/2007NUG Meeting25 The Case for Software Libraries machine tuned and dependent modules application data layout CONTROL I/O algorithmic implementations APPLICATION New architecture: may or may not need re-rewriting New developments: difficult to predict New architecture: extensive re-rewriting New or extended Physics: extensive re-rewriting or increased overhead New architecture: minimal to extensive rewriting New architecture or software: Extensive tuning May require new programming paradigms Difficult to maintain!

26. 09/19/2007NUG Meeting26 ACTS: value-added services PyACTS Requirements for reusable high quality software tools Integration, maintenance and support efforts Interfaces using script languages Software automation  More information: acts-support@nersc.gov http://acts.nersc.gov