1. Introduction to the ACTS Collection Why/How Do We Use These Tools? Tony Drummond Computational Research Division Lawrence Berkeley National Laboratory University of California, Berkeley - CS267 April 20, 2006 - Berkeley, California

2. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)2 Lesson = High Quality Software Reusability Programming Languages Code Optimization Algorithms Data Structures O/S - Compilers General Purpose Libraries Hardware - Middleware - Firmware Numerical Tools Run Time Support Code DevelopmentLibrary Development http://acts.nersc.gov Funded by DOE/ASCR Introduction to The ACTS Collection

3. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)3 ACTS Numerical Tools: Functionality Computational Problem MethodologyAlgorithmsLibrary Systems of Linear Equations Direct Methods LU Factorization ScaLAPACK(dense) SuperLU (sparse) Cholesky Factorization ScaLAPACK LDL T (Tridiagonal matrices) ScaLAPACK QR Factorization ScaLAPACK QR with column pivoting ScaLAPACK LQ factorization ScaLAPACK

4. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)4 ACTS Numerical Tools: Functionality Computational Problem MethodologyAlgorithmsLibrary Systems of Linear Equations (cont..) Iterative Methods Conjugate Gradient AztecOO (Trilinos) PETSc GMRES AztecOO PETSc Hypre CG Squared AztecOO PETSc Bi-CG Stab AztecOO PETSc Quasi-Minimal Residual (QMR) AztecOO Transpose Free QMR AztecOO PETSc

5. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)5 Structure of PETSc Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK Profiling Interface PETSc PDE Application Codes Object-Oriented Matrices, Vectors, Indices Grid Management Linear Solvers Preconditioners + Krylov Methods Nonlinear Solvers, Unconstrained Minimization ODE Integrators Visualization Interface

6. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)6 Data Layout structuredcompositeblock-strucunstrucCSR Linear Solvers GMG,...FAC,...Hybrid,...AMGe,...ILU,... Linear System Interfaces Hypre Conceptual Interfaces

7. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)7 Hypre Conceptual Interfaces INTERFACE TO SOLVERS List of Solvers and Preconditioners per Conceptual Interface

8. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)8 ACTS Numerical Tools: Functionality Computational Problem MethodologyAlgorithmsLibrary Systems of Linear Equations (cont..) Iterative Methods (cont..) SYMMLQ PETSc Precondition CG AztecOO PETSc Hypre Richardson PETSc Block Jacobi Preconditioner AztecOO PETSc Hypre Point Jocobi Preconditioner AztecOO Least Squares Polynomials PETSc

9. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)9 ACTS Numerical Tools: Functionality Computational Problem MethodologyAlgorithmsLibrary Systems of Linear Equations (cont..) Iterative Methods (cont..) SOR Preconditioning PETSc Overlapping Additive Schwartz PETSc Approximate Inverse Hypre Sparse LU preconditioner AztecOO PETSc Hypre Incomplete LU (ILU) preconditioner AztecOO Least Squares Polynomials PETSc MultiGrid (MG) Methods MG Preconditioner PETSc Hypre Algebraic MG Hypre Semi-coarsening Hypre

10. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)10 Computational Problem MethodologyAlgorithmLibrary Linear Least Squares Problems Least Squares ScaLAPACK Minimum Norm Solution ScaLAPACK Minimum Norm Least Squares ScaLAPACK Standard Eigenvalue Problem Symmetric Eigenvalue Problem For A=A H or A=A T ScaLAPACK (dense) SLEPc (sparse) Singular Value Problem Singular Value Decomposition ScaLAPACK (dense) SLEPc (sparse) Generalized Symmetric Definite Eigenproblem Eigenproblem ScaLAPACK (dense) SLEPc (sparse) ACTS Numerical Tools: Functionality

11. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)11 ACTS Numerical Tools: Functionality Computational Problem MethodologyAlgorithmLibrary Non-Linear Equations Newton Based Line Search PETSc Trust Regions PETSc Pseudo-Transient Continuation PETSc Matrix FreePETSc

12. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)12 ACTS Numerical Tools: Functionality Computational Problem MethodologyAlgorithmLibrary Non-Linear Optimization Newton Based Newton OPT++ TAO Finite-Difference Newton OPT++ TAO Quasi-Newton OPT++ TAO Non-linear Interior Point OPT++ TAO CG Standard Non- linear CG OPT++ TAO Limited Memory BFGS OPT++ Gradient Projections TAO Direct Search No derivate information OPT++

13. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)13 TAO - Interface with PETSc

14. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)14 OPT++ Interfaces Four major classes of problems available NLF0(ndim, fcn, init_fcn, constraint) Basic nonlinear function, no derivative information available NLF1(ndim, fcn, init_fcn, constraint) Nonlinear function, first derivative information available FDNLF1(ndim, fcn, init_fcn, constraint) Nonlinear function, first derivative information approximated NLF2(ndim, fcn, init_fcn, constraint) Nonlinear function, first and second derivative information available

15. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)15 ACTS Numerical Tools: Functionality Computational Problem MethodologyAlgorithmLibrary Non-Linear Optimization Newton Based Newton OPT++ TAO Finite-Difference Newton OPT++ TAO Quasi-Newton OPT++ TAO Non-linear Interior Point OPT++ TAO CG Standard Non- linear CG OPT++ TAO Limited Memory BFGS OPT++ Gradient Projections TAO Direct Search No derivate information OPT++

16. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)16 ACTS Numerical Tools: Functionality Computational Problem MethodologyAlgorithmLibrary Non-Linear Optimization (cont..) Semismoothing Feasible Semismooth TAO Unfeasible semismooth TAO Ordinary Differential Equations Integration Adam-Moulton (Variable coefficient forms) CVODE (SUNDIALS) CVODES Backward Differential Formula Direct and Iterative Solvers CVODE CVODES Nonlinear Algebraic Equations Inexact Newton Line Search KINSOL (SUNDIALS) Differential Algebraic Equations Backward Differential Formula Direct and Iterative Solvers IDA (SUNDIALS)

17. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)17 ACTS Numerical Tools: Functionality Computational Problem SupportTechniquesLibrary Writing Parallel Programs Distributed Arrays Shared-Memory Global Arrays Distributed Memory CUMULVS (viz) Globus (Grid) Grid Generation OVERTURE Structured Meshes CHOMBO (AMR) Hypre OVERTURE PETSc Semi-Structured Meshes CHOMBO (AMR) Hypre OVERTURE Distributed Computing GRID Globus Remote Steering CUMULVS Coupling PAWS

18. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)18 ACTS Numerical Tools: Functionality Computational Problem SupportTechniqueLibrary Writing Parallel Programs (cont.) Distributed Computing Check-point/restart CUMULVS Profiling Algorithmic Performance Automatic instrumentation PETSc User Instrumentation PETSc Execution Performance Automatic Instrumentation TAU User Instrumentation TAU Code Optimization Library Installation Linear Algebra Tuning ATLAS Interoperability Code Generation Language BABEL CHASM Components CCA

19. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)19 -ksp_type [cg,gmres,bcgs,tfqmr,…] -pc_type [lu,ilu,jacobi,sor,asm,…] More advanced: -ksp_max_it -ksp_gmres_restart -pc_asm_overlap User Interfaces CALL BLACS_GET( -1, 0, ICTXT ) CALL BLACS_GRIDINIT( ICTXT, 'Row-major', NPROW, NPCOL ) : CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL ) : CALL PDGESV( N, NRHS, A, IA, JA, DESCA, IPIV, B, IB, JB, DESCB, $ INFO ) Library Calls Command lines Data Layout structuredcompositeblockstrcunstrucCSR Linear Solvers GMGFACHybrid,...AMGeILU,... Linear System Interfaces Problem Domain Introduction to The ACTS Collection

20. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)20 User Interfaces ScaLAPACK Chombo PETSc AZTEC PVODE Hypre SuperLUTAO Overture Global Arrays PAWS OPT++ Globus CUMULVSTAU High Level Interfaces View_field(T1) Ax = b matlab*P PyACTS User NetSolve Star-P Introduction to The ACTS Collection

21. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)21 Tool Interoperability Tool-to-Tool TOOL A TOOL B TOOL C TOOL F TOOL E TOOL D PETSc TAU Ex 1 Ex 2 Trilinos Ex 3

22. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)22 Component Technology! Tool A Tool C Tool D Tool B CCA ESI

23. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)23 Introduction to PETSc Introducción a PETSC Introducción a PETSC VectorsVectors MatricesMatrices PETSc viewersPETSc viewers Use of Preconditioners in PETSc Use of Preconditioners in PETSc Some examples using PETSc Some examples using PETSc

24. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)24 What is PETSc? Developed at Argonne National Laboratory Library that supports work with PDE Public Domain Portable to a variety of platforms Project started in 1991. Written in C with a fortran interface and OO flavor PETSc: Portable Extensible Toolkit for Scientific Computation PETSc: Portable Extensible Toolkit for Scientific Computation What is PETSc?

25. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)25 Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK Profiling Interface PETSc PDE Application Codes Object-Oriented Matrices, Vectors, Indices Grid Management Linear Solvers Preconditioners + Krylov Methods Nonlinear Solvers, Unconstrained Minimization ODE Integrators Visualization Interface PETSc Structure

26. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)26 Some Algorithmic Implementations in PETSc Compressed Sparse Row (AIJ) Blocked Compressed Sparse Row (BAIJ) Block Diagonal (BDIAG) DenseOther IndicesBlock IndicesStrideOther Index Sets Vectors Line SearchTrust Region Newton-based Methods Other Nonlinear Solvers Additive Schwartz Block Jacobi ILU ICC LU (Sequential only) Others Preconditioners Euler Backward Euler Pseudo Time Stepping Other Time Steppers GMRESCGCGSBi-CG-STABTFQMR RichardsonChebychevOther Krylov Subspace Methods Matrices Distributed Arrays Matrix-free

27. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)27 Vectors Fundamental Objects to store fields, right-hand side vectors, etc.. Matrices Fundamental Objects to store operators (like Jacobians) Vectors and Matrices in PETSc

28. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)28 Each process owns a piece of the vector VECTOR Types: Sequential, MPI or SHARED proc 3 proc 2 proc 0 proc 4 proc 1 –VecCreate(MPI_Comm Comm,Vec * v) comm - MPI_Comm parallel processes v = vector –VecSetType(Vec,VecType) Valid Types are: –VEC_SEQ, VEC_MPI, o VEC_SHARED –VecSetSizes(Vec *v,int n, int N)  Where n or N (not both) can be PETSC_DECIDE – VecDestroy(Vec *) Basic Vector Operations

29. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)29 PETSc - Vector Example (in C) #include petscvec.h int main(int argc,char **argv) { Vec x; int n = 20,m=4, ierr; PetscInitialize(&argc,&argv); VecCreate(PETSC_COMM_WORLD,&x); VecSetSizes(x,PETSC_DECIDE,n); VecSetFromOptions(x); PetscFinalize(); return 0; } VecCreateMPI(PETSC_COMM_WORLD, m, n, x); Basic Vector Operations

30. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)30 Function Name Operation VecAXPY(Scalar *a, Vec x, Vec y) y = y + a*x VecAYPX(Scalar *a, Vec x, Vec y) y = x + a*y VecWAXPY(Scalar *a, Vec x, Vec y, Vec w) w = a*x + y VecScale(Scalar *a, Vec x) x = a*x VecCopy(Vec x, Vec y) y = x VecPointwiseMult(Vec x, Vec y, Vec w) w_i = x_i *y_i VecMax(Vec x, int *idx, double *r) r = max x_i VecShift(Scalar *s, Vec x) x_i = s+x_i VecAbs(Vec x) x_i = |x_i | VecNorm(Vec x, NormType type, double *r) r = ||x|| Basic Operations with Vectors

31. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)31 Types: default sparse AIJ: MPIAIJ, SEQAIJ block sparse AIJ (for multi-component PDEs): MPIAIJ, SEQAIJ symmetric block sparse AIJ: MPISBAIJ, SAEQSBAIJ block diagonal: MPIBDIAG, SEQBDIAG dense: MPIDENSE, SEQDENSE matrix-free –MatCreate(MPI_Comm Comm, Mat *A) MPI_Comm - group of process that work on A –MatSetSizes(Mat A, PetscInt m, PetscInt n, PetscInt M, PetscInt N) –MatSetType(Mat,MatType) –MatDestroy(Mat) Basic Matrix Operations

32. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)32 Single user interface to matrices (operators) Input values to a PETSc matrix MatSetValues( Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv ) Matriz-vector multiplication For instance: MatMult(A,y,x) (y≠x) Matrix viewing MatView() Multiple underlying implementations AIJ, block AIJ, symmetric block AIJ, block diagonal, dense, matrix-free, etc. Basic Matrix Operations

33. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)33 proc 3 proc 2 proc 1 M=8,N=8,m=3,n=k 1 rstart=0,rend=4 M=8,N=8,m=3,n=k 2 rstart=3,rend=6 M=8,N=8,m=2,n= k 3 rstart=6,rend=8 The values of k1, k2 y k3 depend on the application and must be compatible with vector x (Ax=b) MatGetOwnershipRange(Mat A, int *rstart, int *rend) rstart: first global row number that belongs to a process rend -1: Last row global number that belongs to a process The values of k1, k2 y k3 depend on the application and must be compatible with vector x (Ax=b) MatGetOwnershipRange(Mat A, int *rstart, int *rend) rstart: first global row number that belongs to a process rend -1: Last row global number that belongs to a process Block-Row distribution per process Basic Matrix Operations

34. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)34 PETSc Matrix Example Included in PETSc Distribution: 1)$PETSC_DIR/src/mat/tutorial/ex2.c 2) $PETSC_DIR/src/mat/tutorial/ex5.c

35. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)35 Obtain basic information on PETSc viewers Run time options Extract data that can be later use by other packages/app. programs. vector fields, matrix contents various formats (ASCII, binary) Visualization simple graphics created with X11 to display vector campo, non-zero element structure of a matrix, etc. Basic Viewer Operations

36. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)36 MatView(Mat A, PetscViewer v); Runtime options available after matrix assembly -mat_view_info Information related to matrix type -mat_view_draw nonzero pattern -mat_view data in ASCII etc. Basic Viewer Operations

37. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)37 Examples of Viewers and Matrices Included in PETSc Distribution: 1)$PETSC_DIR/src/mat/tests/ex2.c 1)Using the -mat_view_info_detailed, etc 2) $PETSC_DIR/src/mat/tests/ex3.c 1)Using the -mat-view-draw

38. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)38 Objective: Support the solution of Linear System of the form, Ax=b, Krylov Subspace methods are highly dependent of the spectrum of the matrix A. The use of preconditioners can speed-up their convergence. PETSc KSP KRYLOV SUBSPACE METHODS + PRECONDITIONERS R. Freund, G. H. Golub, and N. Nachtigal. Iterative Solution of Linear Systems,pp 57-100. ACTA Numerica. Cambridge University Press, 1992.

39. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)39 For M L =I For M R =I PETSC Default Right vs. Left Preconditioning Ax=b,

40. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)40 PETSc Application Initialization Evaluation of A and b Post- Processing Solve Ax = b PCKSP Linear Solvers (SLES) PETSc codeUser code Main Routine Using PETSc’s KSP

41. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)41 Conjugate Gradient GMRES CG-Squared Bi-CG-stab Transpose-free QMR Chebychev CGNE Many more (check manual) Block Jacobi Overlapping Additive Schwarz ICC, ILU via BlockSolve95 ILU(k), LU etc. Krylov Methods (KSP)Preconditioners (PC) Solvers and Preconditioners in PETSc KSPPREONLY Interface to Other Packages (like SuperLU, SuperLU_DIST, Hypre, etc)

42. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)42 PETSc codeUser code Application Initialization Function Evaluation Jacobian Evaluation Post- Processing PC PETSc Main Routine Linear Solvers (KSP) Nonlinear Solvers (SNES) Timestepping Solvers (TS) Flow of Control for PDE Solution

43. 04/20/06The DOE ACTS Collection (http://acts.nersc.gov)43 Time Stepper Types (TS) ODE Solvers (TS) in PETSc TS_EULER - Euler TS_SUNDIALS- SUNDIALS interface TS_BEULER - Backward Euler TS_PSEUDO- Pseudo-timestepping Linear TS Example (ex3.c) Non Linear TS Example (ex1.c)