1. ML: A Multilevel Preconditioning Package Copper Mountain Conference on Iterative Methods March 29-April 2, 2004 Jonathan Hu Ray Tuminaro Marzio Sala Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL85000.

2. Outline Overview Multigrid basics Available user options Configuring & building Interoperability with other packages Example program Conclusions

3. ML Package Provides parallel multigrid preconditioning for linear solver methods Current developers: Ray Tuminaro, Jonathan Hu, Marzio Sala Former: Charles Tong (LLNL) Main methods –Geometric Grid refinement hierarchy 2-level FE basis function domain decomposition –AMG (algebraic multigrid based on aggregation) Smoothed aggregation* Edge-element AMG for Maxwell’s equations* n-level (smoothed) aggregation domain decomposition * Classical AMG Written primarily in C –C++ interfaces to various Trilinos packages –102,411 lines of code (as of this afternoon) –26 example programs –Downloadable as part of Trilinos –CVS, bugzilla

4. MG(f, u, k) { if (k == 1) u 1 = (A 1 ) -1 f 1 else { S k (A k, f k,u k ) //pre-smooth r k = f k – A k u k f k-1 = R k-1 r k ; u k-1 = 0 //restrict MG(f k-1, u k-1, k-1) u k = u k + I k-1 u k-1 //interpolate & correct S k (A k, f k,u k ) //post-smooth } A k, R k, I k, S k required on all levels S k : smoothers R k : restriction operators I k : interpolation operators Recursive MG Algorithm (V Cycle) V Cycle to solve A 4 u 4 =f 4 k=4 k=1 W Cycle FMV Cycle

5. ML Capabilities MG cycling: V, W, full V, full W Grid Transfers –Several automatic coarse grid generators –Several automatic grid transfer operators –Coarse “grid” visualization capabilities Smoothers –Jacobi, Gauss-Seidel, Hiptmair, Krylov methods, sparse approximate inverses, Chebyshev Variety of Serial & Parallel Direct Solvers –SuperLU, Umfpack, KLU, MUMPS, etc. Kernels: matrix/matrix multiply, etc.

6. Smoothed Aggregation Developed for linear elasticity (Vanek, Mandel, Brezina) Construct tentative prolongator t P –Aggregation: group unknowns together –Augment: interpolate null space (e.g., rigid body modes) Construct final prolongator P –Smooth: P = (I –  D -1 A) T P lower energy in basis functions

7. 5 55 2 2 2 7 7 7 2 Take null space (e.g., const.) Smooth Split into local basis functions Smoothed Aggregation

8. Uncoupled / MIS Aggregation Greedy algorithm

9. ● Global graph partitioning – Operates on global domain – ParMETIS – Aggregates can span processors Graph partitioning aggregation ● Local graph partitioning – Processors work independently – METIS –  1 aggregate per processor

10. ML Smoother Choices Jacobi, Point/Block Gauss Seidel MLS –Based on Chebyshev polynomials of smoothed operator. –Serial: competitive with true Gauss-Seidel –Parallel: Performance is independent of # processors Hiptmair –Distributed relaxation smoother for Maxwell’s Eqns. –Other smoothers used within each projection step Aztec solvers –Krylov methods, incomplete factorizations Direct solution (via Amesos interface): –UMFPACK, KLU (serial) –SuperLU –MUMPS

11. ML and Other Packages ML Epetra Accepts user data as Epetra objects Can be wrapped as Epetra_Operator TSF TSF interface exists Other matvecs Other solvers Accepts other solvers and MatVecs Amesos Amesos interface for direct solvers Aztecoo Meros Via Epetra & TSF

12. Configuring and Building ML Builds by default when you configure & build Trilinos By default, you get –Epetra & Aztecoo support –Example suite ( Trilinos/packages/ml/examples ) Some options of interest (off by default) –MPI support –Graph partitioning aggregation (METIS, ParMETIS) –Direct solvers (Amesos) –Profiling configure --with-mpi-compilers=/usr/local/mpich/bin --with-ml_amesos \ --with-libs=“-lamesos –lsuperlu”

13. A Small Example: ml/examples/ml_example_epetra_preconditioner.cpp Linear Solver ML: multi- grid pre- cond. AztecOO (Epetra) ML (Teuchos) gmres AMG Solution component Example methods Packages used Solve Ax=b: A: advection/diffusion operator Linear solver: gmres Precond.: 2-level AMG, graph-partitioning aggregation

14. ml_example_epetra_preconditioner.cpp Trilinos_Util_CrsMatrixGallery Gallery(“recirc_2d", Comm); Gallery.Set("problem_size", 10000); // linear system matrix & linear problem Epetra_RowMatrix * A = Gallery.GetMatrix(); Epetra_LinearProblem * Problem = Gallery.GetLinearProblem(); // Construct outer solver object AztecOO solver(*Problem); // Set some solver options solver.SetAztecOption(AZ_solver, AZ_gmres); solver.SetAztecOption(AZ_output, 10); solver.SetAztecOption(AZ_kspace, 160);

15. example (contd.) // Set up multilevel preconditioner ParameterList MLList; // parameter list for ML options MLList.set("max levels",2); MLList.set("aggregation: type", "METIS"); // graph partitioning MLList.set("aggregation: nodes per aggregate", 16); // set up aztecoo smoother MLList.set("smoother: type","aztec"); int options[AZ_OPTIONS_SIZE]; double params[AZ_PARAMS_SIZE]; AZ_defaults(options,params); options[AZ_precond] = AZ_dom_decomp; options[AZ_subdomain_solve] = AZ_ilut; MLList.set("smoother: aztec options", options); MLList.set("smoother: aztec params", params); MLList.set("coarse: type","Amesos_Superludist"); MLList.set("coarse: max processes", 4); ML_Epetra::MultiLevelPreconditioner * MLPrec = new // create preconditioner ML_Epetra::MultiLevelPreconditioner(*A, MLList, true); solver.SetPrecOperator(MLPrec); // tell solver to use ML preconditioner solver.Iterate(500, 1e-12); // iterate at most 500 times

16. AMG for Common Problem Types Trilinos_Util_CrsMatrixGallery Gallery(“laplace_3d", Comm); Gallery.Set("problem_size", 100*100*100); // linear system matrix & linear problem Epetra_RowMatrix * A = Gallery.GetMatrix(); Epetra_LinearProblem * Problem = Gallery.GetLinearProblem(); // Construct outer solver object AztecOO solver(*Problem); solver.SetAztecOption(AZ_solver, AZ_cg); // Set up multilevel precond. with smoothed aggr. defaults ParameterList MLList; // parameter list for ML options ML_Epetra::SetDefaults(“SA”,MLList); ML_Epetra::MultiLevelPreconditioner * MLPrec = new // create preconditioner ML_Epetra::MultiLevelPreconditioner(*A, MLList, true); solver.SetPrecOperator(MLPrec); // tell solver to use ML preconditioner solver.Iterate(500, 1e-12); // iterate at most 500 times Other problem types: “DD”, “DD-ML”, “maxwell”

17. Collaborations Within Sandia –ALEGRA Radiation Maxwell Electrostatic potential –MPSalsa (Shadid, et al.) –CEPTRE External users –EM3D (Lawrence Berkeley NL) –P. Arbenz (ETH Zurich) –R. Geus (PSI) –J. Fish, H. Waisman (RPI) Potential Users –PREMO (Sandia)

18. Getting Help Website: www.cs.sandia.gov/~tuminaro/mlwww.cs.sandia.gov/~tuminaro/ml See Trilinos/packages/ml/examples See guide: –ML User’s Guide, ver. 3.0 (in ml/doc) Mailing lists –ml-users@software.sandia.govml-users@software.sandia.gov –ml-announce@software.sandia.govml-announce@software.sandia.gov Bug reporting, enhancement requests via bugzilla: –http://software.sandia.gov/bugzillahttp://software.sandia.gov/bugzilla Email us directly –jhu@sandia.govjhu@sandia.gov –rstumin@sandia.govrstumin@sandia.gov –msala@sandia.govmsala@sandia.gov

19. Conclusions ML 3.0 release this May –Part of next Trilinos release Existing options –Uncoupled / MIS aggregation –Smoothers –SuperLU New options with ML 3.0 –Graph-based aggregation –“parameter list” setup –Amesos interface to direct solvers –Visualization, aggregate statistics Use ML with Trilinos! –ML as preconditioner is trivial –Rich set of external libraries