Solvers Made Easy (to use and use together) Thyra, Stratimikos, Handles and More... Roscoe A. Bartlett Department of Optimization & Uncertainty Estimation Sandia National Laboratories Trilinos Users Group Meeting, November 7 th, 2006 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL85000.

Outline Overview of Thyra “Why should anyone care about Thyra” Stratimikos (unified access to linear solvers & algebraic preconditioners) Wrapping it up

Outline Overview of Thyra “Why should anyone care about Thyra” Stratimikos (unified access to linear solvers & algebraic preconditioners) Wrapping it up

· Linear Problems: · Linear equations: · Eigen problems: · Preconditioners: · Nonlinear Problems: · Nonlinear equations: · Stability analysis: · Transient Nonlinear Problems: · DAEs/ODEs: · Optimization Problems: · Unconstrained: · Constrained: Trilinos Packages Belos Anasazi NOX LOCA MOOCHO Aristos Categories of Abstract Problems and Abstract Algorithms Meros Rythmos

An ANA is a numerical algorithm that can be expressed abstractly solely in terms of vectors, vector spaces, linear operators, and other abstractions built on top of these without general direct data access or any general assumptions about data locality Example : Linear Conjugate Gradients scalar product defined by vector space vector-vector operations linear operator applications Scalar operations Types of operationsTypes of objects What is an abstract numerical algorithm (ANA)? Linear Conjugate Gradient Algorithm Introducing Abstract Numerical Algorithms Key Points ANAs can be very mathematically sophisticated! ANAs can be extremely reusable! Flexibility needed to achieve high performance!

Trilinos Strategic Goals Scalable Solvers: As problem size and processor counts increase, the cost of the solver will remain a nearly fixed percentage of the total solution time. Hardened Solvers: Never fail unless problem essentially unsolvable, in which case we diagnose and inform the user why the problem fails and provide a reliable measure of error. Full Vertical Coverage: Provide leading edge capabilities from basic linear algebra to transient and optimization solvers. Grand Universal Interoperability: All Trilinos packages will be interoperable, so that any combination of solver packages that makes sense algorithmically will be possible within Trilinos. Universal Solver RAS: Trilinos will be: –Reliable: Leading edge hardened, scalable solutions for each of these applications –Available: Integrated into every major application at Sandia –Serviceable: Easy to maintain and upgrade within the application environment. Courtesy of Mike Heroux, Trilinos Project Leader Thyra is being developed to address this issue

Numerous interactions exist between layered abstract numerical algorithms (ANAs) in a transient optimization problem Iterative Linear Solver AztecOO, Amesos, Belos, ??? Operators and Vectors Epetra, Tpetra, PETSc, ??? Nonlinear Solver NOX, PETSc, ??? Nonlinear Optimizer MOOCHO, ??? Key Points Higher level algorithms, like optimization, require a lot of interoperability Interoperability and layering must be “easy” or these configurations will not be achieved in practice What is needed to solve problem? Standard interfaces to break O(N 2 ) 1-to-1 couplings –Operators/vectors –Linear Solvers –Nonlinear solvers –Transient solvers –etc. Application Charon, Aria, ??? Transient Solver Rythmos, CVODES, IDAS, ??? Interoperability is Especially Important to Optimization Thyra is being developed to address interoperability of ANAs

Examples of ANA Interoperability and Layering : Rythmos Time Stepper Implicit Backward Euler method Nonlinear equations Newton’s method (e.g. NOX) Linear equations Preconditioned GMRES Operator and Preconditioner applications Preconditioners can be defined in many different ways Matrix-free or Matrix?

Fundamental Thyra ANA Operator/Vector Interfaces The Key to success! Reduction/Transformation Operators Supports all needed element-wise vector operations Data/parallel independence Optimal performance R. A. Bartlett, B. G. van Bloemen Waanders and M. A. Heroux. Vector Reduction/Transformation Operators, ACM TOMS, March 2004 A Few Quick Facts about Thyra Interfaces All interfaces are expressed as abstract C++ base classes (i.e. object-oriented) All interfaces are templated on a Scalar data (i.e. generic)

Preconditioners and Preconditioner Factories Create preconditioner prec with preconditioner operators P L and/or P R such that P L A, or AP R, or P L AP R is “easier” to solve than unpreconditioned A. PreconditionerFactoryBase createPrec() : PreconditionerBase initializePrec( in fwdOpSrc, inout prec ) PreconditionerFactoryBase : Creates and initializes PrecondtionerBase objects > prec Allows unlimited creation/reuse of preconditioner objects Supports reuse of factorization structures Adapters currently available for Ifpack and ML New Stratimikos package provides a singe parameter-driver wrapper for all of these PreconditionerBase getLeftPrecOp() : LinearOpBase getRightPrecOp() : LinearOpBase getUnspecifiedPrecOp() : LinearOpBase

Linear Operator With Solve and Factories LinearOpBase LinearOpWithSolveBase solve( in B, inout X, … ) Appropriate for both direct and iterative solvers Supports multiple simultaneous solutions as multi-vectors Allows targeting of different solution criteria to different RHSs Supports a “default” solve LinearOpWithSolveFactoryBase createOp() : LinearOpWithSolveBase initializeOp( in fwdOpSrc, inout Op ) initializePreconditionedOp( in fwdOpSrc, in prec, inout Op) LinearOpWithSolveBase : Combines a linear operator and a linear solver LinearOpWithSolveFactoryBase : Uses LinearOpBase objects to initialize LOWSB objects LinearOpWithSolveBase > Allows unlimited creation/reuse of LinearOpWithSolveBase objects Supports reuse of factorizations/preconditioners Supports client-created external preconditioners (which are ignored by direct solvers) Appropriate for both direct and iterative solvers Concrete adaptors for Amesos, AztecOO, and Belos (not released) are available New Stratimikos package provides a single parameter-driven wrapper to all of these!

Introducing Stratimikos Stratimikos created Greek words "stratigiki“ (strategy) and "grammikos“ (linear) Defines class Thyra::DefaultRealLinearSolverBuilder: Provides common access to: Linear Solvers: Amesos, AztecOO, Belos (dev only), … Preconditioners: Ifpack, ML, … Reads in options through a parameter list (read from XML?) Accepts any linear system objects that provide Epetra_Operator / Epetra_RowMatrix view of the matrix SPMD vector views for the RHS and LHS (e.g. Epetra_[Multi]Vector objects) Provides uniform access to linear solver options that can be leveraged across multiple applications and algorithms Stay tuned for more detailed Stratimikos example to come later …

“Client Support” Software for Developing ANAs 1. C++ wrapper functions for prewritten RTOpT subclasses e.g. norm_1(x), assign(&y,x), scale(alpha,&y), … 2. Composite objects: 1. Product vectors, multi-vectors and vector spaces e.g. x = [ v1; v2; v3; … ; vn ]; 2. Decorator and composite subclasses for building implicitly composed linear operators 1. Zero operator e.g. M = 0 2. Identity operator e.g. M = I 3. Implicit scaled adjoint/transpose: e.g. M = alpha*adjoint(A); 4. Additive linear operators: e.g. M = A + B + C + D; 5. Multiplicative linear operators: e.g. M = A * B * C * D; 6. Block linear linear operators: e.g. M = [ A, B ; C, D ]; 3. Handle classes with operator overloading e.g. y = alpha*trans(A)*x + gamma*B*z + beta*y

Math Notation for CG Math Notation vs. C++ Handle Code : Example, Linear CG // Initialization … Vector r = b - A*x; for( int iter = 0; iter <= maxNumIters; ++iter ) { Scalar rho = inner(r,r); Scalar beta = (iter!=0 ? rho/rho_old : one); if(iter!=0) p = r + beta*p; else copyInto(r,p); q = A*p; Scalar gamma = inner(p,q); Scalar alpha = rho/gamma; x += Scalar(+alpha)*p; r += Scalar(-alpha)*q; // Check convergence … rho_old = rho; } C++ Handles Implementing CG Matlab-like notation for linear algebra! Template meta-program methods used to reduce operator-overloading overhead and avoid creation of temps. Works with any linear operator and vector implementation (e.g. Epetra, PETSc, etc.) Works in any computing configuration, i.e. serial, SPMD, client/server etc.! Works with any Scalar type (i.e. float, double, complex, extended precision, etc.) that has a traits class Still some more work to be done: Better elimination of some temporaries, support for multi-vectors See silliestCgSolve(…) and Meros for other examples …

History Behind Thyra Hilbert Class Library (HCL) Symes et. al. 1995? Rice University Vector spaces, vectors, and operators AbstractLinAlgPack (in rSQP++) Bartlett, 2001 Sandia National Labs RTOp Sundance HCL Long et. al., 2000 Sandia National Labs Smart pointers, Handles … Trilinos Solver Framework (TSF) Long et. al Sandia National Labs Renaming of Sundance HCL TSFCore Bartlett et. al., 2003 Sandia National Labs ANA Operator/Vector Interfaces ??? Thyra Thyra developers et. al., 2005 Sandia National Labs Renaming of TSFCore Operator/Vector clases, LinearOpWithSolve, ModelEvaluator

Finding Your Way Around Thyra Browse Thyra Doxygen documenation: –Go to the website –Click on “Thyra” (on the right side) –Click on “Release 7.0” or “Development” (on the left side) –Browse the different doxygen documentation collections separately + You will see Thyra in small chunks! - It is hard to find things you don’t know where they exist! –Or, click on “Browse all of Thyra as a single doxygen collection” and click on “single doxygen collection” + You will find everything in Thyra! - You will find everything in Thyra! This will not search outside of Thyra pacakge! -Click “Other Software Related to Thyra” and click links to other collection -Use the “Search For” box in a Doxygen documentation collection (on software!): -Within any doxygen collection html page, at the fill in “Search For” box and hit Enter + You will search only that doxygen documentation collection! - You will search only that doxygen documentation collection! -Use the Google Search option on the main website: –Go to the website –Click on “Search” (on the left side) –Click “Google Search” (+ Search all packges, - Will search dev and all releases!) Do a “grep –r ‘What I want to find’.” in your local Trilinos/packages dir.

Outline Overview of Thyra “Why should anyone care about Thyra” Stratimikos (unified access to linear solvers & algebraic preconditioners) Wrapping it up

Why Should Anyone Care about Thyra? We will look at answering this question and provide insights by asking some more questions: Do you just want different solvers to work together without a lot of work on your part? Do you need to access a solver that accepts input through Thyra? Do you want to implement large-scale numerical algorithms? Are you responsible for making sure that many different solvers all work together in a seamless and automatic way?

Do you want different solvers to work together? Do you just want different solvers to work together without a lot of work on your part? If your answer is yes then: You should care about Thyra since it makes this possible like has never been achieved before for as many different types of numerical algorithms and software! What you need to know about Thyra?: Almost nothing! (perhaps not even the name Thyra) What you do not need to know about Thyra? Almost everything about Thyra!

Do you want different solvers to work together? … … … … … Time Stepper Nonlinear Solver Lin. Solver Precond. Linear Solver In the long term, most users will form connections between different solvers by selecting options from in a parameter list! Example: Time Stepper => Nonlinear Solver => Linear Solver => Preconditioner End users may not even see the name “Thyra”!

Do you need to access a solver that accepts Thyra objects? What you need to know about Thyra? How to wrap your objects as Thyra objects Support already provided for your objects (e.g. Epetra, Tpetra, etc.)? => Easy! Support not already provided? => Harder! What you do not need to know about Thyra? How interoperability works What utilities there are for developing algorithms (e.g. implicit ops, handles etc.) … Do you need to be able to access a numerical solver that only accepts objects through Thyra (e.g. Stratimikos)? If your answer is yes then: You have no choice but to care about Thyra since that is the only way to access the solver capability! Stay tuned for more detailed Stratimikos example to come later …

Do you want to write large-scale numerical algorithms? Are you interested in doing algorithmic research and writing numerical algorithms that lend themselves to the ANA approach and … If your answer is yes then: You should care about Thyra since it makes it relatively easy (but perhaps not by MATLAB standards) to achieve these goals and get efficient and maintainable code that will stand up over time. maintain a concise mathematical description/notation? will run in parallel out of the box? are completely independent of computer platform details or MPI? are interoperable with other numerical algorithms? provides a straightforward path to a production capability? have access to a wide range of large-scale production problems? What you need to know about Thyra? “Client support” software for developing ANAs (e.g. implicit ops, handles etc.) What you do not need to know about Thyra? How interoperability works How to define concrete Thyra objects …

Are you responsible for ensuring different solvers work together? Are you responsible for making sure that different solver packages all work together in a seamless and automatic way in many different configurations? If your answer is yes then: you should care about Thyra since it makes this possible like no other middleware has ever done before with the range of algorithms that can and have been addressed using Thyra. What you need to know about Thyra? How interoperability works How to define concrete Thyra objects What you do not need to know about Thyra? “Client support” software for developing ANAs (e.g. implicit ops, handles etc.) …

Why Should Anyone Care about Thyra? Do you just want solver A, solver B, solver C etc. to work together without a lot of work on your part? Do you need to access a solver that accepts input through Thyra? Do you want to implement large-scale numerical algorithms? Are you responsible for making sure that solver A, solver B, solver C etc. all work together in a seamless and automatic way? Note: Very few people (perhaps just me and a few other people) will answer yes to all four of these question! Take-home message: What you need to know about Thyra depends on how you interact with Thyra (if at all)!

Outline Overview of Thyra “Why should anyone care about Thyra” Stratimikos (unified access to linear solvers & algebraic preconditioners) Wrapping it up

Accessing a Solver through Thyra? Stratimikos Example simple_stratimikos_example.cpp Demonstrates how to solve single-RHS linear systems expressed as Epetra objects Typical steps for “using a solver that accepts Thyra objects”: A)Setup the environment for the program => Read in command-line options B)Create the input objects using your data structures (independent of Thyra) => Read in linear system objects in terms of Epetra objects C)Wrap your objects as Thyra objects (The “glue”) => Use Epetra-to-Thyra wrapper functions D)Call the solver through Thyra interfaces (Thyra-specific) => Create the LinearOpWithSolve[Factory]Base objects and solve … E)Post process the solution (if needed) using your objects (independent of Thyra) => Post process solution and check in terms of Epetra objects (Not Thyra!!!)

A) Setup the environment for the program // // A) Program setup code // // Read options from command-line // std::string matrixFile = ""; … Thyra::DefaultRealLinearSolverBuilder linearSolverBuilder; CommandLineProcessor clp(false); // Don't throw exceptions // Set up command-line options for the linear solver that will be used! linearSolverBuilder.setupCLP(&clp); clp.setOption( "matrix-file", &matrixFile,"Defines the matrix and perhaps the RHS and LHS for a linear system to be solved." ); … CommandLineProcessor::EParseCommandLineReturn parse_return = clp.parse(argc,argv); if( parse_return != CommandLineProcessor::PARSE_SUCCESSFUL ) return parse_return; See the file simple_stratimikos_example.cpp for more details …

B) Create the input objects using your data structures // // B) Epetra-specific code that sets up the linear system to be solved // // While the below code reads in the Epetra objects from a file, you can // setup the Epetra objects any way you would like. Note that this next // set of code as nothing to do with Thyra at all, and it should not. // *out << "\nReading linear system in Epetra format from the file \'"<<matrixFile<<"\'...\n"; #ifdef HAVE_MPI Epetra_MpiComm comm(MPI_COMM_WORLD); #else Epetra_SerialComm comm; #endif RefCountPtr epetra_A; RefCountPtr epetra_x, epetra_b; EpetraExt::readEpetraLinearSystem( matrixFile, comm, &epetra_A, NULL, &epetra_x, &epetra_b ); if(!epetra_b.get()) { *out << "\nThe RHS b was not read in so generate a new random vector...\n"; epetra_b = rcp(new Epetra_Vector(epetra_A->OperatorRangeMap())); epetra_b->Random(); } if(!epetra_x.get()) { *out << "\nThe LHS x was not read in so generate a new zero vector...\n"; epetra_x = rcp(new Epetra_Vector(epetra_A->OperatorDomainMap())); epetra_x->PutScalar(0.0); // Initial guess is critical! } Create the Epetra objects any way that you would like …

C) Wrap your objects as Thyra objects (The “glue”) // // C) The "Glue" code that takes Epetra objects and wraps them as Thyra // objects // // This next set of code wraps the Epetra objects that define the linear // system to be solved as Thyra objects so that they can be passed to the // linear solver. // // Create RCPs that will be used to hold the Thyra wrappers RefCountPtr > A; RefCountPtr > x; RefCountPtr > b; // Create the Thyra wrappers if(1) { // Create an RCP directly to the EpetraLinearOp so that we can access the // right range and domains spaces to use to create the wrappers for the // vector objects. RefCountPtr _A = Thyra::create_LinearOp(epetra_A); // Create Thyra wrappers for the vector objects that will automatically // update the Epetra objects. b = Thyra::create_Vector(epetra_b,_A->spmdRange()); x = Thyra::create_Vector(epetra_x,_A->spmdDomain()); // Set the RCP to the base linear operator interface A = _A; } Turning Epetra objects into Thyra objects is easy! Epetra-to-Thyra wrapper functions are found in Thyra_EpetraThyraWrappers.hpp

D) Call the solver through Thyra interfaces // // D) Thyra-specific code for solving the linear system // // Note that this code has no mention of any concrete implementation and // therefore can be used in any use case. // // Reading in the solver parameters from the parameters file and/or from // the command line. This was setup by the command-line options // set by the setupCLP(...) function above. linearSolverBuilder.readParameters(out.get()); // Create a linear solver factory given information read from the // parameter list. RefCountPtr > lowsFactory = linearSolverBuilder.createLinearSolveStrategy(""); // Setup output stream and the verbosity level lowsFactory->setOStream(out); lowsFactory->setVerbLevel(Teuchos::VERB_LOW); // Create a linear solver based on the forward operator A RefCountPtr > lows = Thyra::linearOpWithSolve(*lowsFactory,A); // Solve the linear system (note: the initial guess in 'x' is critical) Thyra::SolveStatus status = Thyra::solve(*lows,Thyra::NOTRANS,*b,&*x); *out << "\nSolve status:\n" << status; // Write the linear solver parameters after they were read linearSolverBuilder.writeParamsFile(*lowsFactory); This part of the code has nothing to do with Epetra! This is really only a trivial interaction with Thyra!

// // E) Post process the solution and check the error // // Note that the below code is based only on the Epetra objects themselves // and does not in any way depend or interact with any Thyra-based // objects. The point is that most users of Thyra can largely gloss over // the fact that Thyra is really being used for anything. // // Wipe out the Thyra wrapper for x to guarantee that the solution will be // written back to epetra_x! x = Teuchos::null; *out << "\nSolution ||epetra_x||2 = " << epetraNorm2(*epetra_x) << "\n"; *out << "\nTesting the solution error ||b-A*x||/||b|| computed through the Epetra objects...\n"; // r = b - A*x Epetra_Vector epetra_r(*epetra_b); if(1) { Epetra_Vector epetra_A_x(epetra_A->OperatorRangeMap()); epetra_A->Apply(*epetra_x,epetra_A_x); epetra_r.Update(-1.0,epetra_A_x,1.0); } const double nrm_r = epetraNorm2(epetra_r), nrm_b = epetraNorm2(*epetra_b), rel_err = ( nrm_r / nrm_b ); const bool passed = (rel_err <= tol); *out << "||b-A*x||/||b|| = " << nrm_r << "/" << nrm_b << " = " << rel_err << " < tol = " << tol << " ? " << ( passed ? "passed" : "failed" ) << "\n"; E) Post process the solution (if needed) using your objects

Stratimikos Parameter List and Sublists Linear Solvers Preconditioners See Doxygen documentation for Thyra::DefaultRealLinearSolverBuilder! Sublists passed on to package code! Top level parameters Every parameter and sublist not in red is handled by Thyra code and is fully validated!

Automatically Generated Parameter List Documentation “Human readable” automatically generated documentation for Stratimikos Linear Solver Type : string = Amesos # Determines the type of linear solver that will be used. # The parameters for each solver type are specified in the sublist "Linear Solver Types" # Valid values: "Belos", "Amesos", "AztecOO" Preconditioner Type : string = ML # Determines the type of preconditioner that will be used. # This option is only meaningful for linear solvers that accept preconditioner factory objects! # The parameters for each preconditioner are specified in the sublist "Preconditioner Types" # Valid values: "None", "Ifpack", "ML" Linear Solver Types -> AztecOO -> Output Every RHS : bool = 0 # Determines if output is created for each individual RHS (true or 1) or if output # is just created for an entire set of RHSs (false or 0). Forward Solve -> Max Iterations : int = 400 # The maximum number of iterations the AztecOO solver is allowed to perform. Tolerance : double = 1e-06 # The tolerence used in the convergence check (see the convergence test # in the sublist "AztecOO Settings") AztecOO Settings -> Aztec Solver : string = GMRES # Type of linear solver algorithm to use. # Valid values: "CG", "GMRES", "CGS", "TFQMR", "BiCGStab", "LU" Convergence Test : string = r0 # The convergence test to use for terminating the iterative solver. # Valid values: "r0", "rhs", "Anorm", "no scaling", "sol"... See Doxygen documentation for Thyra::DefaultRealLinearSolverBuilder! Development version of Trilinos Only!

Open Issues for Stratimikos Thyra::DefaultRealLinearSolverBuilder Not every parameter can be read from an XML file currently. Common “presolve” operations (e.g. prescaling and singleton filter removal) are not supported yet. Only double precision is supported currently. AztecOO, Amesos, Ifpack, and ML all currently require Epetra views of the objects. Future needs Other scalar types (i.e. float, std::complex, std::complex …) Specialized/blocked preconditioners (e.g. Meros …) “Smart” solver managers (e.g. pick the best method automatically) …

Outline Overview of Thyra “Why should anyone care about Thyra” Stratimikos (unified access to linear solvers & algebraic preconditioners) Wrapping it up

Summary Thyra interfaces provide minimal but efficient connectivity between ANAs and linear algebra implementations and applications Thyra is the critical standard for interoperability between ANAs in Trilinos Thyra can be used in Serial/SMP, SPMD, client/server and master/slave Thyra provides a growing set of optional utilities for ANA development and subclass implementation support Thyra support for nonlinear ANAs (i.e. the model evaluator) is being developed as well as general support for linear solvers Thyra interfaces and adapters are provided for preconditioner factories and linear solver factories (Stratimikos) Thyra adapters are available for Epetra, Amesos, AztecOO, Belos, Anasazi, Rythmos, and MOOCHO with others on the way (e.g. NOX, …) Python/Thyra wrappers are on the way as well (Bill Spotz)! Trilinos website