Page 1 Open-Source Software for Interfacing and Support of Large-scale Embedded Nonlinear Optimization Roscoe A. Bartlett Department of Optimization & Uncertainty Estimation Sandia National Laboratories INFORMS Annual Meeting, October 14, 2008 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL C

Page 2 Computational Sciences at Sandia : PDEs and More … Chemically reacting flows Climate modeling Combustion Compressible flows Computational biology Circuit modeling Inhomogeneous fluids Materials modeling MEMS modeling Seismic imaging Shock and multiphysics Structural dynamics Heat transfer Network modeling Large-Scale Parallel Distributed Memory and Implicit Iterative Solvers are Major Themes!

Page 3 Overview of Trilinos Provides a suite of numerical solvers to support predictive simulation for Sandia’s customers => Scope has expended to include discretizations methods, … Provides a decoupled and scalable development environment to allow for algorithmic research and production capabilities => “Packages” Mostly C++ with some C, Fortran, Python … Advanced object-oriented and generic C++ … Freely available under and open-source LGPL license... Current Status Current Release Trilinos 9.0 (September 2008) Next Release Trilinos 10.0? (March 2009?) Trilinos website

Page 4 An ANA is a numerical algorithm that can be expressed abstractly solely in terms of vectors, vector spaces, linear operators, and other abstractions built on top of these without general direct data access or any general assumptions about data locality Example : Linear Conjugate Gradients scalar product defined by vector space vector-vector operations linear operator applications Scalar operations Types of operationsTypes of objects What is an abstract numerical algorithm (ANA)? Linear Conjugate Gradient Algorithm Introducing Abstract Numerical Algorithms Key Points ANAs can be very mathematically sophisticated! ANAs can be extremely reusable! Flexibility needed to achieve high performance!

Page 5 Simulation-Constrained Optimization Methods Basic Steady-State Simulation-Constrained Optimization Problem: Decoupled approach (simulation constraints always satisfied): Optimization method never “sees” the state space! Optimization method deals with the (parallel) state space and the parameter space together! Requires special globalization methods to converge to a minimum! MOOCHO (Trilinos) Two broad approaches for solving optimization problems Embedded approach (converges optimality and feasibility together):

Page 6 Motivation for Embedded Optimization Large Scale Non-Linear Programming for PDE Constrained Optimization, van Bloemen Waanders, B., Bartlett, R., Long, K., Boggs, P., and Salinger, A. Sandia Technical Report SAND , October 2002 Increasing Levels of Coupling, Derivative, and Solve Capabilities Key Point For many/some optimization problems, embedded optimization methods can be much more computationally efficient and more robust than decoupled approaches But: It is hard to get our “foot in the door” with production codes It is hard to keep a “door stop” in place once we are in … Because … Decoupled Finite-Difference Varying levels of Embedded Invasiveness Decoupled Pattern Search Level 2 = Decoupled forward sens. Level 3 = Coupled forward sens. Level 4 = Decoupled adjoint sens. Level 5 = Coupled adjoint sens. Level 6 = Coupled full-Newton

Page 7 Challenges/Barriers to Embedded Optimization Embedded Algorithms R&D with Production APPs Better Algorithms R&D! Better Production APPs! Version Control, Build, Test (incompatible dev sources, environments, tools, lack of testing, …) APP + Trilinos Dev (Bartlett et. al.) Software Infrastructure (narrow forward solvers, inflexible implementation approaches, …) Thyra ModelEvaluator (Bartlett et. al.) Derivatives Support (smoothness, accuracy, parameter derivatives, … ) AD/Sacado (Phipps and Gay) Fleeting effort #1 Fleeting effort #2 SNL APP + Trilinos Dev (Bartlett et.al.) Thyra ModelEvaluator (Bartlett et.al.) AD/Sacado (Phipps et.al.) … We are now addressing these barriers in a fundamental way to provide the foundation for sustained embedded algorithms R&D

Page 8 Example: Numerous interactions exist between layers of abstract numerical algorithms (ANAs) in a transient optimization problem Linear Solver AztecOO, Amesos, Belos, ??? Operators and Vectors Epetra, Tpetra, PETSc, ??? Nonlinear Solver NOX, PETSc, ??? Nonlinear Optimizer MOOCHO, ??? Key Points Higher level algorithms, like optimization, require a lot of interoperability Interoperability and vertical integration must be “easy” or these configurations will not be achieved in practice What is needed to solve problem? Standard interfaces to break O(N 2 ) 1-to-1 couplings Application Charon, Aria, ??? Transient Solver Rythmos, SUNDIALS, ??? Algorithm Vertical Integration for Embedded Optimization Thyra is being developed to address interoperability of ANAs by defining interfaces for: · Linear operators/vectors · Preconditioners / Linear solvers · Nonlinear models · Nonlinear solvers · Transient solvers

Page 9 Overview of Nonlinear Model Evaluator Interface Motivation: An interface for nonlinear problems is needed that will support a variety of different types of problems Nonlinear equations (and senstitivities) Stability analysis and continuation Explicit ODEs (and sensitivities) DAEs and implicit ODEs (and sensitivities) Unconstrained optimization Constrained optimization Uncertainty quantification … as well as different combinations of problem types such as: Uncertainty in transient simulations Stability of an optimum under uncertainty of a transient problem Approach: Develop a single, scalable interface to address all of these problems (Some) Input arguments: State and differential state: Parameter sub-vectors: Time (differential): (Some) Output functions: State function: Auxiliary response functions: State/state derivative operator (LinearOpWithSolve): Key Points Input and output objects strongly typed Mathematical problem loosely typed Key Point The number of combinations of different problem types is large and trying to statically type all of the combinations is not realistic

Page 10 Some Nonlinear Problems Supported by the ModelEvaluator Nonlinear equations: Stability analysis: DAEs/Implicit ODEs: Explicit ODEs: DAE/Implicit ODE Forward Sensitivities: Unconstrained Optimization: Constrained Optimization: ODE Constrained Optimization: Explicit ODE Forward Sensitivities:

Page 11 Nonlinear Algorithms and Applications : Thyra & Model Evaluator! Trilinos and non-Trilinos Preconditioner and Linear Solver Capability NOX / LOCA MOOCHO XyceCharonAriaTramontoAleph … … Key Points Provide single interface from nonlinear ANAs to applications Provides for shared, uniform access to linear solver capabilities Once an application implements support for one ANA, support for other ANAs can be added incrementally Nonlinear ANA Solvers in Trilinos Sandia Applications Rythmos Model Evaluator

Page 12 Si interstitial (I) (+2,+1,0,–1,–2) Vacancy (V) (+2,+1,0,–1,–2) VV (+1,0,–1,–2) B I (+,0,–) C I (+,0,-) VP (0,–) VB (+,0) VO (0,–) B I B (0,–) B I O (+,0) BICBIC Annihilation Defect reactions QASPR Qualification of electronic devices in hostile environments PDE semiconductor device simulation Stockpile BJT

Page 13 Steady-State Parameter Estimation with Charon/MOOCHO Minimize Current model vs. target mismatch Subject to: Steady-state semiconductor defect physic FE model Solved current matching optimization problems to calibrate model parameters against target currents MOOCHO (Bartlett) optimization solver converges simulation model and optimality at same time –Faster and more robust than black-box optimization methods –More accurate solutions Successes –Very accurate inversion of currents and model parameters for contrived “inverse” problems Challenges –Extremely difficult nonlinear solver convergences on model convergence => Opportunities for algorithm research –Inability to match experimental data => May indicate incomplete FE model

Page 14 Embedded Transient Sensitivity Computation QASPR transient current sensitivities w.r.t. reaction parameters for an irradiated semiconductor device modeled with Charon –Embedded sensitivities with AD/Sacado (Phipps) & Rythmos –Finite differences (steplen=1e-2) (optimal steplen=1e-1) –Embedded sensitivities vs. finite diff. Much more accurate and robust! 10x faster for 40 parameters! Bartlett, Roscoe, Scott Collis, Todd Coffey, David Day, Mike Heroux, Rob Hoekstra, Russell Hooper, Roger Pawlowski, Eric Phipps, Denis Ridzal, Andy Salinger, Heidi Thornquist, and Jim Willenbring. ASC Vertical Integration Milestone. SAND , Sandia National Laboratories, 2007 [

Page 15 Summary Abstract numerical algorithm (ANA) approach results in highly flexible and efficient numerical algorithms Embedded optimization support must be well assimilated with forward solver algorithms and related nonlinear analysis algorithms Standard interfaces are needed to achieve vertical integration of numerical algorithms to support embedded optimization Thyra operator/vector, linear solver etc. interfaces form foundation for interoperability of ANAs to support embedded optimization Thyra ModelEvaluator provides foundation for application and algorithmic infrastructure to support embedded optimization and related simulated- constrained analysis methods Thyra, ModelEvaluator, and ANA principles demonstrated with semi- production semiconductor simulation code (Charon)

Page 16 The End