1. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. Andy Salinger, Eric Phipps Computer Science Research Institute, Sandia National Laboratories Albuquerque, NM, USA Tipping Points in Complex Flows - Numerical Methods for Bifurcation Analysis of Large-Scale Systems from 31 Oct 2011 through 4 Nov 2011 General-Purpose Software for Large-Scale Bifurcation Analysis

2. Trilinos: Algorithms and Enabling Technologies for Large-Scale Applications Two-level design: –Self-contained packages (50+) –Leveraged common tools. Version Control Build System Test Harness Nonlinear, Transient & Optimization Solvers Linear & Eigen Solvers Geometry, Meshing & Load Balancing Framework, Tools & Interfaces Discretizations Scalable Linear Algebra http://trilinos.sandia.gov

3. ObjectivePackage(s) Linear algebra objectsEpetra, Tpetra Krylov solversAztecOO, Belos, Komplex ILU-type preconditionersAztecOO, IFPACK, ShyLU Multilevel preconditionersML, CLAPS Eigenvalue problemsAnasazi Block preconditionersTeko Direct sparse linear solversAmesos (MUMPS, SuperLU, UMFPack, …) Load Balancing, Graph AlgsZoltan, Isorropia Continuation/BifurcationLOCA Nonlinear system solverNOX Time Integrators/DAEsRythmos OptimizationMoocho, Aristos, Dakota (via TriKota) Automatic DiffferentiationSacado Parameter List, Timers, MemoryTeuchos Uncertainty QuantificationStokhos, Dakota (via Trikota) Abstract interfacesThyra, EpetraExt Node Kernels on New ArchitecturesKokkos (Cuda, Threads, OpenMP) Trilinos Package Summary

4. LOCA Provides Analysis Capabilities to Large-Scale Applications Pseudo-Arclength Continuation Hopf Bifurcation location and continuation (turning point, pitchfork, and Hopf) Linear eigen-analysis through Anasazi (Thornquist & Lehoucq) Periodic orbit tracking (experimental) Multi-parameter continuation through Multifario (Henderson) Pitchfork

5. Why Do We Need Stability Analysis Tools for Large-Scale Applications Several powerful continuation and bifurcation analysis tools are available: –AUTO (Doedel et al) –CONTENT (Kuznetsov et al) –MATCONT (Govaerts et al) –PyDSTool (Guckenheimer et al) –… Large-scale applications have specific requirements –Massively parallel distributed memory architectures –Complicated parallel data structures and sparse matrices –Application-tuned linear algebra –Limited derivative capabilities Tools and algorithms are needed that –Do not change matrix sparsity or increase memory requirements –Agnostic to linear algebra and architecture –Can be incorporated into existing simulation codes (i.e., libraries)

6. Basic Defining Equations in LOCA Turning Point Pitchfork Hopf Pseudo-Arclength ODE/DAE Linearization

7. Shift and Invert Generalized Cayley Transformation Bifurcations Discovered Through Eigenvalue Analysis and Spectral Transformations Eigenvalue problem Eigenvalues/vectors approximated via Block Krylov-Schur Iterations (Anasazi – Thornquist & Lehoucq) Analogies to time integration can be used to pick transformation parameters (Lehoucq and Salinger, 2001, Burroughs et al, 2004).

8. NOX: Object-Oriented Nonlinear Solver in Trilinos: Pawlowski et al Concrete Implementation Layer Linear Algebra User Interface Residual Jacobian Solver Layer Methods Line Search Trust Region Searches Directions Linear Algebra Application Interface Abstract Layer EpetraLAPACK User DefinedThyra

9. LOCA Built on and around NOX Concrete Implementation Layer Linear Algebra User Interface Residual Jacobian Solver Layer Methods Line Search Trust Region Searches Directions Linear Algebra Application Interface Abstract Layer EpetraLAPACK User DefinedThyra Stepper Layer 1.Step 2.Solve 3.Analyze 4.Predict 5.Stop? Continuation Bifurcation Augmented Equations Layers Update parameters Mass matrix Mix-and-match between Continuation methods Predictor modules Step-size control modules Bifurcation modules Nonlinear solvers Linear solvers/preconditioners Eigensolvers Spectral transformations

10. Block Elimination Algorithm for Turning Point (fold) Tracking Uses 4 Solves Turning Point Bifurcation Full Newton Algorithm Block Elimination Algorithm

11. Solve 5 bordered systems of equations using QR approach Then Modified Turning Point Bordering Algorithm

12. Given and, let then There are constants such that Standard formulation: Note for Newton’s method: 3 linear solves per Newton iteration (5 for modified bordering)! For symmetric problems reduces to 2 solves. Minimally Augmented Turning Point Formulation

13. MPSalsa, Charon, Albany codes Incompressible Navier-Stokes Heat and Mass Transfer, Reactions, variable properties Unstructured Finite Element Galerkin/Least-Squares: Q1Q1 Analytic Jacobian matrix in distributed sparse storage Compute with AD Fully Coupled Newton Method GMRES with ILUT or MultI-Level Preconditioners Flow Calculations Performed with Sandia CFD codes and Trilinos solvers

14. Frank-Kamenetskii Explosion Model (~230K hex elements, ~1.1M unknowns, 128 cores) Scenario: Continuous Stirred Tank Reactor Exothermic Cehmical Reaction Cooling at Walls  The stirring breaks! ?Will natural convection prevent explosion? Arc-length Continuation

15. Frank-Kamenetskii Explosion Model Turning Point Location ILU(k) fill factor: 1 ILU(k) overlap: 2 Max Krylov space: 1000 MS Bordering Minimally augmented

16. Frank-Kamenetskii Explosion Model Turning Point Continuation MethodContinuation Steps Failed Steps Nonlinear Iterations Linear Solves Linear Iterations Total Time (hrs) Moore-Spence Mod. Bordering 495290201247296811.0 Min. Augmented3842148101540276.9

17. “Analysis Beyond Simulation” LOCA

18. Natural Convection Instability in 8x1 and 8x1x1 Cavities

19. Stability Analysis of Impinging Jets Pawlowski, Salinger, Shadid, Mountziaris (2005) Region II Region III Region I H H P

20. Rayleigh-Benard in 5x5x1 cavity with Bifurcation Tracking Codimension 2 Bifurcation near Pr=0.0434, Ra=2106 Eigenvectors at Hopf Hopf Pitchfork

21. Hydromagnetic Rayleigh-Bernard Problem Pawlowski, Shadid Parameters: Q ~ B 0 2 (Chandresekhar number) Ra (Rayleigh number) Buoyancy driven instability initiates flow at high Ra numbers. Increased values of Q delay the onset of flow. Domain: 1x20 Ra (fixed Q) No flow Recirculations B0B0 g

22. Extended MHD Model in Residual Form Involution: Resistive, Extended MHD Equations

23. Hydro-Magnetic Rayleigh-Bernard Stability: Direct Determination of Linear Stability and Nonlinear Equilibrium Solutions (Steady State Solves) 2 Direct-to-steady-state solves at a given Q Arnoldi method using Cayley transform to determine approximation to 2 eigenvalues with largest real part Simple linear interpolation to estimate Critical Ra* Temp. Vx Vy By Bx Leading Eigenvector at Bifurcation Point, Ra = 1945.78, Q=10

24. Q=10 Q=0 Design (Two-Parameter) Diagram Vx Ra Q Q No Flow Buoyancy Driven Flow “No flow” does not equal “no-structure” – pressure and magnetic fields must adjust/balance to maintain equilibrium. LOCA can perform continuation of bifurcation

25. Critical Mode is different for various Q values Analytic solution is on an infinite domain with two bounding surfaces (top and bottom) Multiple modes exist, mostly differentiated by number of cells/wavelength. Therefore tracking the same eigenmode does not give the stability curve!!! Periodic BCs will not fix this issue. Mode: 20 Cells: Q=100, Ra=4017 Mode: 26 Cells: Q=100, Ra=3757 Q Ra Leading mode is 20 cells Leading mode is 26 cells 2000 3000 4000

26. Scaling studies ~20x CoresFine Mesh Level 0 Unkns. Intermed. Level 1 Unkns. Intermed. Level 2 Unkns. Coarse Level 3 Unkns. Newton Iters. Avg. No. Linear Its. / Newton Total Sim. Time* (min.) 24,0001.05 billion23.3M.5M11.2K188633

27. LOCA has impacted several application areas without flow Phase Transitions in a Confined Fluid [Frink] Super-Conductivity Transitions in Ginzburg-Landau [Schlomer, Vanroose] TeraHz Resonance in Quantum Tunneling Diode [Kelley] Pattern Formation in Swift- Hohenberg Eqs [Avitabile, Sanstede]

28. How do I attribute the successes that LOCA has had? 1.Algorithmic research in large-scale bifurcations 2.Science demonstrations 3.LOCA is hooked up to Trilinos linear solvers What broader lesson is there? 10% 15% 75% Build software in independent-yet-interoperable components

29. Nonlinear Solver Time Integration Optimization Continuation Constrained Solves Sensitivity Analysis Stability Analysis Analysis Tools Data Structures Direct Solvers Linear Algebra Preconditioners Iterative Solvers Eigen Solver Matrix Partitioning Derivatives Derivative Tools Sensitivities Build Software in Independent-yet- Interoperable Components Continuation, Bifurcation, Stability Analysis

30. Build Software in Independent-yet- Interoperable Components Nonlinear Solver Time Integration Optimization Continuation Constrained Solves Sensitivity Analysis Stability Analysis Analysis Tools Data Structures Direct Solvers Linear Algebra Preconditioners Iterative Solvers Eigen Solver Matrix Partitioning Derivatives Derivative Tools Sensitivities Transient sensitivity analysis

31. Build Software in Independent-yet- Interoperable Components Initial 4-Param Bound 4-Param Free Nonlinear Solver Time Integration Optimization Continuation Constrained Solves Sensitivity Analysis Stability Analysis Analysis Tools Data Structures Direct Solvers Linear Algebra Preconditioners Iterative Solvers Eigen Solver Matrix Partitioning Derivatives Derivative Tools Sensitivities CVD Reactor Optimization

32. Element Level Fill Material Models Sensitivities Field Manager Discretization Library Remeshing UQ Solver Nonlinear Solver Time Integration Optimization Objective Function Local Fill Mesh Database Mesh Tools I/O Management Input File Parser Utilities UQ (non-invasive) Parameter Studies Solution Control Mesh I/O Optimization Geometry Database Discretizations Derivative Tools Adjoints UQ / PCE Propagation Constraints Error Estimates Continuation Constrained Solves Sensitivity Analysis Stability Analysis V&V, Calibration Parameter List Feature Extraction Embedded Verification Visualization PostProcessing Data Reduction Adaptivity Model Reduction Memory Management System Models MultiPhysics Coupling OUU, Reliability Computational Steering Communicators MultiCore Parallelization Tools Partitioning Load Balancing Analysis Tools (black-box) Physics Fill Composite Physics Data Structures Direct Solvers Linear Algebra Architecture- Dependent Kernels Preconditioners Iterative Solvers Eigen Solver System UQ Analysis Tools (embedded) Matrix Partitioning Inline Meshing MMS Source Terms Grid Transfers Mesh Quality Mesh Database Solution Database Runtime Compiler Derivatives Regression Testing Bug Tracking Version Control Software Quality Porting Performance Testing Code Coverage Mailing Lists Release Process Unit Testing Web Pages Build System Backups “Agile Components” Neighbor Search / Sort Data Structures Particle Code Tools

33. Field Manager PDE Assembly is Templated for AD, PCE Discretization Application Nonlinear Model Nonlinear Transient Stochastic Galerkin Optimization Continuation Solvers w/ Sensitivities Optimization UQ Analysis Tools Iterative Block Iterative Direct Linear Solvers / Preconditioners Domain Decomp MultiLevel Mesh Database Exodus File Hand-Coded: Problem Discretization PDE TERMS Main() ManyCore Node Kernels Mulit-Core Accelerators Inline Mesher Application Linear Solve Eigensolve Bifurcation Albany Code: Demonstrating Component-based Code Design Quality Improvement Load Balancing SchurComp

34. Applications in Albany are born with Transformational Analysis Capabilities LCM: Platform for R&D in mechanics: Load Stepping, AD of material models QCAD: Quantum dot design tool. Optimization of gate voltages 2-Param OptimumInitial Mesh Std Deviation ThermoElectrostatics:  Shape Optimization with Embedded UQ

35. Summary LOCA and Trilinos provide powerful simulation and analysis capabilities –Continuation, bifurcation, and linear stability analysis –Scalable linear algebra –Optimization, time integration, automatic differentiation, uncertainty quantification, discretization, … Missing Capabilities (formerly future work) –More generic algorithms for bordered matrix solves Much is hardwired to our Epetra format –Periodic Orbit tracking beyond initial attempt –Automated initial guess generation for null vectors –Better documentation, examples, error checking, etc. Current Passion –Component-based code design with Embedded Analysis in Mind from the beginning