Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL Objectives A.G. Salinger, I. Kalashnikova, M. Perego, R.S. Tuminaro, M.S. Eldred and J.D. Jakeman, Sandia National Laboratories S. Price, M. Hoffman, Los Alamos National Laboratories Ongoing Work Development of the Albany/FELIX Land Ice Dycore using Software Components SAND 2014-xxxxP Component-Based Strategy Component-based approach enables rapid development of new production codes embedded with transformational capabilities Element Level Fill Material Models Sensitivities Field Manager Discretization Library Remeshing UQ Solver Nonlinear Solver Time Integration Optimization Objective Function Local Fill Mesh Database Mesh Tools I/O Management Input File Parser Utilities UQ (sampling) Parameter Studies Mesh I/O Optimization Geometry Database Discretizations Derivative Tools Adjoints UQ / PCE Propagation Constraints Error Estimates Continuation Constrained Solves Sensitivity Analysis Stability Analysis V&V, Calibration Parameter List Verification Visualization PostProcessing Adaptivity Model Reduction Memory Management System Models MultiPhysics Coupling OUU, Reliability Communicators Partitioning Load Balancing Analysis Tools (black-box) Physics Fill Composite Physics Data Structures Direct Solvers Linear Algebra Architecture- Dependent Kernels Preconditioners Iterative Solvers Eigen Solver System UQ Analysis Tools (embedded) Matrix Partitioning Inline Meshing MMS Source Terms Grid Transfers Quality Improvement Mesh Database Solution Database Derivatives Regression Testing Bug Tracking Version Control Software Quality Porting Performance Testing Code Coverage Mailing Lists Release Process Unit Testing Web Pages Build System Backups Verification Tests DOF map Multi-Core Accelerators Linear Programming Graph Algorithms Data-Centric Algs SVDs Map-Reduce Network Models D ycore Interfaces and Meshes UQ: Bayesian Calibration Convergence & Scalability Sandia’s components effort includes ~100 interoperable libraries Solution Verification using manufactured solutions Defining a UQ workflow for stochastic inversion of Basal sliding coefficients: 1. Model Reduction (KLE) 2. PCE Emulator 3. MCMC Calibration using Emulator Albany/FELIX Ice Sheet Dycore Develop: robust and scalable unstructured-grid finite element ice sheet code:  Stand-alone steady-state model for initialization and calibration  Dynamic model when linked to MPAS-LI or CISM for advection  Future land ice component of DOE-ACME earth system model Support: DOE climate missions, such as providing Sea Level Rise predictions Leverage: software and expertise from SciDAC Institutes (FASTMath, QUEST, SUPER) and hardware from DOE Leadership Class Facilities Funding: “PISCEES” SciDAC Application Partnership (DOE’s BER + ASCR divisions) PIs: S. Price and E. Ng; collaboration with ORNL, LANL, LBNL, UT, FSU, SC, MIT, and NCAR  Mature dynamic evolution capability under MPAS  Perform deterministic and stochastic initialization runs  Improve coupling to full earth system model  Finish conversion to performance-portable kernels We acknowledge the contributions of our PISCEES collaborators, including B. Lipscomb, K. Evans, P. Worley, M. Norman, M. Gunzberger, and C. Jackson, and our many Trilinos/Dakota collaborators, including E. Phipps and L. Swiler Finite Element Discretization (Hex, Tet) Parallel, Unstructured Grid with Partitioning Automatic Differentiation for Jacobians Globalized Newton’s Method Nonlinear Solves Preconditioned Krylov Iterative Solvers Performance-Portable Kernels (in progress) Software tools: git / cmake / ctest / jenkins  =  =  =  = cores 334K DOFs (8km GIS, 5 layers) cores 1.12B DOFs (0.5km GIS, 80 layers) Robust Nonlinear Solves using Homotopy Continuation 3D Mesh convergence study for GIS model gives theoretical 2 nd -Order rate How many vertical layers do you need? Convergence study for GIS 1km mesh: Scalability results over 4 mesh bisections: 8484 The Albany/FELIX solver can be driven by a CISM or MPAS-LI interface: CISM MPAS CISMMPAS-LI T=70 yr T= 0 yr We are beginning to do dynamic runs: CISM (Fortran) Thickness evolution, temperature solve, coupling to ESM simple_glide Albany/FELIX (C++) velocity solve MPAS/Land Ice (Fortran) Thickness evlolution, temperature solve, coupling to ESM LandIce_mode l C++/Fortran interface, mesh conversion Structured rectangles Extruded to Hexs Unstructured polygons Dual mesh of triangles Extruded to Tets Regional Refinement: # vertical layers/# cores # dofsTotal Time - Setup (sec) Solution Average Error 5/ M e-2 10/ M e-3 20/ M e-3 40/ M e-4 80/ M e-4 160/ M e-5