Managed by UT-Battelle for the Department of Energy Challenges in Simulating Subsurface Flow and Reactive Transport using Ultrascale Computers Richard Tran Mills Computational Earth Sciences Group Computer Science and Mathematics Division Oak Ridge National Laboratory

2Managed by UT-Battelle for the Department of Energy Presentation_name Introduction  Funded by SciDAC-II project, “Modeling Multiscale-Multiphase- Multicomponent Subsurface Reactive Flows using Advanced Computing”, involving several institutions: –LANL: Peter Lichtner (PI), Chuan Lu, Bobby Philip, David Moulton –ORNL: Richard Mills –ANL: Barry Smith –PNNL: Glenn Hammond, Steve Yabusaki –U. Illinois: Al Valocchi  Project goals: –Develop a next-generation code (PFLOTRAN) for simulation of multiscale, multiphase, multicomponent flow and reactive transport in porous media. –Apply it to field-scale studies of  Geologic CO2 sequestration,  Radionuclide migration at Hanford site, Nevada Test Site,  Others…

3Managed by UT-Battelle for the Department of Energy Presentation_name  At the 300 area, U(VI) plumes continue to exceed drinking standards.  Calculations predicted cleanup by natural attenuation years ago!  Due to long in-ground residence times, U(VI) is present in complex, microscopic inter-grain fractures, secondary grain coatings, and micro-porous aggregates. (Zachara et al., 2005).  Models assuming constant K d (ratio of sorbed mass to mass in solution) do not account for slow release of U(VI) from sediment grain interiors through mineral dissolution and diffusion along tortuous pathways.  In fact, the K d approach implies behavior opposite to observations!  We must accurately incorporate millimeter scale effects over a domain measuring approximately 2000 x 1200 x 50 meters! MoMotivating example -- Hanford 300 Area

4Managed by UT-Battelle for the Department of Energy Presentation_name Fundamental challenge:  Need to capture millimeter-scale (or smaller) processes within kilometer scale domains! (Similar variations in time scales.)  Discretizing 2km x 1 km x 500 m domain onto cubic millimeter grid means 10^18 computational nodes!  Address the problem via –Multi-continuum (“sub-grid”) models  Multiplies total degrees of freedom in primary continuum by number of nodes in sub-continuum –Massively parallel computing  Continuing development of PFLOTRAN code –Adaptive mesh refinement  Allows front tracking  Introduce multi-continuum models only where needed

5Managed by UT-Battelle for the Department of Energy Presentation_name Modeling multiscale proceses  Represent system through multiple interacting continua with a single primary continuum coupled to sub-grid scale continua.  Associate sub-grid scale model with node in primary continuum –1D computational domain –Multiple sub-grid models can be associated w/ primary continuum nodes –Degrees of freedom: N x N K x N DCM x N c

6Managed by UT-Battelle for the Department of Energy Presentation_name PFLOTRAN governing equations Mass Conservation: Flow Equations Energy Conservation Equation Multicomponent Reactive Transport Equations Mineral Mass Transfer Equation Total Concentration Total Solute Flux

7Managed by UT-Battelle for the Department of Energy Presentation_name PFLOTRAN governing equations Mass Conservation: Flow Equations Energy Conservation Equation Multicomponent Reactive Transport Equations Mineral Mass Transfer Equation Total Concentration Total Solute Flux Darcy’s law (homogenized momentum eq.)

8Managed by UT-Battelle for the Department of Energy Presentation_name Integrated finite-volume discretization Form of governing equation: Discretized residual equation: (Inexact) Newton iteration: Integrated finite-volume discretization

9Managed by UT-Battelle for the Department of Energy Presentation_name PFLOTRAN architecture  PFLOTRAN designed from the ground up for parallel scalability.  Built on top of PETSc, which provides –Management of parallel data structures, –Parallel solvers and preconditioners, –Efficient parallel construction of Jacobians and residuals  We provide –Initialization, time-stepping, equations of state –Functions to form residuals (and, optionally, Jacobians) on a local patch (PETSc routines handle patch formation for us)

10Managed by UT-Battelle for the Department of Energy Presentation_name PFLOTRAN strong scaling  25 million DoF density driven flow problem

11Managed by UT-Battelle for the Department of Energy Presentation_name PFLOTRAN strong scaling  25 million DoF (256 x 128 x 256 grid)

12Managed by UT-Battelle for the Department of Energy Presentation_name PFLOTRAN strong scaling  Dot products (all-reduces) become limiting factor  Keep in mind: Only 6144 unknowns per processor core at 4096

13Managed by UT-Battelle for the Department of Energy Presentation_name Adaptive mesh refinement (AMR)  Incorporating AMR via the SAMRAI package from LLNL.  AMR introduces local fine resolution only in regions where needed.  Significant reduction in memory and computational costs for simulating complex physical processes exhibiting localized fine scale features.  AMR provides front tracking capability in the primary grid that can range from centimeter to tens of meters.  Sub-grid scale models can be introduced in regions of significant activity and not at every node within the 3D domain.  It is not necessary to include the sub-grid model equations in the primary continuum Jacobian even though these equations are solved in a fully coupled manner.

14Managed by UT-Battelle for the Department of Energy Presentation_name Upscaling  Governing equations depend on averages of highly variable properties (e.g., permeability) averaged over a sampling window (REV).  Upscaling and ARM go hand-in-hand: as the grid is refined/coarsened, material properties such as permeability must be calculated at the new scale in a self-consistent manner. Above: A fine-scale realization (128 x 128) of a random permeability field, followed by successively upscaled fields (N x N, N = 32, 16, 4, 1) obtained with Multigrid Homogenization (Moulton et al., 1998)

15Managed by UT-Battelle for the Department of Energy Presentation_name Upscaling  Coarse-Scale Anisotropy: permeability must, in general, be considered as a tensor at larger scales even if it is a scalar (i.e., isotropic) at the finest scale.  A single multi-dimensional average is inadequate for modeling flow (MacLachlan and Moulton, 2006)  Upscaling that captures full-tensor permeability includes multigrid homogenization, and asymptotic theory for periodic media.  Theory is limited to periodic two-scale media (well separated scales)  Upscaling reactions poses a significant challenge as well. In some aspects of this work volume averaging will suffice, while in others new multiscale models will be required.  Uniform flow from left to right governed by harmonic mean.  Uniform flow from bottom to top governed by arithmetic mean.  Suggests a diagonal permability tensor; HOWEVER, if stripes not aligned with coordinate axes, equivalent permeability must be described by a full tensor.

16Managed by UT-Battelle for the Department of Energy Presentation_name Laundry list of challenges Unique to application domain:  Upscaling  Improved discretization schemes –Needed for full tensor formulation on unstructured grids Shared with many other applications:  Unstructured mesh management -- using PETSc Sieves  Load balancing for AMR, unstructured meshes  Nonlinear solvers –Phase transitions! Problems w/ variable switching schemes  Linear solvers –Block Krylov methods for multi-core?  Preconditioners –Physics-based –Multigrid (must be aware of upscaling issues)

17Managed by UT-Battelle for the Department of Energy Presentation_name Additional slides….

18Managed by UT-Battelle for the Department of Energy Presentation_name Geologic CO2 sequestration  Capture CO2 from power production plants, and inject it as supercritical liquid in abandoned oil wells, saline aquifers, etc.  Must be able to predict long-term fate: –Slow leakage defeats the point. –Fast leakage could kill people!  Many associated phenomena are very poorly understood. LeJean Hardin and Jamie Payne, ORNL Review, v.33.3.

19Managed by UT-Battelle for the Department of Energy Presentation_name Grid effects Plots of CO2 concentration dissolved in a brine at different times and depths following injection of supercritical CO2 at depth. No flow boundaries are imposed at top and bottom. Strong grid effects appear with variable grid spacing when modeling density instabilities.

20Managed by UT-Battelle for the Department of Energy Presentation_name Grid effects Density instabilities occur only along coordinate axes. Hole in middle may be due to grid effects.