Deflated Conjugate Gradient Method 16/11/2018 Deflated Conjugate Gradient Method for modelling Groundwater Flow in a Layered Grid Lennart Ros Deltares & TU Delft Utrecht July 3 2008: 10.30 www.deltares.com Thesis Committee: Prof. Dr. Ir. C. Vuik (TU Delft) Dr M. Genseberger (Deltares) Ir. J. Verkaik (Deltares) Dr H.M. Schuttelaars (TU Delft)

16/11/2018 Outline Deflated CG method 16/11/2018

Outline Introduction Deltares Subsurface, Geohydrology, Clay & Faults MODFLOW IBRAHYM & problem Equation, Discretization & Method A Simple Example (Matlab) Deflation Techniques Results for IBRAHYM Conclusions & Recommendations Questions Deflated CG method 16/11/2018

16/11/2018 Introduction Deflated CG method 16/11/2018

Introduction Deltares January 1st 2008 Deflated CG method 16/11/2018

Introduction Subsurface Subsurface is schematized in layers . Successive sand and clay (aquifers and aquitards) Assumption: Horizontal flow in aquifer Vertical flow in aquitard Deflated CG method 16/11/2018

Introduction Geohydrology The driving force for groundwater flow is the difference in height and pressure. To represent this difference we introduce the concept of hydraulic heads, h [L]. Deflated CG method 16/11/2018

Introduction Clay Medium Faults Very high resistance. Main property: Extreme low permeability (vertical) Medium Faults Vertical barriers inside aquifers. Main property: Extreme low permeability (horizontal). Deflated CG method 16/11/2018

Consequence: Large contrast in medium parameters Introduction Clay Very high resistance. Main property: Extreme low permeability (vertical) Consequence: Large contrast in medium parameters Medium Faults Vertical barriers inside aquifers. Main property: Extreme low permeability (horizontal). Deflated CG method 16/11/2018

Introduction MODFLOW: MODFLOW is a software package which calculates hydraulic heads. Developed by the U.S. Geological Survey. Open-source code: everyone can use and improve this program Rectangular grid and uses cell- centered variables. Quasi-3D model. Deflated CG method 16/11/2018

Introduction IBRAHYM: groundwater model developed for several waterboards in Limburg. uses at most 19 layers to model groundwater flow area. uses grid cells of 25 times 25 meter to get detailed information. a lot of clay and faults these cause model to suffer from bad convergence behavior of solver. Deflated CG method 16/11/2018

Equation, Discretization & Method 16/11/2018 Equation, Discretization & Method Deflated CG method 16/11/2018

Equation, Discretization & Method Governing Equation: Where: hydraulic conductivities along x,y, and z coordinate axes [LT-1], h head [L], W volumetric flux per unit volume representing sources and sinks of water [T-1], Ss specific storage of porous material [L-1], t Time [T] Deflated CG method 16/11/2018

Equation, Discretization & Method Finite Volume Discretization: Deflated CG method 16/11/2018

Equation, Discretization & Method Discretized Equation Using Finite Volume Method: Where: Deflated CG method 16/11/2018

Equation, Discretization & Method Solution Method in MODFLOW: stop criteria inner loop: or: maximal number of inner iteration is reached MODFLOW uses outer and inner iteration loops We look at inner iteration loop: solves a linear system of equations: preconditioner: iterative method: MODFLOW uses a: Modified Incomplete Cholesky Conjugate Gradient Method with: SOR Deflated CG method 16/11/2018

16/11/2018 A Simple Example Deflated CG method 16/11/2018

A Simple Example Simple Testcase: 2 Dimensional Problem 15 rows, 15 colums A contrast in the parameter on 1/3th of the domain Deflated CG method 16/11/2018

Observations for a simple testcase in Matlab: A Simple Example Observations for a simple testcase in Matlab: Assume A has eigenvalues: Preconditioning MODFLOW: Modified Incomplete Cholesky Preconditoning generally works Deflated CG method 16/11/2018

Observations for a simple testcase in Matlab: A Simple Example Observations for a simple testcase in Matlab: Deflated CG method 16/11/2018

Observations for a simple testcase in Matlab: A Simple Example Observations for a simple testcase in Matlab: Smallest eigenvalue: 0.00010283296716 Next eigenvalue: 0.04870854847951 Deflated CG method 16/11/2018

IMPROVE CONVERGENCE BEHAVIOUR OF THE IBRAHYM MODEL A Simple Example Observations for a simple testcase in Matlab: Due to the small eigenvalue we have a slow converging model. Want to get rid of this eigenvalue(s) GOAL: IMPROVE CONVERGENCE BEHAVIOUR OF THE IBRAHYM MODEL IDEA: USE DEFLATION BASED PRECONDITIONER Deflated CG method 16/11/2018

16/11/2018 Deflation Techniques Deflated CG method 16/11/2018

Basic Idea of Deflation: Deflation Techniques Basic Idea of Deflation: General linear system of equations: Now define: And instead we solve the deflated system: Deflated CG method 16/11/2018

PROBLEM: eigenvalues and eigenvectors are generally unknown Deflation Techniques Deflation using Eigenvectors: Assume that A has eigenvalues: and we choose the corresponding eigenvectors such that If we now define Then: PROBLEM: eigenvalues and eigenvectors are generally unknown Deflated CG method 16/11/2018

Alternative Deflation Techniques: Random Subdomain Deflation Deflation based on Physics: Use layers as boundary of domain (1 domain is 1 layer) Original domain Subdomains Deflated CG method 16/11/2018

Results for the simple problem: Deflation Techniques Results for the simple problem: Deflation using subdomain deflation 1 vector represents left part of domain 1 vector represents right part of domain The eigenvector corresponding to the smallest eigenvalue is in the span of these two vectors. Eigenvalues of and are almost the same, but the smallest is cancelled now. Deflated CG method 16/11/2018

Results for the simple problem: Deflation Techniques Results for the simple problem: Less iterates are needed Residuals go faster to zero when using deflation GOAL: IMPROVE CONVERGENCE BEHAVIOUR OF THE IBRAHYM MODEL or REDUCE NUMBER OF ITERATIONS Deflated CG method 16/11/2018

16/11/2018 Results for IBRAHYM Deflated CG method 16/11/2018

Results for IBRAHYM Process First: Subdomain deflation while storing matrix Z and AZ Problem: Memory limiting for large areas. Optimization concerning memory: Claypackages  Layer based deflation (nice structure of Z) Re-using vectors. Deflated CG method 16/11/2018

Results for IBRAHYM Process Problem: No gain of wall-clock times. Optimization concerning wall-clock times: ‘Storing’ AZ using one vector. Pointer-vector instead of IF-loops. Deflated CG method 16/11/2018

Results for IBRAHYM Results (small area) Claypackages  Layer based deflation For 3 small areas (7x7 km): 1.489.600 cells Area original code deflation code 1 (yellow) 62 53 2 (blue) 164 121 3 (orange) 287 168 Deflated CG method 16/11/2018

Results for IBRAHYM Deflated CG method 16/11/2018

stop criteria inner loop: Results for IBRAHYM Results (small area) REMEMBER: MODFLOW uses outer and inner iteration loops Concerning maximal number of inner iterations: Max. inner iteration Original code Deflation code 20 566 271 30 383 199 40 335 179 50 287 168 75 230 153 total number of iterations when varying the maximal number of inner iterations per inner loop stop criteria inner loop: less iterations needed also: less fluctuation in solution when varying maximal number of inner iterations Deflated CG method 16/11/2018

Layer based deflation also ‘solves’ for the faults. Results for IBRAHYM Results (small area) Concerning faults: Faults Original code Deflation code With 287 171 Without 254 168 Total number of iterations when we take and do not take faults into account Layer based deflation also ‘solves’ for the faults. Deflated CG method 16/11/2018

wall-clock time gained Results for IBRAHYM Results (bigger area) bigger area is 18 x 18 km = 9.849.600 cells Original code Deflation code Iterations 1082 757 Wall-clock time 1664.40 sec 1542.20 sec Error -0.31188E-02 0.49375E-03 less iterations needed wall-clock time gained error smaller Error = inflow – outflow also: less fluctuation in solution when varying maximal number of inner iterations Deflated CG method 16/11/2018

Conclusions & Recommendations 16/11/2018 Conclusions & Recommendations Deflated CG method 16/11/2018

Conclusions & Recommendations In general: deflation works for modelling groundwater flow. Less iterations are required. Deflation preconditioner makes solution more robust. Wall clock times can be gained, but depends strongly in code used. Recommendations: Deflation in horizontal direction. Code should be further optimised for Fortran: Minimizing memory, IF-statements, Using smart mappings. Cluster multiple layers in one subdomain. Deflated CG method 16/11/2018

16/11/2018 QUESTIONS? Deflated CG method 16/11/2018

Storing AZ Deflated CG method 16/11/2018

Storing AZ Deflated CG method 16/11/2018

Conclusions & Recommendations In general: deflation works for modelling groundwater flow. Less iterations are required. Deflation preconditioner makes solution more robust. Wall clock times can be gained, but depends strongly in code used. Recommendations: Deflation in horizontal direction. Code should be further optimised for Fortran: Minimizing memory, IF-statements, Using smart mappings. Cluster layers in one subdomain. Deflated CG method 16/11/2018