An efficient parallel particle tracker For advection-diffusion simulations In heterogeneous porous media Euro-Par 2007 IRISA - Rennes August 2007

Jocelyne Erhel INRIA Rennes Jean-Raynald de Dreuzy Geosciences Rennes Anthony Beaudoin LMPG, Le Havre Partly funded by Grid’5000 french project

From Barlebo et al. (2004) Dispersion Flow Injection of tracer Tracer evolution during one year (Made, Mississippi) Heterogeneous permeability Physical context: groundwater flow Flow governed by the heterogeneous permeability Solute transport by advection and dispersion

Physical context: groundwater flow Spatial heterogeneity Stochastic models of flow and solute transport -random velocity field -random solute transfer time and dispersivity Lack of observationsPorous geological media Solute dispersionHorizontal velocity Permeability field

Longitudinal dispersion Transversal dispersion Macro-dispersion analysis

HYDROLAB: parallel software for hydrogeoloy Numerical methods Physical models Porous Media Solvers PDE solvers ODE solvers Linear solvers Particle tracker Utilitaries Input / Output Visualization Results database Parameters database Parallel and grid tools Geometry PARALLEL-BASED SCIENTIFIC PLATFORM HYDROLAB Open source libraries Boost, FFTW, CGal, MPI, Hypre, Sundials, OpenGL, Xerces-C UQ methods Monte-Carlo Fracture Networks Fractured- Porous Media Object-oriented and modular with C++ Parallel algorithms with MPI Efficient numerical libraries

Parallel computing facilities ©INRIA/Photo Jim Wallace Numerical model Clusters at Irisa Grid’5000 project Funded by French Government and Brittany council

Solute transport Fixed head and C=0 Fixed head and  C/  n=0 Nul flux and  C/  n = 0 injection Advection-dispersion equations Boundary conditions Initial condition Flow equations

injection Solute transport : particle tracker Many independent particles Bilinear interpolation for V Homogeneous molecular diffusion Stochastic differential equation First-order explicit scheme D=D m I, U=mean(V), Pe= U / D m

particle tracker : convergence analysis N p = 2000 is a good trade-off between efficiency and convergence Longitudinal dispersion Transversal dispersion Pure advectionAdvection-diffusion with Pe=100

Not very sensitive to heterogeneity More efficient in pure advection case Linear complexity with the number of cells in each direction particle tracker : impact of diffusion and heterogeneity

Parallel particle tracker injection Domain decomposition Velocity V already distributed in parallel flow computations Each processor tracks the particles inside its subdomain

One processor owns one subdomain If there are particles at interfaces then Track them inside the subdomain Until they reach an interface The particles arriving at the right boundary go out of the domain Send particles (non blocking) To all neighbors (empty buffer if no particle) Recv particles (blocking) From all neighbors (empty buffer if no particle)

safeguards When is it finished ? No deadlock ? No starvation ? Buffers emptied before a new send ? Local counter Np Global counter N Global Sum of all local counters of particles The End when N=0 No Deadlock and no starvation

Workload Injection by packets of independent particles Injection at time 0 in only a few processors Load balancing ? Performances ?

Injection by packets Pipelined Packets: balanced workload

Scalability issues Good speed-up and good scalability

Large scale computations Grid size 4096*4096 Good speed-up in any configuration

Current work and perspectives Current work 3D heterogeneous porous media Grid computing and Monte-Carlo simulations Future work Dispersion tensor Reactive transport UQ methods