1. Innovative solutions for a safer, better world BUILDING STRONG ® Efficient Resolution of Complex Transport Phenomena Using Eulerian-Lagrangian Techniques Matthew Farthing and Hwai-Ping Cheng Branch Chief: Aaron Byrd FY12 Flood & Coastal Storm Damage Reduction IPR Wednesday Sept 12, 2012

2. Innovative solutions for a safer, better world BUILDING STRONG ® Motivation  Accurate transport resolution is critical for many of the scientific and engineering challenges facing the Corps. For example,  Evolution and cleanup of contaminant plumes  Sediment transport  Movement of fluid/fluid and fluid solid interfaces in multiphase flow  Existing models are limited in their ability to obtain accurate answers to many large-scale problems with available computational resources  Low-order Eulerian methods that are overly diffusive  Explicit, FV schemes with restricted time steps  Eulerian-Lagrangian (EL) methods that are difficult to extend to nonlinear problems

3. Innovative solutions for a safer, better world BUILDING STRONG ® Project Funding  FY 12: $125K (Flood and Coastal), $125K (NavSys)  $250K requested, $250K executed UNC MIPR $37K should be expended by Jan. 2013 Leveraged $40K from DOER for tech transfer  FY11: $245K, FY10: $171K from Civil Works Basic Research Program  Collaborators:  Ruth Cheng and Corey Trahan: pPT123 development  Tahirih Lackey: Tech transfer to PTM  Kevin Winters, Amanda Hines: CMB interface development  Stacy Howington, Chris Kees, modeling transport phenomena

4. Innovative solutions for a safer, better world BUILDING STRONG ® Completed Products  Farthing, Cheng et al., Accurate Particle Tracking for Large-Scale Unstructured Meshes, Paper CP-010, Army Science Conference (2010).  Cheng, Farthing et al., PT123: A Multi-Dimensional Particle Tracking Computer Program. ERDC/CHL Technical Report, TR-11-10 (2011). PT123: PT123 is a code for tracking particles in specified, transient velocity fields on unstructured meshes in 1,2, or 3 space dimensions. PT123 supports meshes with triangular and/or quadrilateral elements in 2d as well as hexahedral, tetrahedral, and/or prismatic elements in 3d.  Computational Model Builder Interface for PT123 (2011).  Gasda, Farthing et al., Adaptive split-operator methods for modeling transport phenomena in porous medium systems. Advances in Water Resources (2011).  Miller, Dawson, et al., Numerical Simulation of Water Resources Problems: Models, Methods, and Trends. Advances in Water Resources (2012).  Farthing, et al., The influence of porous media heterogeneity on NAPL dissolution fingering and upscaled mass transfer. WRR (2012).  Povich, Dawson, et al., Finite element Methods for Variable Density Flow and Solute Transport. In press, Computational Geosciences (2012).

5. Innovative solutions for a safer, better world BUILDING STRONG ® FY12 Activities  Technology transfer of PT123 numerics to PTM  Design of general particle tracking test suite  Ray-tracing based search, Courant number-based temporal adaption  PT123: added random-walk diffusion with heterogeneous material properties  pPT123: completed initial design and preliminary tests for parallel particle tracking  Trained members of Philadelphia and Kansas City Districts on use of PT123 for subsurface transport problems  Presented PT123 at Computational Methods in Water Resources conference, and published journal articles in AWR, WRR, Computational Geosciences  Remaining milestones/tasks  Parallel particle tracking code 70% complete  Journal article on particle tracking 60% complete.  Journal article on ELLAM 30% complete  Secured PETTT year-long pre-planned project for HPC particle tracking

6. Innovative solutions for a safer, better world BUILDING STRONG ® Eulerian-Lagrangian Numerics  Extension of Farthing et al, ELLAM to nonlinear, two-phase flow  Extension of selective lumping, FCT work to Eulerian-Lagrangian methods for unstructured meshes and multidimensional problems CFL=16.5, nn-41CFL=8.5, nn=41 Gaussian IC CFL 8.5, nn=41 CFL 16.5, nn=41 Slug IC Shock char. struct. Adjoint char. struct. 2p Flow CFL 16.5, slump

7. Innovative solutions for a safer, better world BUILDING STRONG ® Random Walk Particle Tracking (1/2)  Fokker-Planck Equation:  Particle displacement associated with Fokker-Planck Equation: which is equivalent to solving  1-D advective-diffusive transport with instantaneous point source:  Point source at x = 100 and time = 0  Use 2 nd -order RK  Use composite EBE-NEBE (EN) tracking  Uniform but transient background velocity: u = 2, 0 and -2 at time = 0, 100, and 200  Particle density  concentration f = particle density distribution deterministic part: stochastic part:

8. Innovative solutions for a safer, better world BUILDING STRONG ® Random Walk Particle Tracking (2/2)  2-D advective-diffusive transport with continuous point source:  Particles exit via open boundary Particles move along closed boundary  3-D advective-diffusive transport with anisotropy

9. Innovative solutions for a safer, better world BUILDING STRONG ® PT123-PTM Tech Transfer (1/2)  Worked with PTM group in  defining common set of particle tracking benchmarks, including solid body rotation, channel flow, isotropic diffusion, etc.  comparing PTM and PT123 performance using defined benchmarks.  identifying ray-trace based search as first priority, then (simple) adaptive time integration, to improve PTM performance in FY12.  Developed a ray-tracing Fortran 90 library routines.  Tested the incorporation of the ray-tracing library Into PTM with two test examples provided by PTM group: OC2012_RARG Model (in CMS format) and Cleveland Harbor Model (in ADCIRC format).  Provided source and a write-up detailing the contents and verification of the ray-tracing library as well as its linkage with PTM. Ray tracing from P to Q

10. Innovative solutions for a safer, better world BUILDING STRONG ® PT123-PTM Tech Transfer (2/2) PTM simulation statistics for Cleveland Harbor Model Particle Distribution in Harbor Area (consistent results with new scheme) Cleveland Harbor Model Mesh CPU Time Comparison The total CPU time reduced by 37.8 times

11. Innovative solutions for a safer, better world BUILDING STRONG ® Parallel Particle Tracking Framework  PT123 computational kernel for ODE solvers.  P2P communication model.  Partitioning based on mesh only, particles only, or combination of mesh and particles.  Framework has a plug-in for particle behavior/interaction. Parallel framework Swirl Test Example: Time = 0 Time = 4 Time = 8 Time = 8 (zoom-in)

12. Innovative solutions for a safer, better world BUILDING STRONG ® Issues & Help Required  Execution issues (include impacts): delays in approval process lead to MIPR not fully expended this FY.  Technical challenges (include impacts): Efficient methods for nonlinear multi-dimensional problems still a (surmountable) challenge. Focus on technology transfer and PT123 code development meant less time and money to focus on nonlinear research issues.  Anticipated cost growth (include impacts): No scheduled funding for next FY.  Opportunities: We have engaged UT via PETTT project to explore particle tracking methods on HPC platforms. We could leverage this effort much more with some internal funding for ERDC personnel.

13. Innovative solutions for a safer, better world BUILDING STRONG ® Good News Stories We Can Tout  Publications & Presentations  Farthing, Fowler, et al., The effect of model resolution on a set of community problems for optimal design in water resources. Advances in Water Resources (2011).  Miller, Dawson, et al., Numerical Simulation of Water Resources Problems: Models, Methods, and Trends. Advances in Water Resources (2012).  Farthing, Seyedabbasi, et al., The influence of porous media heterogeneity on NAPL dissolution fingering and upscaled mass transfer. WRR (2012).  Povich, Dawson, et al., Finite Element Methods for Variable Density Flow and Solute Transport. In press, Computational Geosciences (2012).  Lackey, Cheng, et al., Lagrangian Determination Of the Fate of Sediment Due to Dredging. PIANC Dreging 2012. October 22--25, San Diego, CA (2012).  Lackey, Cheng, et al., Advanced Methods in Lagrangian Far Field Fate Modeling: Part 1. In review, CHL Technical Report (2012).  Code development and Tech Transfer:  A ray-tracing Fortran library routines extracted from PT123 was developed and has been successfully linked with PTM to improve computational efficiency when particles encounter the land boundary.  A random-walk particle tracking using the NEBE technique or the composite EBE-NEBE technique has been developed to mimic diffusion/dispersion accurately in homogeneous media.

14. Innovative solutions for a safer, better world BUILDING STRONG ® Lessons Learned  Balancing basic R&D efforts with technology transfer and product development is a challenge.  The change in emphasis due to end of Civil Works Basic R&D program meant we focused more on tech transfer and maturing of PT123 in last phase of project.  It takes time to build communication and collaboration across research groups. Going about tech transfer the right way (create benchmark problems, profile code, and identifying key bottlenecks) is critical but takes time and effort.

15. Innovative solutions for a safer, better world BUILDING STRONG ® Technology Transfer Plan  FY12: transitioned some core PT123 tracking numerics to PTM as Fortran library routines.  Next phase would be to focus on random walk algorithms (no funding yet identified though)  Success in FY12 measured in terms of efficiency gains for PTM on previous District applications. Future metrics could also include new capabilities within PTM  Expect use of PT123 as a standalone application for subsurface transport by Philadelphia district as part of HHD project  Incorporation of Eulerian-Lagrangian/PT techniques in other ERDC transport codes

16. Innovative solutions for a safer, better world BUILDING STRONG ® FY13 Proposed Activities  Continue PT improvements and collaboration with PTM group  Develop efficient and effective random walk tracking algorithms to mimic diffusion/dispersion in heterogeneous media  Develop schemes for 2+1 D tracking (unstructured horizontal + structured vertical) with both z and sigma grids.  Develop tracking algorithms to account for mesh refinement/unrefinement.  Develop tracking algorithms to account for wetting/drying.  Incorporate Eulerian-Lagrangian approximations for linear problems in ERDC transport codes.  Improve pPT123 as parallel agent/particle tracking engine.