Update on the CCA Groundwater Simulation Framework: the BOCCA Experience Bruce Palmer, Yilin Fang, Vidhya Gurumoorthi, James Fort, Tim Scheibe Computational Sciences and Mathematics Division Pacific Northwest National Laboratory Richland, WA 99352

Progress on Smoothed Particle Hydrodynamics (SPH) Framework Implementation of component to model transport of a passive contaminant Refinement of existing components to enhance scalability to an modularity Ported to Cray XT4 platform (Franklin) Large runs of 7-14 million SPH particles Incorporated SPH applications into SALSSA workflow framework Moved entire SPH framework into BOCCA

3 SPH Framework

4

Progress on Subsurface Transport Over Multiple Phases (STOMP) Framework Developed an interface for a grid component using an unstructured grid syntax Implemented grid component using GA for a structured grid Modified STOMP kernel to use grid component

STOMP Framework STOMP componentGrid component Input Grid Port PhysicsSolvers Time Integrator Output GA Data Mapping GA MPI Chemistry

7 STOMP/CCA Simulations Simulation of transport of contaminant through an array of monitoring wells at the Hanford IFRC site, using the component version of STOMP

Current Status A complete framework for performing Smoothed Particle Hydrodynamics simulations exists Validation of SPH hydrodynamic flows in 3D is underway using the SALSSA workflow environment Initial simulations of tracer transport in 3D porous media are in progress Ported to Chinook and Franklin computers Development of additional chemistry modules will be starting soon The underlying grid in STOMP has been separated out as a standalone component Grid component supports an unstructured syntax All references to a structured grid in STOMP have been eliminated except the output Collaboration underway with ITAPS project to develop an unstructured grid component Work is underway to split out solvers and output into separate components

Platforms Used Chinook Running SPH with HPMPI and OpenMPI. MVAPICH seems to have problems with dangling ports. This is a problem for SPH but not for STOMP Franklin Running with static builds (kudos to Jim Kohl)

Using the BOCCA Developer Environment./$SPH_BOCCA_HOME create_sphbocca env_gcc_3.4.csh env_gwbocca.csh sph-orig-impls/ sph-bocca/

create_sphbocca #========================================== # Creating ports: #========================================== bocca create port rdcomp.ReadInputPort --import-sidl= $WORKDIR/sph-orig-impls/ports/sidl/rdcomp.ReadInputPort.sidl #========================================== # Creating Components: #========================================== bocca create component drivers.SPHDriver --import-sidl= $WORKDIR/sph-orig-impls/components/sidl/drivers.SPHDriver.sidl --import-impl= "drivers.SPHDriver:$WORKDIR/sph-orig- impls/components/drivers.SPHDriver“ --go=GoPort --uses=rdcomp.ReadInputPort:ReadInputPort --uses=rdprtcl.ReadPrtclPort:ReadPrtclPort --uses=wrtprtcl.ExportXYZPort:ExportXYZPort --uses=lgrng.LagrangePort:LagrangePort --uses=stepper.StepPort:StepPort --uses=dtmgr.DataPort:DataPort --language=f90

create_sphbocca (cont) #=============================================== # Copy the top level make files #=============================================== cd $WORKDIR/sph-orig-impls cp make.vars.user $SPH_HOME/ #=============================================== # Copy files in individual component directories #=============================================== cd $WORKDIR/sph-orig-impls/components/drivers.SPHDriver/ cp make.vars.user common.fh $SPH_HOME/components/drivers.SPHDriver/

Observations on BOCCA Having a repository for files (sph-orig-impls) and a separate working directory (sph-bocca) is confusing and error prone We could REALLY use a “duplicate” capability inside BOCCA. This would take an existing component, make a copy of it and give it a new name. Not having this function is a major impediment to getting new developers on board. The contractor build was not producing compilable F90 files after importing. Don’t know what the status of this bug is.

Observations on BOCCA (cont) The only way to build components is to go into the components directory and hit “make”. You cannot go into the directory for individual components and build them one by one nor can you go into the component directory and go “make individual.component” and get a single component to build. Having the make system compile every component, or at least check every component for changes, takes a long time if you have many components We could use the ability to create static builds as a standard feature We are having trouble changing the home machine that the sph- origs-impls directory is located on using the contractor build

Miscellaneous Issues It would be useful to be able to call component methods from inside the component using the SIDL interface Need mechanisms for provenance capture Version of components Version of CCA and configuration information Components utilized in application and their linkages

Acknowledgements Funding for this project was provided by DOE’s Office of Advanced Scientific Computing Research under the Scientic Discovery through Advanced Computing program Computer time on Chinook supplied by the Environmental and Molecular Sciences Laboratory at Pacific Northwest National Laboratory through its Science Theme program Computer time on Franklin supplied by the National Energy Research Scientific Computing Center