UTAM Reproducible Research Package and C++ Seismic Library Samuel Brown November 5, 2008

Outline

URRP Motivation Create a centralized, version-controlled, software release with regression tests, promoting code reuse by students and sponsors. Provide the ability to reproduce results from all UTAM reports in a uniform and comprehensible manner.

URRP Distribution URRP is available through a secure subversion repository. Reproducible scripts have an interface for downloading images and intermediate results through sftp.

Firewall? No svn, no sftp? Compressed repository snapshots for annual and midyear releases will be available on the UTAM website. Compressed images and intermediate results reside in a directory tree on the UTAM website, which mirrors the URRP reports directory.

Source Code/Compilation bash and csh scripts for environment setup. Top-level SConstruct file with autoconf functionality for C/C++/F90 programs. Matlab library.

Reproducibility Report directories contain: paper.tex – latex report run.py - python script other scripts, directories, etc.

run.py run.py: uniform interface for reproducing results python only, does not use scons imperative, not declarative generates and runs shell scripts interactively simple mechanisms for downloading data and controlling computation with sources and targets can interface with PBS

run.py run.py consists of a number of individual processes A ‘process’ is a call to the URRP python function process( ) 1 process for downloading images 1 process for compiling latex paper 1 or more processed for reproducing results or downloading intermediate results

A Simple Process Process( cmds=[’matlab –nosplash –nodisplay < xcorr.m’] )

A More Involved Process Process( cmds=[‘ ucsl_fdac par=mod1.par’], sources=[ ‘vp.rsf’, ‘recv_coord.txt’], targets=[ ‘csg1.su’], docmds=1, wdir=‘./csgs’, bdir=‘./batch’, sdir=‘./mod’, pbs=1, nodes=4, ppn=2, walltime=0:30:00 )

Outline

UCSL Motivation Provide library for rapid development of flexible, robust, high-performance research codes. Find an appropriate balance between imperative and object-oriented programming. Provide a high level of abstraction to enable performance and flexibility, ie task-specific file objects with optional MPI I/O.

Applications Development initially driven by: Flexible 2D/3D modeling/RTM. Ray tracing and interferometric imaging of earthquake data.

Forward Modeling – PML Problem: When implementing PMLs, there are up to 26 regions requiring a different combination of fields/damping. To get the best result, all valid regions should be implemented, with full ghost region communication. For simplicity and performance in the FD kernel, PML regions should reside in separate volumes. This greatly complicates domain decomposition and communications.

cart_decomp C++ domain decomposition object. Computes balanced domain decomposition distributed along any combination of axes using a boundary condition cost function. Uses MPI topologies. Builds arrays of subdomain volumes for each PE. FD application only has to ask for as many volume arrays as are necessary for a given implementation.

cart_halo C++ ghost region communication object. Each subdomain group requiring ghost region communication creates a halo object. For 2 nd order time acoustic modeling, the subdomain group would be a 2 x nsubdomain volume array. cart_halo handles all communication between local and remote subdomains with two functions: start_update(int tslice, int half) finish_update() Ability to overlap communication and computation with option to split subdomains along the z axis.

Flexible Implementations Source and receiver groups are also abstracted as objects. Implementing a new parallel modeling code is simply a matter of: Providing kernels for interior and boundary regions. Providing a time-stepping loop. Writing a small amount of initialization code.