1. What is SeisFlows? -Provides a complete, customizable waveform inversion workflow -So far, has been used for 3D production runs with up to 10 9 model parameters -Has run on clusters managed by Princeton University, Department of Defense, Chevron Corp, Total S.A. and other institutions -At the same time, can be used for very small 2D inversions on a laptop or desktop -Currently being applied to earthquake tomography, oil and gas exploration, and general nonlinear optimization research problems -Developers: Ryan Modrak, Matthieu Lefebvre, growing GitHub user community

2. Theory Adjoint solver f † : D → M g(m) = f † (f(m)) Misfit function χ(m)=|| f(m) – d || Solver f: M → D Earth model m M Seismic data d D Gradient g(m) = ∂ m χ(m) Data Model Gradient We use spectral element method to numerically solve elastic wave PDE SPECFEM2D SPECFEM3D SPECFEM3D_GLOBE

3. How to implement an inversion? Given a model m and data d, an inversion is a nonlinear optimization procedure which iteratively improves the model m and reduces the misfit || f(m) – d ||. Having implemented the solver f, one could, in principle, carry out an inversion just through a call to a function minimization routine from a standard numerical library. In practice, there are so many complications along the way that such a simple approach is not really feasible.

4. At the highest level, inversion is a nonlinear optimization loop 1. Compute gradient g(m) = ∂ m χ(m) 2. Compute search direction p ≈ H –1 g 3. Determine step length determine α by minimizing ϕ (α) = f(m + α p) 4. Update model m i+1 = m i + α p

5. Structure of a gradient computation Solver parallelized over both model region (MPI) and events (embarrasingly parallel) Data preprocessing parallelized over traces (embarrasingly parallel) Adjoint solver parallelized over both model region (MPI) and events (embarrasingly parallel) Gradient postprocessing parallelized over model region (MPI)

6. Survey Not at all Somewhat Very familiar Familiarity with regional/global adjoint tomography literature1 2 3 4 5 Familiarity with exploration geophysics literature1 2 3 4 5 Familiarity with nonlinear optimization algorithms1 2 3 4 5 Familiarity with Unix1 2 3 4 5 Familiarity with Python scientific computing tools1 2 3 4 5 Familiarity with LSF/PBS/SLURM/etc.1 2 3 4 5 Familiarity with SPECFEM code/ development practices1 2 3 4 5

7. 1. SeisFlows Documentation http://seisflows.readthedocs.org/en/latest/manual/contents.html 2. SeisFlows Source Code http://github.com/PrincetonUniversity/seisflows 3. SeisFlows Example http://seisflows.readthedocs.org/en/latest/intructions_remote.html 4. Presentation Slides http://tigress-web.princeton.edu/~rmodrak/slides/SeisFlows.pptx Useful Links

8. SeisFlows example: 2D checkerboard inversion Included in download - SeisFlows parameter files - SPECFEM2D Par_file, interfaces, stations, and sources files - Target model in form readable by SPECFEM2D - Homogenous starting model in form readable by SPECFEM2D Based on: Tape et al. 2007, Finite frequency tomography using adjoint methods, GJI Target model

9. 1. Open terminal: (mac)Applications > Utilities > Terminal (windows)Programs > cygwin 2. Log onto cluster: ssh -Y guest##@mcmillan-vis.princeton.edu If prompted, type yes to add new.ssh entry Enter password when prompted 3. Load required dependencies: module load anaconda/2.5.0 4. Now try your hand at the 2D checkerboard inversion: http://seisflows.readthedocs.org/en/latest/instructions_remote.html Compile SPECFEM2D with gfortran i.e../configure FC=gfortran 5. Immediately before "run checkerboard test in parallel", use the following SLURM command to open an interactive job on one of the compute nodes: salloc --nodes=1 --ntasks=6 Then, in parameters.py add a new parameter NPROCMAX=6 Instructions