EPOS use case: High-resolution seismic tomography of Italy Giuseppe Fiameni with the collaboration of Alberto Michelini, Federica Magnoni, Emanuele Casarotti

Outline Objective Motivation EPOS’s use case Requirements Actual status Proposal

Objective High resolution tomographic velocity model of Italy (i.e., invert the recorded seismic waveform data of earthquakes in Italy and adjacent regions for the 3D, laterally heterogeneous velocity structure of Italy.)

Motivation Big scientific challenge: waveform tomography of te 3D seismic velocity structure of the Italian region up to frequencies of 0.5 Hz (i.e., maximum spatial resolution ~2 Km) has never been feasible before Very relevant implications for modeling at high resolution strong ground shaking (large earthquakes) scenarios with evident seismic engineering applications The use case workflow conjugates data storage retrieval, staging, HPC modeling and stage-out of large volumes of simulated data, detailed 3D velocity model and adjoint kernels

EPOS Workflow Observed Data Forward Modeling Inverse Visualization Input Parameters

Forward Modeling velocity model Synthetic 3D wavefield source model 2009 L’Aquila earthquake simulation for 3D velocity model and finite fault La struttura del progetto è articolata in due fasi velocity model source model topography/geology RUN Synthetic 3D wavefield simulations

Iteratively improve the velocity model Inverse Modeling velocity model source model topography/geology TOMOGRAPHY Misfit between synthetics and data Iteratively improve the velocity model observed data

Remarks on Computational Resources All the computing times and storage presented refers to: Central Italy adjoint tomography (Magnoni 2012) using SPECFEM3D INGV HPC Resources: ELIOS 64 compute nodes, each with 2 quad-core AMD Opteron 2374 processors at 2.4 GHz and with 16 GB RAM (512 total cores, 2 GB RAM/core) SELENE 32 compute nodes, each with 2 oct-core AMD Opteron 6136 processors at 2.4 GHz and with 32 GB RAM (512 total cores, 2 GB RAM/core) then extrapolated for Italy

DATA & Input Parameters Forward Modeling Inverse Visualization Observed Data Input Parameters Observed waveforms Source solutions Geology i dati li prendiamo in FULL SEED format tutti e tre gli elementi sono ottenuti ‘manualmente’ geology: include known information depending on frequency x x Quality check: no gaps no saturation signal/noise station metadata Selection: spatial distribution magnitude MESH

MESH Geological Discontinuities (Moho, basins...) Topography Forward Modeling Inverse Visualization Observed Data Input Parameters Geological Discontinuities (Moho, basins...) Topography Velocity model MESHER (CUBIT/GEOCUBIT)

MESH Geological Discontinuities (moho, basin...) Topography Velocity model MESHER (CUBIT/GEOCUBIT)

Db of saved state variables for the last timestep Forward Modeling Observed Data Forward Modeling Inverse Visualization Input Parameters Station location Source solutions Mesh Partitioning SCOTCH Synthetic seismogram db (ASCII - SAC) Generate Database Db of saved state variables for the last timestep (binary) SPECFEM3D Solver SPECFEM3D

Processing (user routines) Inverse Modeling Synthetic seismogram db Observed Processing (user routines) Source timing Instrument metadata Synchronization Instrument response Filtering Data Forward Modeling Inverse Visualization Input Parameters station metadata --> strumento

Inverse Modeling Syncronization Instrument response Filtering Synthetic seismogram db Observed Processing Source timing Instrument metadata Syncronization Instrument response Filtering Data Forward Modeling Inverse Visualization Input Parameters Text file with selected windows 55DATA/MN.AQU..HHR.D.SAC.6.20.realSYN/AQU.MN.FXR.semd.sac.6.20.synt 2 -1.8396 29.4404 29.4404 42.8104DATA/MN.AQU..HHT.D.SAC.6.20.realSYN/AQU.MN.FXT.semd.sac.6.20.synt 1 -1.5896 25.7604 ... Time window selection Serial fortran code FLEXWIN (Maggi et al 2009) Adjoint source db (ASCII) Misfit measurement and adjoint source calculation Serial fortran code measure_adj (Tape et al. 2009) Text file with measurement estimates + Adjoint source db (ASCII)

(Forward simulation only if attenuation is included) Inverse Modeling Observed Data Forward Modeling Inverse Visualization Input Parameters Adjoint source db (ASCII) Station location Source solutions Mesh Saved state variables for the last timestep (Forward simulation only if attenuation is included) Adjoint event kernel db (binary) Adjoint Simulations SPECFEM3D

Inverse Modeling Adjoint event kernel db (binary) Adjoint misfit Observed Data Forward Modeling Inverse Visualization Input Parameters Adjoint event kernel db (binary) Adjoint misfit kernel db (binary) Regularized misfit kernel db (binary) SUM SUM Sum SPECFEM3D fortran utilities Preconditioning - Smoothing SPECFEM3D fortran utilities

Inverse Modeling Regularized misfit kernel Starting velocity model Observed Data Forward Modeling Inverse Visualization Input Parameters Regularized misfit kernel db (binary) Starting velocity model (binary) SPECFEM3D fortran utilities computationally inexpensive 1.7 external_mesh.bin 3 vp vs rho in vtk UPDATED velocity models (binary & VTK ASCII) Forward Simulations for steplength test SPECFEM3D

Seismic source inversion Input Parameters Observed Data Inverse Modeling Forward Simulations Forward Modeling Forward Modeling Inverse Modeling Visualization 1 2 3 Updated velocity model db 4 5 Iterative improvement of the tomographic model 6 Seismic source inversion FUTURE TASK

KERNEL SUMMATION and REGULARIZATION Inverse Modeling Wall Time on 256 CPUs Cluster C. Italy 256 cores 63 events Italy 2300 cores 150 events Forward Simulations Forward Modeling Forward Modeling Inverse Modeling Visualization FORWARD SIMULATIONS 42 h + 0.5/31 h 84 h 3 h 169/231 h 4 days + 0.5/75 h 8 days 3 h 16/22 days WINDOW SELECTION MEASUREMENTS ADJOINT SOURCES Updated velocity model db (binary & vtk ASCII) ADJOINT SIMULATIONS -il numero prima dello slash è se faccio selez win e meas (compreso quello nello steplength test) in parallelo cioè per tutti gli ev contemp -steplength test ha sia le sim forw che il calcolo delle meas Per l’Italia è fino a 0.5 Hz KERNEL SUMMATION and REGULARIZATION Iterative improvement of the tomographic model STEP LENGTH TEST ~ 13/16 days ~ 28/37 days TOTAL TIME 1 ITERATION

KERNEL SUMMATION and REGULARIZATION Inverse Modeling Storage C. Italy 256 cores 63 events Italy 2300 cores 150 events Forward Modeling Forward Simulations Forward Modeling Inverse Modeling Visualization FORWARD SIMULATIONS 60 GB + 15 GB 70 GB 7 GB 240 GB + 600 GB WINDOW SELECTION MEASUREMENTS ADJOINT SOURCES 144 GB Updated velocity model db (binary & vtk ASCII) ADJOINT SIMULATIONS -lo storage per la win select è pratic nullo, è tutto nel calcolo meas (cioè stima adj src) -lo storage per adj sim comprende sia ev kernel che sismogrammi -nella voce kernel c’è anche lo storage x model update di 1 iteraz (slide 14) -steplength test = forw*4 825 GB KERNEL SUMMATION and REGULARIZATION 30 GB Iterative improvement of the tomographic model STEP LENGTH TEST 2400 GB 392 GB 4 TB TOT STORAGE 1 ITERATION

Visualization Seismograms (SAC, GMT, ObsPy, Google Earth) Observed Data Forward Modeling Inverse Visualization Input Parameters Seismograms (SAC, GMT, ObsPy, Google Earth) Mesh - Velocity Models (Paraview) parallelo spt per ultimi 3 punti Movies (Paraview) Kernels (Paraview) Presently using ‘manual’ scripts Parallel environment