Overview of Broadband Platform Software as used in SWUS Project Philip Maechling BBP Modelers Meeting 12 June 2013

SCEC Software Activities on BBP SCEC software responsibilities on project include: 1.Integration of scientific methods into Broadband Platform Added Irikura, Composite Source Model, EXSIM in 2013 Added multiple Part A and Part B reports Added GMPE codes and utilities (e.g. rotD50, pynga) 2.Running BBP simulations using SCEC computing resources and posting simulation results Ran full Part A and Part B (BBP v13.5) for most methods Ran portions of Part A and Part B (BBP v13.6) for most methods

Define the BBP Platform The SCEC BBP Platform (BBP) integrates: scientific software (e.g. C and Fortran) formatting scripts (e.g. python) seismic software utilities (e.g. rotd50, pynga) post-processing plotting routines observational data (e.g. seismograms) expected results (e.g. seismograms, Peak SA values)

Standard BBP Inputs and Outputs Standard Inputs: Earthquake Source File (.src file) Station List (.stl file) Velocity model specification (region) Computational Method Each method has region specific input files (e.g. Greens functions) Standard outputs: Velocity seismograms Acceleration seismograms Standard Rupture Format (4 methods), Metadata list of programs and input files used Metadata compute environment information

Running the BBP Platform '/home/epicenter/fsilva/bbp/13.6.0/comps/run_bb p.py', '-x', '/home/epicenter-03/fsilva/june13/gp-nps- 50r/Xml/northpalmsprings-gp-0000.xml', '-s', ' ', '-l', '/home/epicenter-03/fsilva/june13/gp-nps- 50r/Sims/logs/ / _northpalmspri ngs-gp.log'

Sim Specification FileName: Simulation_ID.xml (e.g xml) Example Simulation Spec for Part A Validation Simulation: 1.Genslip $BBP_INSTALL_GF/Mojave/gp/mojave_generic1d-urs01.vmod /home/epicenter- 03/fsilva/june13/gp-nps-50r/Src/northpalmsprings-gp-0049.src mojave_generic1d- urs01.vmod northpalmsprings-gp-0049.src metadata.txt northpalmsprings-gp-0049.srf Mojave 2.Jbsim $BBP_INSTALL_GF/Mojave/gp/mojave_generic1d-urs01.vmod /home/epicenter-03/ 3.Hfsims $BBP_INSTALL_GF/Mojave/gp/mojave_generic1d-urs01.vmod /home/epicenter-03 4.Plot_Map /home/epicenter-03/fsilva/june13/gp-nps-50r/Src/northpalmsprings-gp src 5.PlotSeis $BBP_INSTALL_VAL/NORTHPS/gp/northps_v13_3_1.stl northps_v13_3_1.stl True 6.RotD50 $BBP_INSTALL_VAL/NORTHPS/gp/northps_v13_3_1.stl northps_v13_3_1.stl GMPEComparison $BBP_INSTALL_VAL/NORTHPS/gp/northps_v13_3_1.stl /home/ 7.GenPlots $BBP_INSTALL_VAL/NORTHPS/gp/northps_v13_3_1.stl northps_v13_3_1.stl 8.AccPEER3 acc 6.12 NORTHPS 90 False GenHTML $BBP_INSTALL_VAL/NORTHPS/gp/

BBP Software Version Strategy As a general principle, the newest version of the broadband platform is better than any previous version and users should upgrade at first opportunity. Older versions of the code will be archived to support reproducibility of previously obtained results. Any use of an older version of the platform should be examined carefully to understand the scientific goals of using an older version.

Versioning BBP Results BBP results should be attributed to specific BBP software and configuration versions because results may change as methods improve. For a Part A Validation Result, changes to any of these three versions that may impact the results: Broadband Version May This is the source code version Velocity model versionLABasin - 29-May This is the region specific input parameters used in the simulation Validation package versionNR - 29-May This is the set of observational data and associated corrections used in the GOF plots for this validation simulation These are found in the index.html file in the outdata directory for each simulation.

Planned BBP Distributions Plans to create 2 BBP Distributions SWUS (Complete) Standard (no obs data): SWUS Distribution 1.Broadband Version May Velocity model versionLABasin - 29-May Validation package versionNR - 29-May Standard Distribution:. 1.Broadband Version May Velocity model versionLABasin - 29-May

Comparing Results Between BBP Software Versions When comparing BBP results, all results should be produced with the same version of the broadband platform. Standard problems should be re-run when new versions of the platform are released. We assume that: Post processing changes affect results for all methods. Changes to any method’s source code, including python wrappers, may affect results from other methods Changes to one method parameters do not affect results from another method We make this assumption to simplify our analysis of results: We do not have the resources to detect all possible sources of coupling between scientific programs in broadband platform This assumption can be relaxed, if additional analysis is done to confirm results are equivalent between BBP software releases.

Results From Two Software Versions We have posted results. These should be considered results from BBP v Results are posted under two studies: When accessing these results: Use v13.6 results for any simulation if the results exist. If no v13.6 results exist, use v13.5 results. If no v13.5 results exist, simulation was not completed.

BBP Simulation Results Posted Results are Organized by Release, Method, Sim Type List of Methods:Within Method (GP): csm/ gmpe/ exsim/ landers/ gp/lomap/ irikura/ niigata/ sdsu/nps/ ucsb/nr/ whittier/ tottori/

BBP Results Directory Results Organized in two Base Directory: http//bbpvault.usc.edu/bbp/BBP_June_2013/ Part A Results: Method Name: [csm,exsim,gp,irikura,sdsu,ucsb] Event Name: [landers,lomap,niigata,nps,nr,whittier,tottori] Simulation ID: [ , ] Absolute pathname carries information about what method was used and what event was used to produce the simulation results contained in a directory. In each simulation directory, a simulation specification file (sim_id.xml) provides detailed information about the programs run to produce the results in the directory.

Part A Results Directory Listing: ata/ /index html

BBP Output (Results) Directory Results Organized in two Base Directories: http//bbpvault.usc.edu/bbp/BBP_June_2013/ Part B Results: Base File location: Study Name/Method Name/gmpe/ Filename formed from: [method, mag, mechanism, region, dist] bp-gmpe-ucsb-62-ss-nocal-20km.pdf bp-gmpe-ucsb-66-ss-socal-50km_w_gmpe.pdf

BBP Metadata Files Simulation Directories Metadata Files: index html xml system_status txt software_status txt Example: software_status txt Python version: 2.7 (r27:82500, Sep , 18:02:00) GCC Version: gcc (GCC) (Red Hat ) ICC Version: icc (ICC) IFORT Version: ifort (IFORT) G77 Version: GNU Fortran (GCC) (Red Hat fc14) GFORTRAN Version: GNU Fortran (GCC) (Red Hat ) NumPy Version: SciPy Version: Matplotlib Version: 1.0.1

BBP Seismogram Files EXSIM: BBP file with 2 header lines. dt: (200 sps) npts: ( secs) CSM: BBP file with 2 header lines: dt: (50 sps) npts: 4096 (81.92 secs) GP: BBP file with 1 header lines. dt: (40 sps) npts: 4096 (102.4 secs) Irikura: BBP file with 1 header lines. dt: ( sps) npts: ( secs) UCSB: BBP file with 1 header lines. dt: (50 sps) npts: 8192 ( secs) SDSU: BBP file with 1 header lines. dt: (320 sps) npts: ( secs)

BBP Compute and Storage Estimates Estimates for Part A and Part B Simulations Method Compute Hours Outputs (GB) Outdata (GB) GP SDSU EXSIM CSM Irikura UCSB * 32 CPUs 19,488 CPU Hrs 3222 GB137 GB Estimate per Part A and Part B runs 20k SU (3.2 TB (Total), 0.2 TB (Outdata))

BBP Compute and Storage Estimates Estimates for Fling Study Method Compute Hours Outputs (GB)Outdata (GB) GP 1416 ??? * 16 CPUs = Assume all 6 methods run Fling * 6 = * 6 = 960 (GB) Cumulative Totals (Part A & Part B + Fling) For 6 Methods (Outdata Only): (20k + 135k) = 155k CPU hours ( ) = 1.2 TB Storage

BPP Software Personnel BBP Developers by Method: GP – Robert Graves, Arbin Pitarka Composite Source Model – John Anderson SDSU – Kim Olsen, Rumi Takedatsu UCSB – Jorge Crempien, Ralph Archuleta EXSIM – Karen Assatourians, Gail Atkinson Irikura – Jeff Bayless, Yajie (Jerry) J. Lee, Paul Somerville BBP Software Developers At SCEC: Fabio Silva Philip Maechling Scott Callaghan David Gill

End