CyberShake Study 17.3 Science Readiness Review

Study 17.3 Scientific Goals Expand CyberShake to Central California Calculate 1 Hz map with Vs min = 900 m/s Two velocity models CCA-06 CCA 1D model Calculate hazard at sites of interest to PG&E Compare Southern California and Central California results at overlapping sites

Proposed Study sites (438) 10 km spacing (purple) 5 km spacing (green) CISN + OBS stations (orange) Missions (blue) USGS California Gazetteer locations (red) PG&E pumping stations (cyan) Diablo Canyon was removed, but CISN station is ~1.4 km away Pink box is 180x240 km

Study 17.3 Data Products CCA-06 and CCA 1D Central California hazard curves for 438 sites RotD100 2, 3, 4, 5, 7.5, 10 sec RotD50 2, 3, 4, 5, 7.5, 10 sec Geometric mean 2, 3, 5, 10 sec RotD50 hazard maps at 2, 3, 5, 10 sec Seismograms for all ruptures (~438M) Intensity measures in DB for 2s, 3s, 4s, 5s, 7.5s, 10s RotD100, RotD50 and geometric mean SA Durations in DB for velocity and acceleration 5-75%, 5-95%, 20-80%

Study 17.3 Notables First Central California CyberShake calculation First study with CCA models + smoothing First study with a different minimum Vs First study with new workflow approach on Titan, enabling end-to-end CyberShake First deterministic study to include duration calculations

Velocity Models: 3D Pink: CCA region box White: s1252 SGT simulation box Red: CVM- S4.26.m01 Blue: CCA-06 Green: USGS Bay Area

Velocity Model: 3D order CCA-06 (with trilinear interpolation and projection fixes) USGS Bay Area (https://earthquake.usgs.gov/data/3dgeologic/) CVM-S4.26.m01 (GTL will be lost due to Vs min) CVM-S4.26.m01 has a 1D model everywhere, so this will be used outside of 3D model regions

Velocity Model: Cutoffs Set Vs min = 900 Corresponds to tomographic inversion Plotted and examined Vp, rho Smallest values found were Vp=1801, rho=2057 Set Vp min = 1800, rho min = 2000 Vp, surface rho, surface

Velocity Model: Smoothing To avoid reflection effects, we apply smoothing We identify all transition boundaries at the surface Select all points within 58 grid points (10.15 km) of a boundary We average over 58 grid points (10.15 km) in the X and Y direction to get the smoothed value Only applied along velocity model boundaries

Smoothing, surface plots Unsmoothed Smoothed

Smoothing, vertical plots Top: unsmoothed Bottom: smoothed

Velocity Model, 1D 1D model produced by averaging the raw CCA- 06 data 500 m grid spacing On land, within the CyberShake CCA region No smoothing required

Study 17.3 Parameters Source filtered at 2.0 Hz 1.0 Hz deterministic 175 m spacing dt=0.00875 sec nt for SGTs =23000 timesteps (201.25 sec) nt for seismograms is 5000 (437.5 sec) Source filtered at 2.0 Hz UCERF 2 Graves & Pitarka (2014) rupture variations 200 m rupture grid point spacing

Inclusion of northern SAF events Using 200 km cutoff, 1/3 of sites capture northern events Ran tests by using sites close to cutoff and comparing include/exclude Decided to keep 200 km cutoff s1207 (198 km) s1252 (193 km)

Include/exclude plots

Verification with Study 15.4 Compared Study 15.4, Blue Waters codebase, Titan codebase All curves overlap All seismograms overlap

Study order Complete each site (3D and 1D) before moving on 30 overlapping sites (with So Cal region) 4 PG&E sites 58 CISN broadband, PG&E, and OBS stations 310 10 km grid sites 36 cities and historic missions

Computational Plan Run all SGTs on Titan Run 25% of PP on Titan, 75% on Blue Waters SGTs = 400 node-hrs (12,000 SUs) per site PP per site: Titan = 2400 node-hrs (72,000 SUs) Blue Waters = 720 node-hrs (23,040 SUs) Titan more expensive due to 16 cores/node and restart cost Total computational time Titan: 1.1M node-hours / 33M SUs Blue Waters: 591K node-hours Titan has 92M SUs remaining in 2017 Blue Waters has 990K node-hrs remaining (5/31?)

Storage Requirements Titan Blue Waters Purged: 1591 TB SGTs + 3.3 TB data products Blue Waters Purged: 577 TB SGTs + 10 TB data products Will clean up as we go to avoid exceeding quotas SCEC Archived: 13.3 TB Seismograms, PSA, RotD, durations Database: 918 GB Geom @ 4, RotD100 @ 6, RotD50 @ 6, 8 durations Temporary: 1 TB (workflow logs) Shared SCEC disks have 94 TB free

Estimated Duration Limiting factors: Estimated completion is 5 weeks Titan queue time Unscheduled downtime Titan workflow performance New database performance Longer jobs on Titan to avoid PP restart? Estimated completion is 5 weeks Based on same node availability as Study 15.4 Planning to request XE reservation on Blue Waters Planning to request increased scratch quota on Titan

Personnel Support Scientists Technical Lead NCSA Support Titan Support Tom Jordan, Kim Olsen, Rob Graves, Christine Goulet Technical Lead Scott Callaghan NCSA Support Tim Bouvet, Greg Bauer Titan Support ? USC Support Phil Maechling, Kevin Milner, John Yu, David Gill Workflow Support Karan Vahi, Mats Rynge

Science To-dos Pending Calculate hazard curves s001 on Titan and Blue Waters for verification

Risks Queue times on Titan Unforeseen complications with Titan workflows Small tests have worked OK, but issues at scale? Database performance (moment.usc.edu) Study 15.4 was OK Changed DB configuration for better performance, but not tested in production yet

Action Items Calculate s001 curve on Blue Waters for V&V Have calls with OLCF and NCSA

Thanks for your time!