CyberShake Study 16.9 Science Readiness Review

Study 16.9 Scientific Goals Expand CyberShake to Central California Calculate 1 Hz map with Vs min = 900 m/s Two velocity models CCA-06 CCA 1D model Calculate hazard at sites of interest to PG&E Compare Southern California and Central California results at overlapping sites

Proposed Study sites (438) 10 km spacing (purple) 5 km spacing (green) CISN + OBS stations (orange) Missions (blue) USGS California Gazetteer locations (red) PG&E pumping stations (cyan) Diablo Canyon was removed, but CISN station is ~1.4 km away Pink box is 180x240 km

Study 16.9 Data Products CCA-06 and CCA 1D Central California hazard maps RotD100 2, 3, 4, 5, 7.5, 10 sec Geometric mean 2, 3, 5, 10 sec Hazard curves for 438 sites x 2 velocity models, at 2s, 3s, 4s, 5s, 7.5s, 10s Seismograms for all ruptures (~438M) Peak amplitudes in DB for 2s, 3s, 4s, 5s, 7.5s, 10s RotD100, RotD50 and geometric mean SA Durations in DB for velocity and acceleration 5-75%, 5-95%, 20-80%

Study 16.9 Notables First Central California CyberShake calculation First study with CCA models First study with a different minimum Vs First study with new workflow approach on Titan, enabling end-to-end CyberShake First deterministic study to include duration calculations

Study 16.9 Parameters 1.0 Hz deterministic Vs min = 900 m/s 175 m spacing dt=0.00875 sec nt=23000 timesteps (201.25 sec) Vs min = 900 m/s CCA-06 model Volumes extending outside of CCA-06 will use CVM-S4.26 if possible, then the SCEC 1D model CCA 1D model UCERF 2 Graves & Pitarka (2014) rupture variations 200 m rupture grid point spacing Source filtered at 2.0 Hz

Inclusion of northern SAF events Using 200 km cutoff, 1/3 of sites capture northern events Will continue to use 200 km cutoff, so some sites will include northern SAF events and some won’t

Computational Requirements Per site: 1220-1940 node-hrs SGTs: (20-30%) Titan = 400 node-hrs (12,000 SUs) Blue Waters = 400 node-hrs (6,400 SUs) PP: (60-75%) Titan = 1440 node-hrs (43,200 SUs) Blue Waters = 720 node-hrs (23,040 SUs) More expensive for Titan due to 16 cores/node Total computational time Titan (219 runs): 425K node-hours / 15.9M SUs Blue Waters (657 runs): 1.00M node-hours Titan has 23M SUs remaining in 2016 Blue Waters has 3M node-hrs remaining (11/16?)

Storage Requirements Titan Blue Waters Purged: 398 TB SGTs + 3.3 TB data products Blue Waters Purged: 1193 TB SGTs + 10 TB data products Will clean up as we go to avoid exceeding quotas SCEC Archived: 13.3 TB Seismograms, PSA, RotD, durations Database: 918 GB Geom @ 4, RotD100 @ 6, RotD50 @ 6, 8 durations Temporary: 1 TB (workflow logs) Shared SCEC disks have 109 TB free

Estimated Duration Limiting factors: Estimated completion is 5 weeks XK node queue time Has been long outside of Blue Waters reservation Unscheduled downtime Titan workflow performance New database performance Estimated completion is 5 weeks Based on same node availability as Study 15.4 Planning to request reservation on Blue Waters Planning to request increased quota on Titan

Personnel Support Scientists Technical Lead NCSA Support Titan Support Tom Jordan, Kim Olsen, Rob Graves Technical Lead Scott Callaghan NCSA Support Tim Bouvet, Greg Bauer Titan Support Judy Hill USC Support John Yu Workflow Support Karan Vahi

Science To-dos Pending Calculate hazard curves for 3 overlapping test sites for both CCA-06 and CCA 1D Calculate hazard curves for 3 non-overlapping sites with varied Vs30 values with both CCA-06 and CCA 1D Confirm runs on Titan give same result as runs on Blue Waters Calculate two duplicate sites on Blue Waters and Titan and confirm results match

Risks Currently unable to run on Titan due to certificate issues Working with OLCF staff to resolve; if not, move full calculation to Blue Waters Queue times on Blue Waters for XK nodes Unforeseen complications with Titan workflows Small tests have worked OK, but issues at scale? Database performance Study 15.4 was OK, but now including durations Changed DB configuration for better performance, but not tested in production yet

Action Items Calculate test hazard curves for V&V Add duration calculation to deterministic code Have meetings with OLCF and NCSA

Thanks for your time!