1. CyberShake Study 17.3 Science Readiness Review

2. Study 17.3 Scientific Goals
Expand CyberShake to Central California
Calculate 1 Hz map with Vs min = 900 m/s
Two velocity models
CCA-06
CCA 1D model
Calculate hazard at sites of interest to PG&E
Compare Southern California and Central California results at overlapping sites

3. Proposed Study sites (438)
10 km spacing (purple)
5 km spacing (green)
CISN + OBS stations (orange)
Missions (blue)
USGS California Gazetteer locations (red)
PG&E pumping stations (cyan)
Diablo Canyon was removed, but CISN station is ~1.4 km away
Pink box is 180x240 km

4. Study 17.3 Data Products
CCA-06 and CCA 1D Central California hazard curves for 438 sites
RotD100 2, 3, 4, 5, 7.5, 10 sec
RotD50 2, 3, 4, 5, 7.5, 10 sec
Geometric mean 2, 3, 5, 10 sec
RotD50 hazard maps at 2, 3, 5, 10 sec
Seismograms for all ruptures (~438M)
Intensity measures in DB for 2s, 3s, 4s, 5s, 7.5s, 10s
RotD100, RotD50 and geometric mean SA
Durations in DB for velocity and acceleration
5-75%, 5-95%, 20-80%

5. Study 17.3 Notables First Central California CyberShake calculation
First study with CCA models + smoothing
First study with a different minimum Vs
First study with new workflow approach on Titan, enabling end-to-end CyberShake
First deterministic study to include duration calculations

6. Velocity Models: 3D Pink: CCA region box
White: s1252 SGT simulation box
Red: CVM- S4.26.m01
Blue: CCA-06
Green: USGS Bay Area

7. Velocity Model: 3D order
CCA-06 (with trilinear interpolation and projection fixes)
USGS Bay Area (
CVM-S4.26.m01 (GTL will be lost due to Vs min)
CVM-S4.26.m01 has a 1D model everywhere, so this will be used outside of 3D model regions

8. Velocity Model: Cutoffs
Set Vs min = 900
Corresponds to tomographic inversion
Plotted and examined Vp, rho
Smallest values found were Vp=1801, rho=2057
Set Vp min = 1800, rho min = 2000
Vp, surface
rho, surface

9. Velocity Model: Smoothing
To avoid reflection effects, we apply smoothing
We identify all transition boundaries at the surface
Select all points within 58 grid points (10.15 km) of a boundary
We average over 58 grid points (10.15 km) in the X and Y direction to get the smoothed value
Only applied along velocity model boundaries

10. Smoothing, surface plots
Unsmoothed
Smoothed

11. Smoothing, vertical plots
Top: unsmoothed
Bottom: smoothed

12. Velocity Model, 1D 1D model produced by averaging the raw CCA- 06 data
500 m grid spacing
On land, within the CyberShake CCA region
No smoothing required

13. Study 17.3 Parameters Source filtered at 2.0 Hz 1.0 Hz deterministic
175 m spacing
dt= sec
nt for SGTs =23000 timesteps ( sec)
nt for seismograms is 5000 (437.5 sec)
Source filtered at 2.0 Hz
UCERF 2
Graves & Pitarka (2014) rupture variations
200 m rupture grid point spacing

14. Inclusion of northern SAF events
Using 200 km cutoff, 1/3 of sites capture northern events
Ran tests by using sites close to cutoff and comparing include/exclude
Decided to keep km cutoff
s1207 (198 km)
s1252 (193 km)

15. Include/exclude plots

16. Verification with Study 15.4
Compared Study 15.4, Blue Waters codebase, Titan codebase
All curves overlap
All seismograms overlap

17. Study order Complete each site (3D and 1D) before moving on
30 overlapping sites (with So Cal region)
4 PG&E sites
58 CISN broadband, PG&E, and OBS stations
km grid sites
36 cities and historic missions

18. Computational Plan Run all SGTs on Titan
Run 25% of PP on Titan, 75% on Blue Waters
SGTs = 400 node-hrs (12,000 SUs) per site
PP per site:
Titan = 2400 node-hrs (72,000 SUs)
Blue Waters = 720 node-hrs (23,040 SUs)
Titan more expensive due to 16 cores/node and restart cost
Total computational time
Titan: 1.1M node-hours / 33M SUs
Blue Waters: 591K node-hours
Titan has 92M SUs remaining in 2017
Blue Waters has 990K node-hrs remaining (5/31?)

19. Storage Requirements Titan Blue Waters
Purged: 1591 TB SGTs TB data products
Blue Waters
Purged: 577 TB SGTs + 10 TB data products
Will clean up as we go to avoid exceeding quotas
SCEC
Archived: 13.3 TB
Seismograms, PSA, RotD, durations
Database: 918 GB
4, 6, 6, 8 durations
Temporary: 1 TB (workflow logs)
Shared SCEC disks have 94 TB free

20. Estimated Duration Limiting factors: Estimated completion is 5 weeks
Titan queue time
Unscheduled downtime
Titan workflow performance
New database performance
Longer jobs on Titan to avoid PP restart?
Estimated completion is 5 weeks
Based on same node availability as Study 15.4
Planning to request XE reservation on Blue Waters
Planning to request increased scratch quota on Titan

21. Personnel Support Scientists Technical Lead NCSA Support Titan Support
Tom Jordan, Kim Olsen, Rob Graves, Christine Goulet
Technical Lead
Scott Callaghan
NCSA Support
Tim Bouvet, Greg Bauer
Titan Support ?
USC Support
Phil Maechling, Kevin Milner, John Yu, David Gill
Workflow Support
Karan Vahi, Mats Rynge

22. Science To-dos Pending
Calculate hazard curves s001 on Titan and Blue Waters for verification

23. Risks Queue times on Titan
Unforeseen complications with Titan workflows
Small tests have worked OK, but issues at scale?
Database performance (moment.usc.edu)
Study 15.4 was OK
Changed DB configuration for better performance, but not tested in production yet

24. Action Items Calculate s001 curve on Blue Waters for V&V
Have calls with OLCF and NCSA

25. Thanks for your time!