1. CyberShake Study 16.9 Science Readiness Review

2. Study 16.9 Scientific Goals
Expand CyberShake to Central California
Calculate 1 Hz map with Vs min = 900 m/s
Two velocity models
CCA-06
CCA 1D model
Calculate hazard at sites of interest to PG&E
Compare Southern California and Central California results at overlapping sites

3. Proposed Study sites (438)
10 km spacing (purple)
5 km spacing (green)
CISN + OBS stations (orange)
Missions (blue)
USGS California Gazetteer locations (red)
PG&E pumping stations (cyan)
Diablo Canyon was removed, but CISN station is ~1.4 km away
Pink box is 180x240 km

4. Study 16.9 Data Products
CCA-06 and CCA 1D Central California hazard maps
RotD100 2, 3, 4, 5, 7.5, 10 sec
Geometric mean 2, 3, 5, 10 sec
Hazard curves for 438 sites x 2 velocity models, at 2s, 3s, 4s, 5s, 7.5s, 10s
Seismograms for all ruptures (~438M)
Peak amplitudes in DB for 2s, 3s, 4s, 5s, 7.5s, 10s
RotD100, RotD50 and geometric mean SA
Durations in DB for velocity and acceleration
5-75%, 5-95%, 20-80%

5. Study 16.9 Notables First Central California CyberShake calculation
First study with CCA models
First study with a different minimum Vs
First study with new workflow approach on Titan, enabling end-to-end CyberShake
First deterministic study to include duration calculations

6. Study 16.9 Parameters 1.0 Hz deterministic Vs min = 900 m/s
175 m spacing
dt= sec
nt=23000 timesteps ( sec)
Vs min = 900 m/s
CCA-06 model
Volumes extending outside of CCA-06 will use CVM-S4.26 if possible, then the SCEC 1D model
CCA 1D model
UCERF 2
Graves & Pitarka (2014) rupture variations
200 m rupture grid point spacing
Source filtered at 2.0 Hz

7. Inclusion of northern SAF events
Using 200 km cutoff, 1/3 of sites capture northern events
Will continue to use km cutoff, so some sites will include northern SAF events and some won’t

8. Computational Requirements
Per site: node-hrs
SGTs: (20-30%)
Titan = 400 node-hrs (12,000 SUs)
Blue Waters = 400 node-hrs (6,400 SUs)
PP: (60-75%)
Titan = 1440 node-hrs (43,200 SUs)
Blue Waters = 720 node-hrs (23,040 SUs)
More expensive for Titan due to 16 cores/node
Total computational time
Titan (219 runs): 425K node-hours / 15.9M SUs
Blue Waters (657 runs): 1.00M node-hours
Titan has 23M SUs remaining in 2016
Blue Waters has 3M node-hrs remaining (11/16?)

9. Storage Requirements Titan Blue Waters
Purged: 398 TB SGTs TB data products
Blue Waters
Purged: 1193 TB SGTs + 10 TB data products
Will clean up as we go to avoid exceeding quotas
SCEC
Archived: 13.3 TB
Seismograms, PSA, RotD, durations
Database: 918 GB
4, 6, 6, 8 durations
Temporary: 1 TB (workflow logs)
Shared SCEC disks have 109 TB free

10. Estimated Duration Limiting factors: Estimated completion is 5 weeks
XK node queue time
Has been long outside of Blue Waters reservation
Unscheduled downtime
Titan workflow performance
New database performance
Estimated completion is 5 weeks
Based on same node availability as Study 15.4
Planning to request reservation on Blue Waters
Planning to request increased quota on Titan

11. Personnel Support Scientists Technical Lead NCSA Support Titan Support
Tom Jordan, Kim Olsen, Rob Graves
Technical Lead
Scott Callaghan
NCSA Support
Tim Bouvet, Greg Bauer
Titan Support
Judy Hill
USC Support
John Yu
Workflow Support
Karan Vahi

12. Science To-dos Pending
Calculate hazard curves for 3 overlapping test sites for both CCA-06 and CCA 1D
Calculate hazard curves for 3 non-overlapping sites with varied Vs30 values with both CCA-06 and CCA 1D
Confirm runs on Titan give same result as runs on Blue Waters
Calculate two duplicate sites on Blue Waters and Titan and confirm results match

13. Risks Currently unable to run on Titan due to certificate issues
Working with OLCF staff to resolve; if not, move full calculation to Blue Waters
Queue times on Blue Waters for XK nodes
Unforeseen complications with Titan workflows
Small tests have worked OK, but issues at scale?
Database performance
Study 15.4 was OK, but now including durations
Changed DB configuration for better performance, but not tested in production yet

14. Action Items Calculate test hazard curves for V&V
Add duration calculation to deterministic code
Have meetings with OLCF and NCSA

15. Thanks for your time!