CyberShake Study 15.3 Science Readiness Review

Study 15.3 Scientific Goals Calculate a 1 Hz map of Southern California Produce meaningful 2 second results for the UGMS RotD50 and RotD100 at 2, 3, 4, 5, 7.5, 10 seconds Contour maps Compare 0.5 Hz and 1 Hz hazard maps Use Graves & Pitarka (2014) rupture generator with regular spaced hypocenters 336 sites (10 km mesh + points of interest + “gap” sites) Run 14 UGMS sites first Produce 1 Hz seismograms which could be combined with BBP high-frequency seismograms

Proposed Study sites (336) Green sites are the 50 new “gap” sites

Study 15.3 Data Products CVM-S4.26 Los Angeles-area hazard maps RotD100 2, 3, 4, 5, 7.5, 10 sec RotD50 2, 3, 4, 5, 7.5, 10 sec Geometric mean 2, 3, 5, 10 sec Hazard curves for 286 sites, at 2s, 3s, 5s, 10s 336 sets of 2-component SGTs Seismograms for all ruptures (~160M) Peak amplitudes in DB for 2s, 3s, 5s, 10s RotD100, RotD50 and geometric mean SA

Study 15.3 Notables First 1 Hz hazard maps First study with RotD50 and RotD100 calculated First study to use OLCF Titan First study with Graves & Pitarka (2014) rupture generator with uniformly spaced hypocenters First study with 200 m rupture grid point spacing First study with source filtered at a different frequency than the simulation frequency

Study 15.3 Parameters 1.0 Hz deterministic 100 m spacing dt=0.005 sec nt=40000 timesteps CVM-S 4.26 Vs min = 500 m/s UCERF 2 Graves & Pitarka (2014) rupture variations 200 m rupture grid point spacing Source filtered at 2.0 Hz

Rupture Generator Differences When rupture geometry was changed from 1000m to 200m resolution, hazard curves changed dramatically TEST site: blue = 200 m, black = 1000 m 0.5Hz UCERF2 3 sec SA CVM-S4

Rupture Generator We determined that the change in hazard curves was due to hypocenter undersampling M6.55, Puente Hills

Rupture Generator changes Previous number of realizations related to fault length # of realizations = max(10, C * Area/10.0) C = 0.5 Each realization is unique slip + hypocenter location Supports either random or uniform hypocenter distribution

Rupture Generator v3.3.1 Use of new G&P rupture generator (v3.3.1) brought 1000m and 200m curves into agreement TEST site, black = 200 m, magenta = 1000 m 0.5Hz UCERF2 3 sec SA CVM-S4

Random vs Uniform Hypocenters Variation counts G&P 2010: 423k Uniform: 485k Random: 542k Uniform easier to interpolate 0.5Hz UCERF2 3 sec SA CVM-S4.26 WNGC site: black=random, magenta=uniform

Source Filtering, 0.5 Hz simulation Changed frequency of 4 th order lowpass Butterworth filtering of SGT source from 0.5 to 1 Hz WNGC site: Blue = filtered at 1 Hz; Black = filtered at 0.5 Hz

PseudoAA content by source filter WNGC site, blue = filtered at 0.5 Hz, green = filtered at 1 Hz M8.05, Elsinore

Source Filtering, 1 Hz simulation Blue = filtered at 2 Hz; Black = filtered at 1 Hz Changed frequency of 4 th order lowpass Butterworth filtering of SGT source from 1 Hz to 2 Hz

PseudoAA content by source filter WNGC site, blue = filtered at 1 Hz, green = filtered at 2 Hz M8.05, Elsinore

Fourier content by source filter WNGC site blue = filtered at 1 Hz green = filtered at 2 Hz M8.05, Elsinore

Computational Requirements Per site: ~3720 node-hrs SGTs: depends on execution site (~50%) Titan = 2110 node-hrs / 63,300 SUs Blue Waters = 1760 node-hrs / 30,200 SUs More expensive for Titan because of padding in pilot jobs and different node-hrs -> SU conversion PP: 1880 node-hrs / 60,200 SUs (~50%) Computational time: Titan (SGTs): 355K node-hours / 10.7M SUs Blue Waters: 928K node-hours SGTs: 275K GPU node-hrs, 21K CPU node-hrs PP: 632K CPU node-hrs Titan has 104M SUs remaining Blue Waters has 5.3M node-hrs remaining

Storage Requirements Titan Purged: 526 TB (for SGTs and temp data) Blue Waters Delayed purge: 506 TB (for Titan SGTs) Purged: 526 TB SGTs + 9 TB data products SCEC Archived: 9.1 TB (seismograms, PSA, RotD) Database: 268 GB 4 periods, 6) Temporary: 608 GB (workflow logs) Shared SCEC disks have 171 TB free

Estimated Duration Limiting factors: XK node queue time 800 XK nodes is 19% of Blue Waters Titan -> Blue Waters If throughput is very high, transfer could be bottleneck USC HPC downtime for ~1 week in April Estimated completion is 12 weeks (11 running + 1 downtime) Based on same node availability as Study 14.2 Planning to request reservation on Blue Waters Planning to request high priority on Titan

Personnel Support Scientists Tom Jordan, Kim Olsen, Rob Graves Technical Lead Scott Callaghan Job Submission / Run Monitoring Scott Callaghan, David Gill, Phil Maechling NCSA Support Omar Padron, Tim Bouvet Titan Support Val Anantharaj USC Support John Yu, John Mehringer Workflow Support Karan Vahi, Gideon Juve

Science To-dos Pending Confirm SGTs from Titan give same result as SGTs from Blue Waters Calculate two duplicate SGT sites on Blue Waters and Titan and confirm results match Forward simulations versus reciprocity? Run forward simulations to confirm reciprocity and forward calculations match Requires converting SRF to AWP-ODC source input format

Risks Queue times on Blue Waters for XK nodes Will try to dynamically assign SGT jobs to resources Unforeseen complications with Titan pilot jobs Small tests have worked OK, but issues at scale? Congestion protection events (network overloaded) If triggered consistently, will need to limit number of post-processing workflows

Action Items Confirm 200s is long enough for SGT simulation Insert ERF 36 hypocenters into DB Decide whether or not to run forward simulation Determine why 2 Hz filtered source isn’t showing expected differences in seismograms or hazard curves Select additional sites to help fill in gaps and discontinuities in the hazard map

Thanks for your time!