Southern California Earthquake Center CyberShake Progress Update November 3, 2014 – 4 May 2015 UGMS May 2015 Meeting Philip Maechling SCEC IT Architect 4 May 2015

Southern California Earthquake Center CyberShake Platform: Physics-Based PSHA

Southern California Earthquake Center November 2014 UGMS Meeting Review of CyberShake Study 14.3 Hazard models for 14 Los Angeles Region Sites. Reviewed CyberShake using 3D velocity model CVM-S4.26 Min Vs : 500 m/s Max CyberShake Frequency : 0.5Hz Frequency Range: 3S Period and longer Compared MCER Hazard Curves GMPE vs CyberShake for 14 Sites In Los Angeles Region

Southern California Earthquake Center November 2014 UGMS MCER Sites

Southern California Earthquake Center November 2014 UGMS MCER Sites

Southern California Earthquake Center Overview New Results for May 2015 UGMS Meeting Los Angeles region MCER Contour Maps based on 284 sites of CyberShake Study 14.3 Initiate Processing for CyberShake 15.4 Study using NSF and DOE Computers Los Angeles region Hazard Model based on 336 sites at 1Hz Estimated 40M Computer Hours Estimated 500TB+ temporary data at DOE,NSF Computers Estimated 11TB persistent data at SCEC Current Status: CyberShake 15.4 hazard curves for 14 Sites: Maximum Frequency: 1Hz 3D Velocity Model: CVM-S4.26 Min Vs: 500 m/s Comparison MCER curves between GMPE and CyberShake 15.4

Southern California Earthquake Center CyberShake Study 15.4 sites (336) 336 sites (10 km mesh + points of interest + “gap” sites) Green sites are the 50 new “gap” sites (Run 14 UGMS sites first )

Southern California Earthquake Center Study 15.3 Scientific Goals Calculate a CyberShake Hazard Model of Southern California using 1 Hz simulated seismograms CyberShake 15.4 Hazard Model Study: Earthquake Rupture Forecast: UCERF2 Rupture Generator: Graves & Pitarka (2014) rupture generator with regular spaced hypocenters 3D Velocity Model: CVM-S4.26 Min Vs: 500 m/s Max Simulated Freq: 1Hz Produce meaningful 2 second ground motion amplitudes RotD50 and RotD100 at 2, 3, 4, 5, 7.5, 10 seconds Contour maps

Southern California Earthquake Center New Computational Results This Meeting Results for May 2015 UGMS Meeting CS14.3 maps at ≥ 3 s for the LA region In November 2014, we had RotD100 amplitudes needed for MCER at each site, for only 14 sites. We have now completed RotD100 and MCER calculations for all 286 sites using CyberShake 14.3 seismograms. Added difference maps requested at that meeting CS14.3 hazard curves, RTGM curves, and combined (probabilistic plus deterministic) curves for the 14 sites In November 2014, we had these results for 14 sites. Now, these can be calculated for any of the CS sites. CS15.4 hazard curves, RTGM curves, and combined (probabilistic plus deterministic) curves for the 14 site Preliminary results from CyberShake 15.4 production calculation.

Southern California Earthquake Center CyberShake 14.3 MCER Contour Maps

Southern California Earthquake Center CyberShake 15.4 RotD100 at 2S period (LADT)

Southern California Earthquake Center CyberShake 15.4 Deterministic RotD100 (COO)

Southern California Earthquake Center Study 15.4 Parameters 1.0 Hz deterministic –100 m spacing –dt=0.005 sec –nt=40000 timesteps CVM-S 4.26 –Vs min = 500 m/s UCERF 2 Graves & Pitarka (2014) rupture variations –200 m rupture grid point spacing Source filtered at 2.0 Hz

Southern California Earthquake Center Expected Study 15.4 Data Products CVM-S4.26 Los Angeles-area hazard maps RotD100 2, 3, 4, 5, 7.5, 10 sec RotD50 2, 3, 4, 5, 7.5, 10 sec Geometric mean 2, 3, 5, 10 sec Hazard curves for 286 sites, at 2s, 3s, 5s, 10s 336 sets of 2-component SGTs Seismograms for all ruptures (~160M) Peak amplitudes in DB for 2s, 3s, 5s, 10s RotD100, RotD50 and geometric mean SA

Southern California Earthquake Center Estimated Duration Limiting factors: –XK node queue time 800 XK nodes is 19% of Blue Waters –Titan -> Blue Waters If throughput is very high, transfer could be bottleneck –USC HPC downtime for ~1 week in April Estimated completion is 12 weeks (11 running + 1 downtime) –Based on same node availability as Study 14.2 Planning to request reservation on Blue Waters Planning to request high priority on Titan

Southern California Earthquake Center End

Southern California Earthquake Center CyberShake 15.4 RotD100 at 3S period (LADT)

Southern California Earthquake Center CyberShake 15.4 RotD100 at 2S period (LADT)

Southern California Earthquake Center CyberShake 15.4 PSA3.0 Hazard Curve (LADT)

Southern California Earthquake Center CyberShake 14.3 PSA3.0 Hazard Curve (LADT)

Southern California Earthquake Center Rupture Generator We determined that the change in hazard curves was due to hypocenter undersampling M6.55, Puente Hills

Southern California Earthquake Center Rupture Generator changes Previous number of realizations related to fault length # of realizations = max(10, C * Area/10.0) –C = 0.5 Each realization is unique slip + hypocenter location Supports either random or uniform hypocenter distribution

Southern California Earthquake Center Rupture Generator v3.3.1 Use of new G&P rupture generator (v3.3.1) brought 1000m and 200m curves into agreement TEST site, black = 200 m, magenta = 1000 m 0.5Hz UCERF2 3 sec SA CVM-S4

Southern California Earthquake Center Random vs Uniform Hypocenters Variation counts –G&P 2010: 423k –Uniform: 485k –Random: 542k Uniform easier to interpolate 0.5Hz UCERF2 3 sec SA CVM- S4.26 WNGC site: black=random, magenta=uniform

Southern California Earthquake Center Storage Requirements Titan –Purged: 526 TB (for SGTs and temp data) Blue Waters –Delayed purge: 506 TB (for Titan SGTs) –Purged: 526 TB SGTs + 9 TB data products SCEC –Archived: 9.1 TB (seismograms, PSA, RotD) –Database: 268 GB 4 periods, 6) –Temporary: 608 GB (workflow logs) –Shared SCEC disks have 171 TB free

Southern California Earthquake Center Computational Requirements Per site: ~3720 node-hrs –SGTs: depends on execution site (~50%) Titan = 2110 node-hrs / 63,300 SUs Blue Waters = 1760 node-hrs / 30,200 SUs More expensive for Titan because of padding in pilot jobs and different node-hrs -> SU conversion –PP: 1880 node-hrs / 60,200 SUs (~50%) Computational time: –Titan (SGTs): 355K node-hours / 10.7M SUs –Blue Waters: 928K node-hours SGTs: 275K GPU node-hrs, 21K CPU node-hrs PP: 632K CPU node-hrs Titan has 104M SUs remaining Blue Waters has 5.3M node-hrs remaining

Southern California Earthquake Center End