SI2 Project Highlights SI2-SSI: Community Software for Extreme-Scale Computing in Earthquake System Science PIs: Thomas H. Jordan, Yifeng Cui, Kim B. Olsen, Ricardo Taborda Project Dates: 1 September 2015 through August 31, 2019 Award Number: ACI-1450451 SCEC Extreme Scale Earthquake Simulation Software Recent Accomplishments 19 July 2017 Thomas H. Jordan (tjordan@usc.edu) Yifeng Cui (yfcui@sdsc.edu) Kim Bak Olsen (kbolsen@mail.sdsu.edu) Ricardo Taborda (ricardo.taborda@memphis.edu)

6.5x AWP-ODC on NVIDIA GPUs A first 4-Hz nonlinear M7.7 earthquake simulation on the southern San Andreas Fault conducted using 4,200 Blue Waters GPUs 100% of parallel efficiency achieved for both linear/ nonlinear versions of AWP-ODC up to 8,192 GPUs Accelerated time-to-solution from original nonlinear 0.68sec to 0.29sec per iteration Blue Waters PAID project provided additional support. Snapshots from the 4 Hz San Andreas simulation. (a-c) and (d-f) show fault-parallel velocity for the linear and nonlinear cases, respectively, and (g-i) depict the evolution of permanent plastic strain at the surface obtained from the nonlinear simulation. The dashed line shows the fault trace. (Roten, D., Y. Cui, K. Olsen, S. Day, K. Withers, W. Savran, P. Wang and D. Mu, High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers, SC’16, 1-10, Nov 13-18, Salt Lake City, 2016) 6.5x Speedup of CyberShake SGT version on Cray XK7 compared to XE6 at node-to-node level (Roten et al., SC’16) (https://blogs.nvidia.com/blog/2015/08/31/gpu-quake-hazard)

NVIDIA Share Your Science Video – SC’16 https://www.youtube.com/watch?v=fYV_DawfuCg&list=PL5B692fm6--tF7nnrkGv02V9lrKVqoCpl&index=59 http://on-demand.gputechconf.com/supercomputing/2016/video/sc6106-phil-maechling-earthquake-simulations-extreme-scaling.html (Roten et al., SC’16)

AWP-ODC on Intel Xeon Phi Stencil generation and vector folding through YASK tool: https://github.com/01org/yask Hybrid placement of grids in DDR and MCDRAM Normalized cross architecture evaluation in Mega Lattice Updates per Second (MLUPS): Xeon Phi KNL 7290 achieves 2x speedup over NVIDIA K20X, 97% of NVIDIA Tesla P100 performance Performance on 9,000 nodes of Cori-II equivalent  to performance of over 20,000 K20X GPUs at 100% scaling Open Source: https://github.com/HPGeoC/awp-odc- os Single node performance comparison of AWP-ODC-OS on a variety of architectures. Also displayed is the bandwidth of each architecture, as measured by a STREAM and HPCG-SpMv. AWP-ODC-OS weak scaling on Cori Phase II and TACC Stampede KNL. We attain 91% scaling from 1 to 9000 nodes. The problem size required 14GB on each node. (Tobin, J., Breuer, A., Heinecke, A., Yount, C. and Cui, Y., Accelerating Seismic Simulations using the Intel Xeon Phi Knights Landing Processor, ISC High Performance'17, Frankfurt, June 18-22, 2017) https://www.hpcwire.com/off-the-wire/sdsc-achieves-record-performance-seismic-simulations-intel/

AWP-ODC on Sunway TaihuLight Tsinghua University/Wuxi SC Center ported open source AWP-ODC using Sunway OpenACC and fully optimized the code on TaihuLight Sustained 15-Pflop/s or 12.5% of the peak achieved, a ACM Gordon Bell Finalist in 2017 1976 Mw7.2 Tangshan earthquake scenario and high-fidelity simulation using 10 million cores A spatial resolution of 25m 320km x 320km x 60km Frequency of up to 10Hz Included non-linear near-fault physics (Haohuan Fu, Conghui He, Bingwei Chen, Zekun Yin, Zhenguo Zhang, Wenqiang Zhang, Tingjian Zhang, Wei Xue, Weiguo Liu, Wanwang Yin, Guangwen Yang, Xiaofei Chen (2017) 15-Pflops Nonlinear Earthquake Simulation on Sunway TaihuLight: Enabling Depiction of Realistic 10 Hz Scenarios, SC’17, Nov 13-16, Denver, 2017) Dr. Haohuan Fu of Tsinghua University/NSCW addressing Gordon Bell Weather and earthquake work at PASC’17 keynote

AWP-DM, Wavefield Estimation using a Discontinuous Mesh Interface (WEDMI) Motivation: Uniform-grid methods inefficient for large contrasts in seismic wave speeds, such as basin models. Challenges: Stability is inherently difficult to obtain in overlap between fine and coarse meshes. Approach: Factor-of-three contrast in grid spacing along all three dimensions (1), 4th-oder staggered grid (2,3). Status: Stable to 1M+ timesteps for factor-of-three velocity contrast inside overlap zone (4), accurate in realistic basin velocity models using finite fault sources in overlap zone (5,6), scalable to 1024+ GPUs (7), manuscript in press (Nie et al., 2017), student (Nie, MS, graduated) and researcher (Roten) supported from SI2. 4) 1) 2) 3) fine overlap zone coarse 5) 6) 7) Uniform Mesh DM Ideal Scaling DM (Nie, S., Wang, Y., Olsen, K.B. and Day, S.M., 4th-order Staggered-grid Finite Difference Seismic Wavefield Estimation using a Discontinuous Mesh Interface (WEDMI), Bull. Seism. Soc. Am. 2017, in press)

References In the News Publications Invited Talks 15-Pflop/s Nonlinear Earthquake Simulation on Sunway TaihuLight (using AWP-ODC), Gordon Bell Finalist in 2017, Tsinghua University/Wuxi SC Center, appeared on Tshinghua News, ISC’17 Invited Talk, PASC’17 Keynote SDSC Achieved Record Seismic Simulation Performance with Intel, appeared on HPCWire, insideHPC, NERSC news, Phys.Org News, UCSD News, SDSC facebook, diane bryant on Twitter Yifeng Cui Named SDSC Pi Person of 2016, SDSC Magazine, SCEC Twitter Publications Tobin, J., A. Breuer, C. Yount, A. Heinecke, Y. Cui, Accelerating Seismic Simulations Using the Intel Xeon Phi Knights Landing Processor, Proceedings of International Supercomputing ISC’17, June 18-22, Frankfurt, 2017 Nie, S., Y. Wang, K. Olsen and S. Day: 4th-order Staggered-grid Finite Difference Seismic Wavefield Estimation using a Discontinuous Mesh Interface (WEDMI), BSSA (accepted), 2017 Roten, D., Y. Cui, K. Olsen, S. Day, K. Withers, W. Savran, P. Wang and D. Mu, High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers, SC’16, Nov 13-18, Salt Lake City, 2016 Roten, D., K.B. Olsen, S.M. Day, Y. Cui, Quantification of Fault-Zone Plasticity Effects with Spontaneous Rupture Simulations, Pure Appl. Geophys., pp 1-23, doi:10.1007/s00024-017-1466-5, 2017 Invited Talks Breuer, A., Intel Booth, at ISC’17, June 18-22, Frankfurt, 2017 Cui, Y., High-frequency nonlinear earthquake simulations on Titan and Blue Waters, GTC’17, May 8-10, San Jose, 2017 Maechling, P., Earthquake Simulations at Extreme Scales, NVIDIA Technology Theater @ SC’16, Nov 13-18, Salt Lake City, 2016 Jordan, T., Earthquake Simulations at Extreme Scales, GTC-DC, Oct 26-28, Washington DC, 2016 Cui, Y., Regional scale earthquake simulations on OLCF Titan and NCSA Blue Waters, Perspectives of GPU Computing in Science 2016, Sept 26-28, Rome, 2016 (Keynote) Tobin, J., Breuer, A., Yount, C., Heinecke, A. and Cui, Y., Accelerating AWP-ODC-OS using Intel Xeon Phi Processors, Intel IXPUG Workshop, Sept 19-22, Chicago, 2016 (Keynote)