1. SI2 Project Highlights
SI2-SSI: Community Software for Extreme-Scale Computing in Earthquake System Science
PIs: Thomas H. Jordan, Yifeng Cui, Kim B. Olsen, Ricardo Taborda
Project Dates: 1 September 2015 through August 31, 2019
Award Number: ACI
SCEC Extreme Scale Earthquake Simulation Software
Recent Accomplishments
19 July 2017
Thomas H. Jordan
Yifeng Cui
Kim Bak Olsen
Ricardo Taborda

2. 6.5x AWP-ODC on NVIDIA GPUs
A first 4-Hz nonlinear M7.7 earthquake simulation on the southern San Andreas Fault conducted using 4,200 Blue Waters GPUs
100% of parallel efficiency achieved for both linear/ nonlinear versions of AWP-ODC up to 8,192 GPUs
Accelerated time-to-solution from original nonlinear 0.68sec to 0.29sec per iteration
Blue Waters PAID project provided additional support.
Snapshots from the 4 Hz San Andreas simulation. (a-c) and (d-f) show fault-parallel velocity for the linear and nonlinear cases, respectively, and (g-i) depict the evolution of permanent plastic strain at the surface obtained from the nonlinear simulation. The dashed line shows the fault trace.
(Roten, D., Y. Cui, K. Olsen, S. Day, K. Withers, W. Savran, P. Wang and D. Mu, High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers, SC’16, 1-10, Nov 13-18, Salt Lake City, 2016)
6.5x
Speedup of CyberShake SGT version on Cray XK7 compared to XE6 at node-to-node level
(Roten et al., SC’16) (

3. NVIDIA Share Your Science Video – SC’16
(Roten et al., SC’16)

4. AWP-ODC on Intel Xeon Phi
Stencil generation and vector folding through YASK tool:
Hybrid placement of grids in DDR and MCDRAM
Normalized cross architecture evaluation in Mega Lattice Updates per Second (MLUPS): Xeon Phi KNL 7290 achieves 2x speedup over NVIDIA K20X, 97% of NVIDIA Tesla P100 performance
Performance on 9,000 nodes of Cori-II equivalent  to performance of over 20,000 K20X GPUs at 100% scaling
Open Source: os
Single node performance comparison of AWP-ODC-OS on a variety of architectures. Also displayed is the bandwidth of each architecture, as measured by a STREAM and HPCG-SpMv.
AWP-ODC-OS weak scaling on Cori Phase II and TACC Stampede KNL. We attain 91% scaling from 1 to 9000 nodes. The problem size required 14GB on each node.
(Tobin, J., Breuer, A., Heinecke, A., Yount, C. and Cui, Y., Accelerating Seismic Simulations using the Intel Xeon Phi Knights Landing Processor, ISC High Performance'17, Frankfurt, June 18-22, 2017)

5. AWP-ODC on Sunway TaihuLight
Tsinghua University/Wuxi SC Center ported open source AWP-ODC using Sunway OpenACC and fully optimized the code on TaihuLight
Sustained 15-Pflop/s or 12.5% of the peak achieved, a ACM Gordon Bell Finalist in 2017
1976 Mw7.2 Tangshan earthquake scenario and high-fidelity simulation using 10 million cores
A spatial resolution of 25m
320km x 320km x 60km
Frequency of up to 10Hz
Included non-linear near-fault physics
(Haohuan Fu, Conghui He, Bingwei Chen, Zekun Yin, Zhenguo Zhang, Wenqiang Zhang, Tingjian Zhang, Wei Xue, Weiguo Liu, Wanwang Yin, Guangwen Yang, Xiaofei Chen (2017) 15-Pflops Nonlinear Earthquake Simulation on Sunway TaihuLight: Enabling Depiction of Realistic 10 Hz Scenarios, SC’17, Nov 13-16, Denver, 2017)
Dr. Haohuan Fu of Tsinghua University/NSCW addressing Gordon Bell Weather and earthquake work at PASC’17 keynote

6. AWP-DM, Wavefield Estimation using a Discontinuous Mesh Interface (WEDMI)
Motivation: Uniform-grid methods inefficient for large contrasts in seismic wave speeds, such as basin models.
Challenges: Stability is inherently difficult to obtain in overlap between fine and coarse meshes.
Approach: Factor-of-three contrast in grid spacing along all three dimensions (1), 4th-oder staggered grid (2,3).
Status: Stable to 1M+ timesteps for factor-of-three velocity contrast inside overlap zone (4), accurate in realistic basin velocity models using finite fault sources in overlap zone (5,6), scalable to GPUs (7), manuscript in press (Nie et al., 2017), student (Nie, MS, graduated) and researcher (Roten) supported from SI2. 4) 1) 2) 3)
fine
overlap zone
coarse 5) 6) 7)
Uniform Mesh DM
Ideal Scaling DM
(Nie, S., Wang, Y., Olsen, K.B. and Day, S.M., 4th-order Staggered-grid Finite Difference Seismic Wavefield Estimation using a Discontinuous Mesh Interface (WEDMI), Bull. Seism. Soc. Am. 2017, in press)

7. References In the News Publications Invited Talks
15-Pflop/s Nonlinear Earthquake Simulation on Sunway TaihuLight (using AWP-ODC), Gordon Bell Finalist in 2017, Tsinghua University/Wuxi SC Center, appeared on Tshinghua News, ISC’17 Invited Talk, PASC’17 Keynote
SDSC Achieved Record Seismic Simulation Performance with Intel, appeared on HPCWire, insideHPC, NERSC news, Phys.Org News, UCSD News, SDSC facebook, diane bryant on Twitter
Yifeng Cui Named SDSC Pi Person of 2016, SDSC Magazine, SCEC Twitter
Publications
Tobin, J., A. Breuer, C. Yount, A. Heinecke, Y. Cui, Accelerating Seismic Simulations Using the Intel Xeon Phi Knights Landing Processor, Proceedings of International Supercomputing ISC’17, June 18-22, Frankfurt, 2017
Nie, S., Y. Wang, K. Olsen and S. Day: 4th-order Staggered-grid Finite Difference Seismic Wavefield Estimation using a Discontinuous Mesh Interface (WEDMI), BSSA (accepted), 2017
Roten, D., K.B. Olsen, S.M. Day, Y. Cui, Quantification of Fault-Zone Plasticity Effects with Spontaneous Rupture Simulations, Pure Appl. Geophys., pp 1-23, doi: /s , 2017
Roten, D., Y. Cui, K. Olsen, S. Day, K. Withers, W. Savran, P. Wang and D. Mu, High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers, SC’16, Nov 13-18, Salt Lake City, 2016
Invited Talks
Breuer, A., Intel Booth, at ISC’17, June 18-22, Frankfurt, 2017
Cui, Y., High-frequency nonlinear earthquake simulations on Titan and Blue Waters, GTC’17, May 8-10, San Jose, 2017
Maechling, P., Earthquake Simulations at Extreme Scales, NVIDIA Technology SC’16, Nov 13-18, Salt Lake City, 2016
Jordan, T., Earthquake Simulations at Extreme Scales, GTC-DC, Oct 26-28, Washington DC, 2016
Cui, Y., Regional scale earthquake simulations on OLCF Titan and NCSA Blue Waters, Perspectives of GPU Computing in Science 2016, Sept 26-28, Rome, 2016 (Keynote talk)
Tobin, J., A. Breuer, C. Yount, A. Heinecke, Y. Cui, Accelerating AWP-ODC-OS using Intel Xeon Phi Processors, Intel IXPUG Workshop, Sept 19-22, Chicago, 2016 (Keynote talk)