1. Visualizing large scale earthquake simulations Amit Chourasia Visualization Scientist San Diego Supercomputer Center Presented to: Advanced User Support, TeraGrid May 06, 2009

2. Slip deficit on the southern SAF since last event (1690): 315 years x 16 mm/year = 5.04 m -> M w 7.7 1857 M 7.9 ~1690 M 7.7 Major Earthquakes on the San Andreas Fault, 1690-present1906 M 7.8 146+91-60 yrs 220±13 yrs Slide: Courtesy Kim Olsen

3. Simulation Domain

4. TeraShake-2 Data Flow TS2.dyn.200m 30x 256 procs, 12 hrs, TG IA-64 GPFS Okaya 200m Media Okaya 100m Media 100m Reformatting 100m Transform 100m Filtering 200m moment rate SDSC IA-64 TS2.dyn.100m 10x 1024 procs, 35 hrs Initial 200m Stress modify Initial 100m Stress modify TS2.wav.200m 3x 1024 procs, 35 hrs NCSA IA-64 Datastar p690 Datastar p655 Visualization Analysis Network TG IA-64 GPFS-wan NCSA-SAN SDSC-SAN Velocity mag. & cum peak Displace. mag & cum peak Seismograms Registered to Digital Library SRB SAM-QFS HPSS Datastar GPFS Slide: Courtesy Yifeng Cui

5. Data  Domain Resolution: 3000x1500x400 uniform  3 scalar fields (2D and 3D, floating point)  Derived data (magnitudes, vorticity, etc)  Temporal (22727 steps)  Size 10-50TB per simulation (potentially 430 TB)  Input characteristics (volumetric stiffness, etc)

6. General Questions & Problems  Regions of impact  Occurrence of peak velocities and there location  Understand wave propagation  How to verify and validate ?  How to analyze and gain scientific insight ?  How to share information among experts and others?

7. Visualization

8. Comparative Surface Viz  Movie # 1 Movie # 1

9. Map service portal 9

10. Topography Visualization  Wave propagation in 3d  Topography with wave propagation  Movie # 2 Movie # 2

11. Topography Visualization  Wave propagation in 3d  Topography with wave propagation  Movie # 3 Movie # 3

12. Cross Sectional 12 TeraShake2.1: Velocity Magnitude and Sliprate

13. Volumetric 13 TeraShake 2.2: Velocity-X Movie # 4

14. Design Evolution 14

15. Does Visualization help?  Diagnosis, monitoring and verification  Identification of fairly simple aggregate behavior of the phenomenon (wavefields) that could not be guessed at by simply examining standard output  Integrate disparate data  Makes the results palatable to broader audience

16. Maps 16 TeraShake1.3: Spectral Acceleration at 3.0 sec

17. Self Contouring Technique 17 TeraShake1.3: Spectral Acceleration at 3.0 sec

18. Lessons Learned (TeraShake)  In the north-westward propagating rupture scenario the wave propagation is strongly guided toward the Los Angeles basin after leaving the San Andreas Fault (unexpected)  The sediment filled basin acts as an amplification source for trapped waves. A Strong amplification is observed in the LA basin long after the initial rupture (unexpected)  Contiguous basins act as energy channels, enhancing ground motion in parts of the San Gabriel and Los Angeles basins.  Identification of regions of particularly strong shaking  Validation of input rupture model and instability identification  Observation of star burst patterns in the Spectral Amplification maps (unexpected) Over 130,0000 images, Consumed 40,000 hrs of compute time, More than 50 unique animations 18

19. Software  Scalable Visualization Toolkit* (batch volume rendering)  Deskvox, Meshviewer (interactive volume rendering)  Storage Resource Broker* (SRB)  Maya  After Effects  Teem toolkit  Google Earth * TeraGrid friendly

20. References  ShakeOut-D: Ground Motion Estimates Using an Ensemble of Large Earthquakes on the Southern San Andreas Fault With Spontaneous Rupture Propagation, Geophysical Research Letters, 2009  The TeraShake Computational Platform for Large-Scale Earthquake Simulations, Advances in Geocomputing, Lecture Notes in Earth Sciences, 2009  TeraShake2: Spontaneous Rupture Simulations of Mw 7.7 Earthquakes on the Southern San Andreas Fault, Bulletin of the Seismological Society of America, 2008  Toward petascale earthquakes simulations, Acta Geotechnica, 2008  Visual insights into high resolution earthquake simulations, IEEE Computer Graphics and Applications, 2007  Enabling very-large scale earthquake simulations on parallel machines. Computational Science – ICCS 2007  Strong shaking in Los Angeles expected from southern San Andreas earthquake, Geophysical Research Letters, 2006

21. Acknowledgements  Kim Olsen, Steve Day, Bernard Minster, Tom Jordan, Phil Maechling, Steve Cutchin, Yifeng Cui, Reagan Moore and more….. Southern California Earthquake Center (SCEC) SDSC, NCSA, TACC, NICS

22. Thanks for your patience Q&A Websites: http://scec.org http://www.sdsc.edu/us/visservices/projects/scec/ http://scec.org http://www.sdsc.edu/us/visservices/projects/scec/ (Visualization) Movie # 5 Movie # 6