1. Danny Dunlavy, Andy Salinger Sandia National Laboratories Albuquerque, New Mexico, USA SIAM Parallel Processing February 23, 2006 SAND2006-1075C Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. Preconditioners for the Space-Time Solution of Large-Scale PDE Applications

2. SIAM Parallel Processing 2006 Motivation Large-scale Transient Applications Space-Time Formulations – Transient calculations: Initial conditions and parameter – Space-time formulations: Parallelism in time (and space) Intermediate/final values Integrated values Periodic orbits – Applications Current: Fluid flow (MPSalsa) Planned: Semiconductor devices (Charon) Fluid/structure problems (Aria/Sierra)

3. SIAM Parallel Processing 2006 Space-Time Formulation Transient Simulation of: First solve: Then solve: Instead, solve for all solutions at once: where … and with Newton solve: Solve system with GMRES (right preconditioning)

4. SIAM Parallel Processing 2006 Space-Time Preconditioners Global Sequential Parallel Block Diag “Parareal” (Multilevel) = Solve/Precondition = Multiply, Add

5. SIAM Parallel Processing 2006 proc 0: proc 1: proc 0: proc 1: proc 3: proc 2: Space and Time Partitioned Independently Ex: 4 Time Steps on 4 Procs Spatial DomainsSpace-Time Domains Proc 0: Proc 1: Proc 3: Proc 2: Each processor owns 1 time step for the entire spatial domain Each processor owns 4 time steps for ¼ of the spatial domain Each processor owns 2 time steps for ½ of the spatial domain proc 0:

6. SIAM Parallel Processing 2006 Preliminary Analysis – Computational Time Time Integration Sequential (preconditioning only, 1 time domain) Sequential (preconditioning only, N proc time domains) Parallel (N proc time domains) Parareal (N proc time domains) Global (N proc time domains)

7. SIAM Parallel Processing 2006 Demonstration Problem Frank-Kamenetskii explosion model –Extended to include reactant consumption term –5 scalar PDEs –5 unknowns: insulated axis of symmetry

8. SIAM Parallel Processing 2006 Numerical Experiments Methods – MPSalsa: FEM: 64 x 48 elements, time steps: 32, unknowns: 509,600 – Trilinos: Newton (NOX) : 4–7 iterations GMRES (Aztec) : 400 max. outer, 200 max. inner iterations ILUk (Ifpack) : k=1 (fill) Continuation in (LOCA): 1 step Fixed Number of Spatial Domains (4) – Processors: 4 8 16 32 64 128 – Time Domains: 1 2 4 8 16 32 – How much can parallelism in time speed up the solve? Fixed Number of Processors (32) –Spatial domains: 12481632 –Time domains: 32168421 – How can space-time parallelism be used most effectively?

9. SIAM Parallel Processing 2006 Results – Fixed Number of Spatial Domains (4) Processors48163264128 Time Domains12481632 Sequential (1e-6, P)236164131115108104 Sequential (1e-2, P)21713994746765 Sequential (P, 1e-3)931636477380352357 Parallel (1e-6, 1e-3)3312101481169893 Parallel (P, 1e-3)9434772461086153 Block Diag (P, 1e-3)10275232631106453 Global (1e-3)9584912441055746 Parareal (1e-6, P)237112145119 Parareal (P, 1e-3)950277181106 Preconditioner (block solve tolerance, GMRES tolerance); P = preconditioning only

10. SIAM Parallel Processing 2006 Results – Fixed Number of Spatial Domains (4) Best Results Sequential (1e-2, P) Parallel (P, 1e-3) Global (1e-3)

11. SIAM Parallel Processing 2006 Results – Fixed Number of Processors (32) Spatial Domains32168421 Time Domains12481632 Sequential (1e-6, P)727187100168122 Sequential (1e-2, P)5552596610384 Sequential (P, 1e-3)551310339359548625 Parallel (1e-6, 1e-3)1179599107154170 Parallel (P, 1e-3)5482171621358470 Block Diag (P, 1e-3)5502041611378869 Global (1e-3)3651721431258157 Parareal (1e-6, P)7075110226 Parareal (P, 1e-3)551188184399 Preconditioner (block solve tolerance, GMRES tolerance); P = preconditioning only

12. SIAM Parallel Processing 2006 Summary Conclusions – Several preconditioners improve performance of space-time solves – Achieve time parallelism for serial codes (fixed spatial domains) Future Work – More time steps (study limits of time parallelism) – Comparison of analysis to experimental timing results – Periodic orbit tracking – Initial guesses for Newton (mesh refinement/preconditioning) – Other time discretizations (p-refinement) – Adaptive time steps (r-adaptivity) and time domain partitioning

13. SIAM Parallel Processing 2006 Thank You MS44 – Parallel Space-Time Algorithms Friday, 9:45 – 11:45 AM (Carmel Room) Space-Time Solution of Large-Scale PDE Applications Andy Salinger, 11:15 – 11:40 AM Danny Dunlavy dmdunla@sandia.gov Andy Salinger agsalin@sandia.gov