1. Integrating Trilinos Solvers to SEAM code Dagoberto A.R. Justo – UNM Tim Warburton – UNM Bill Spotz – Sandia

2. SEAM (NCAR) SpectralElementAtmosphericMethod AztecOO AztecOO Epetra Epetra Nox Nox Ifpack Ifpack PETSc PETSc Komplex Komplex Trilinos (Sandia Lab)

3. AztecOO Solvers Solvers –CG, CGS, BICGStab, GMRES, Tfqmr Preconditioners Preconditioners –Diagonal Jacobi, Least Square, Neumann, Domain Decomposition, Symmetric Gauss-Seidel Matrix Free implementation Matrix Free implementation C++ (Fortran interface) C++ (Fortran interface) MPI MPI

4. Implementation SEAM CODE.. Pcg_solver. (F90) Pcg_solver. Aztec_solvers( ). (F90) Sub Aztec_solvers. AZ_Iterate( ) (C) Matrix_vector_C (C) Matrix_vector. (F90) Prec_Jacobi. (F90) Prec_Jacobi_C (C) AZTECAZTEC

5. Machines used Pentium III Notebook (serial) Pentium III Notebook (serial) –Linux, LAM-MPI, Intel Compilers Los Lobos at HPC@UNM Los Lobos at HPC@UNM –Linux Cluster –256 nodes –IBM Pentium III 750 MHz, 256 KB L2 Cache, 1 Gb RAM –Portland Group compiler –MPICH for Myrinet interconnections

6. Graphical Results from SEAM Energy Mass

7. Memory (in Mbytes per processor)

8. Speed Up From 1 to 160 processors. From 1 to 160 processors. Time of Simulation Time of Simulation 144 time iterations 144 time iterations x 300 s = 12 h simulation x 300 s = 12 h simulation Verify results using mass, energy,… Verify results using mass, energy,… –(Different result for 1 proc)

9. Speed Up – SEAM selecting # of elements ne=24x24x6

10. Speed Up – SEAM selecting order np=6

11. Speed Up – SEAM+Aztec best: cgs solver

12. Speed Up – SEAM+Aztec best: cgs solver + Least Square preconditioner

13. Speed Up – SEAM+Aztec increasing np -> increases speedup

14. Upshot – SEAM (One CG iteration)

15. Upshot – SEAM (matrix times vector communication)

16. Upshot – SEAM+Aztec (One CG iteration)

17. Upshot – SEAM+Aztec (Matrix times vector communication)

18. Upshot – SEAM+Aztec (Vector Reduction)

19. Time (24x24x6 elements, 2 proc.) SolverIter. Time (loop) Time/iter SEAM p=6 33.0 it 7.48 s 0.22 s/it SEAM p=12 56.9 it 81.2 s 1.42 s/it Cg p=6 87.1 it 28.2 s 0.32 s/it Cgs p=6 74.1 it 28.6 s 0.38 s/it Tfqmr p=6 75.2 it 31.1 s 0.41 s/it Bicg p=6 94.1 it 29.4 s 0.31 s/it Cgs ls p=6 35.1 it 42.0 s 1.19 s/it CG Jacobi p=6 45.8 it 17.2 s 0.37 s/it Cgs Jacobip=6 31.7 it 15.3 s 0.48 s/it Cgs p=12 60.4 it 274. S 4.53 s/it

20. Conclusions & Suggested Future Efforts SEAM+Aztec works! SEAM+Aztec works! SEAM+Aztec is 2x slower SEAM+Aztec is 2x slower  difference in CG algorithms  SEAM+Aztec time-iteration is 50% slower  0.1% of time lost in calls, preparation for Aztec. More time  better tune-up. More time  better tune-up. Domain decomposition Preconditioners Domain decomposition Preconditioners

21. SEAM + Aztec works! SEAM + Aztec works! More time  better tune-up. More time  better tune-up. Conclusions & Suggested Future Efforts