1. Parallel Algorithms for Solution of Large Sparse Linear Systems with Applications Murat Manguoğlu Middle East Technical University, Ankara, Turkey Prace workshop: HPC approaches on life sciences and chemistry February 17, 2012 Sofia, Bulgaria

2. Target Computational Loop Integration Newton iteration Linear system solvers kk kk tt Most time consuming part in large simulations

3. Design principles Reducing memory references and interprocessor communications at the cost of extra arithmetic operations. Allowing multiple levels of parallelism. Creating many algorithms – versions vary depending on architectural characteristics of the underlying computing platform.

4. DS factorization vs LU factorization A = U D L S ? 0 0 0 0 D -1 A

5. Parallel solution of banded linear systems (SPIKE algorithm) A. H. Sameh, D. J. Kuck, On stable parallel linear system solvers, J. ACM 25 (1) (1978) 81–91.

7. Solving Ax=f Solve Mz = r using DS fact. (M is “banded”) BiCGstab (or any other Krylov subspace method) Step 1: weighted matrix reordering (symmetric/nonsymmetric) Step 2: extract a “banded” preconditioner,M, and solve the system: Manguoglu M, Koyuturk M, Sameh A, Grama A, SIAM Journal on Scientific Computing, 32(3), pp.1201- 1216, 2010

8. reordering with weights and extracting the preconditioner A1A1 A2A2 A3A3 A4A4 C2C2 C3C3 C4C4 B1B1 B2B2 B3B3 PSPIKE M. Manguoglu, A. Sameh, O. Schenk, A parallel hybrid sparse linear system solver, LNCS - Proceedings of EURO-PAR09 5704 (2009) 797–808.

9. Eigenvalues of the Laplacian of a Graph Case 1: – A is a symmetric matrix of order n – B = A – The weighted Laplacian matrix L is given by: L(i,i) = ∑ |B(i,k)| ; for k = 1,2,…,n; k ≠ i L(i,j) = - |B(i,j)| ; for i ≠ j Case 2: – A is nonsymmetric – B= (|A|+|A T |)/2 – L is obtained as in Case 1.

10. The Fiedler vector Obtain the eigenvector of the second smallest eigenvalue of: L x = λ x ; λ:= {0 = λ 1 ≤ λ 2 ≤ ……≤ λ n } Minimize σ = ∑ |B(i,j)| (x(i) – x(j)) 2 for all i, j for which B(i,j) ≠ 0, such that: e T x = 0, and x T x = 1. In other words: σ = x T Lx Miroslav Fiedler, Algebraic connectivity of graphs(English), Czechoslovak Mathematical Journal, vol. 23 (1973), issue 2, pp. 298-305

11. Trace Minimization (TraceMin) A x = λ B x – A :=symmetric; B:= symmetric positive definite Minimize tr(Y T A Y) s.t. (Y T BY)=I p Solution: min tr(Y T AY) = Σ λ j (j=1,2,..,p) λ 1 ≤ λ 2 ≤ λ 3 ≤ …≤ λ p < λ p+1 ≤ ……≤ λ n A. Sameh & J. Wisniewski: SIAM J. Num. Analysis, 1982 + A. Sameh & Z. Tong: J. Comp. Appl. Math., 2000.

12. TraceMin iteration

13. A Parallel Algorithm for Computing the Fiedler Vector: TraceMin-Fiedler Based on the Trace Minimization algorithm computes λ 2 and v 2 of L x = λ x – Forms the small Schur complement explicitly – Solution of the linear systems involving L via CG with diagonal preconditioning – L is not spd (it is spsd) Comparison against the best sequential scheme (Harwell Subroutine MC73) Manguoglu M, Cox E, Saied F, Sameh A:, “TRACEMIN-Fiedler: A Parallel Algorithm for Computing the Fiedler Vector”, Lecture Notes in Computer Science (proceedings of VECPAR10), Volume 6449, pp.449-455, 2011 Source code(GPL): www.cs.purdue.edu/homes/mmanguog/fiedler.html

14. Most time consuming part

15. f2: structral mechanics N: 71,505 NNZ: 5,294,285 Original matrixAfter MC73After TraceMin-Fiedler

17. Platform: a cluster where each node has two Intel (Westmere) X5670@2.93Ghz / 12 cores per nodeX5670@2.93Ghz Number of cores (using at most 8 cores per node) Stopping tolarence: ||Lx 2 -λ 2 x 2 || ∞ /||L|| ∞ ≤ 10 -5

18. 2 nd approach for solving Ax=f Solve Mz = r using DS factorization BiCGstab (or any other Krylov subspace method) Step 1: extract a “sparse” preconditioner,M, and solve the system: Manguoglu M, Journal of Computational and Applied Mathematics, 236(3), pp.319-325, 2011

19. Parallel solution of sparse linear systems (Domain Decomposing Parallel Solver - DDPS)

21. 3D lid-driven cavity problem ● Standard test problem used in CFD community, problem becomes difficult as the Reynold's number increase ● We use OpenFOAM to extract linear systems with 4 million unknowns and a Reynold's number of 5000 ● Curie cluster located at CEA, France is used. Curie consists of 360 “fat” nodes where each node has 4 eight core Intel EX X7560 processors running at 2.26 GHz and 128 GB of memory. The nodes are connected with an InfiniBand QDR Full Fat Tree network.

22. DDPS solver parameters: ● BiCGStab as the inner and outer iterative solver ● Incomplete-LU factorization (not threaded) with drop tolerance 10 -2 and fill-in 5 for the diagonal blocks ● outer stopping tolerance is 10 -10 and inner stopping tolerance is 10 -2

26. DDPS variations DDPS-D : use the direct solver Pardiso for D -1 A DDPS-I1: use ILUT(10 -1, 5) for D -1 A DDPS-I3: use ILUT(10 -3, 5) for D -1 A DDPS-I4: use ILUT(10 -5,10) for D -1 A In all cases the outer stopping tolerance is 10 -7 and inner stopping tolerance is 10 -5

27. Flow field at time level 78 of the FSI computation. Stream ribbons are colored by the velocity magnitude, with the colors ranging from blue (low velocity) to red (high velocity). Manguoglu M, Takizawa K, Sameh A, Tezduyar T, Computational Mechanics, 48, pp.377-384, 2011

28. Volumetric flow rate, with the triangle representing the instant (time level 78+0.5) when the test data is extracted. Linear systems to solve at each nonlinear iteration has 1,399,566 unknowns and the coefficient matrix 167,638, 284 nonzeros

29. cluster which consists of two Intel X5670 processors per node (total 12 cores per node) and 24 GB of ram. Computing platform

30. 1 st nonlinear iteration solution time (in seconds)

31. 3 rd nonlinear iteration solution time (in seconds)

32. Robustness of DDPS-D - systems from UF sparse matrix collection-

33. Robustness of DDPS-D - number of iterations -

34. Conclusions ● New factorization : DS vs. LU ● ● DDPS-D/DDPS-I solvers are scalable and robust involving inner-outer iterations and hence flexible ● Hybrid: combining direct and iterative solvers ● Hybrid: can use MPI/OpenMP at the same time

35. Acknowledgments I would like to thank Ahmed Sameh, Ananth Grama, Eric Cox, Faisal Saied, Olaf Schenk, Madan Sathe, Joerg Hertzer, Kenji Takizawa, and Tayfun Tezduyar for the numerous discussions and for their support. This work has been partially supported by the European Community's Seventh Framework Programme (FP7/2007- 2013) under grant agreement no: RI-261557 and METU BAP-08-11-2011-128 grant

36. Thank you!