1. CADD: Component-Averaged Domain Decomposition Dan Gordon Computer Science University of Haifa Rachel Gordon Aerospace Engg. Technion January 13, 20081 Component-Averaged Domain Decomposition

2. January 13, 2008 Component-Averaged Domain Decomposition 2 Outline of Talk  CADD – Algebraic explanation  CADD – DD explanation  The Kaczmarz algorithm (KACZ)  KACZ  CARP (a CADD method)  Applications of CARP  CARP-CG: CG acceleration of CARP  Sample results

3. January 13, 2008 Component-Averaged Domain Decomposition 3 CADD – Algebraic Explanation – 1  The equations are divided into blocks (not necessarily disjoint)  Initial estimate: vector x=(x 1,…,x n )  Suppose x 1 is a variable (component of x) that appears in 3 blocks  x 1 is “cloned” as y 1, z 1, t 1 in the different blocks.  Perform one (or more) iterative operation(s) on each block (independently, in parallel)

4. January 13, 2008 Component-Averaged Domain Decomposition 4 CADD – Algebraic Explanation – 2  The internal iterations in each block produce 3 new values for the clones of x 1 : y 1 ’, z 1 ’, t 1 ’  The next iterative value of x 1 is x 1 ’ = (y 1 ’ + z 1 ’ + t 1 ’)/3  The next iterate is x’ = (x 1 ’,..., x n ’)  Repeat iterations as needed for convergence

5. January 13, 2008 Component-Averaged Domain Decomposition 5 CADD – non-overlapping domains xx y 0 1 1 domain A domain B external grid point of A clone of x 1

6. January 13, 2008 Component-Averaged Domain Decomposition 6 CADD – overlapping domains xx y 01 1 domain A domain B external grid point of A clone of x 1 A∩BA∩BA∩BA∩B

7. January 13, 2008 Component-Averaged Domain Decomposition 7 CADD – Parallel Implementation  Every processor is in charge of one domain  Mode: BLOCK-PARALLEL –All processors operate in parallel (each on its domain) –Processors exchange clone values –All values are updated –New values = average of clones

8. January 13, 2008 Component-Averaged Domain Decomposition 8 CADD Vs. Standard DD Methods  Difference is in handling external grid points:  CADD: A clone of the external grid point is modified by the domain's equations and contributes to new value of grid point.  Standard DD: Value of external grid point is fixed (contributes to RHS)

9. January 13, 2008 Component-Averaged Domain Decomposition 9 KACZ: The Kaczmarz algorithm  Iterative method, due to Kaczmarz (1937). Rediscovered for CT as ART  Basic idea: Consider the hyperplane defined by each equation  Start from an arbitrary initial point  Successively project current point onto the next hyperplane, in cyclic order

10. January 13, 2008 Component-Averaged Domain Decomposition 10 KACZ: Geometric Description eq. 1 eq. 2 eq. 3 initial point

11. January 13, 2008 Component-Averaged Domain Decomposition 11 KACZ with Relaxation Parameter  KACZ can be used with a relaxation parameter λ  λ=1: project exactly on the hyperplane  λ<1: project in front of hyperplane  λ>1: project beyond the hyperplane  Cyclic relaxation: Eq. i is assigned a relaxation parameter λ i

12. January 13, 2008 Component-Averaged Domain Decomposition 12 Convergence Properties of KACZ  KACZ with relaxation (0< λ <2) converges for consistent systems: –Herman, Lent & Lutz, 1978 –Trummer, 1981  For inconsistent systems, KACZ converges cyclically: –Tanabe, 1971 –Eggermont, Herman & Lent, 1981 (for cyclic relaxation parameters).

13. January 13, 2008 Component-Averaged Domain Decomposition 13 Another view of KACZ  Given the systemAx = b  Consider the "normal equations" systemAA T y = b, x=A T y  Well-known: KACZ is simply SOR applied to the normal equations  The relaxation parameter of KACZ is the usual relax. par. of SOR

14. January 13, 2008 Component-Averaged Domain Decomposition 14 CARP: Component-Averaged Row Projections  CARP is a CADD method with KACZ iterations in each domain:  The system Ax=b is divided into blocks B 1,…,B n (need not be disjoint)  Each processor is assigned one (or more) block  All the blocks are processed in parallel  Results from blocks are “merged” to form the next iterate  Merging is done by CADD averaging

15. January 13, 2008 Component-Averaged Domain Decomposition 15 Overview of CARP domain A domain B KACZiterationsKACZiterations averaging cloning KACZ in superspace (with cyclic relaxation)

16. January 13, 2008 Component-Averaged Domain Decomposition 16 Convergence of CARP  Averaging Lemma: the component- averaging and cloning operations of CARP are equivalent to KACZ row- projections in a certain superspace (with λ = 1)   CARP is equivalent to KACZ in the superspace, with cyclic relaxation parameters – known to converge

17. January 13, 2008 Component-Averaged Domain Decomposition 17 "Historical" Note  The term "component averaging" was first used by Censor, Gordon & Gordon (2001) for CAV and BICAV, used for image reconstruction in CT  These are Cimmino-type algorithms, with weights related to the sparsity of the system matrix  CADD, CARP use this concept in a different sense

18. January 13, 2008 Component-Averaged Domain Decomposition 18 CARP Application: Solution of stiff linear systems from PDEs  Elliptic PDEs w/large convection term result in stiff linear systems (large off-diagonal elements)  CARP is very robust on these systems, as compared to leading solver/preconditioner combinations  Downside: Not always efficient

19. January 13, 2008 Component-Averaged Domain Decomposition 19 CARP Application: Electron Tomography (joint work with J.-J. Fernández)  3D reconstructions: Each processor is assigned a block of consecutive slices. Data is in overlapping blobs.  The blocks are processed in parallel.  The values of shared variables are transmitted between the processors which share them, averaged, and redestributed.

20. January 13, 2008 Component-Averaged Domain Decomposition 20 CARP-CG: CG acceleration of CARP  CARP is KACZ in some superspace (with cyclic relaxation parameter)  Björck & Elfving (BIT 79): developed CGMN, which is a (sequential) CG- acceleration of KACZ (double sweep, fixed relax. parameter)  We extended this result to allow cyclic relaxation parameters  Result: CARP-CG

21. January 13, 2008 Component-Averaged Domain Decomposition 21 CARP-CG: Properties  Same robustness as CARP  Very significant improvement in performance on stiff linear systems derived from elliptic PDEs  Very competitive runtime compared to leading solver/preconditioner combinations on systems derived from convection-dominated PDEs  Improved performance in ET

22. CARP-CG on PDEs  CG acceleration of projection methods was done before, but with block projections, requiring multi- coloring. CARP-CG avoids this.  Tests were run on 9 convection- dominated PDEs, comparing CARP- CG with restarted GMRES, CGS and Bi-CGSTAB, with and without various preconditioners. Also tested: CGNR.  Domain: unit cube – 80x80x80. January 13, 2008 Component-Averaged Domain Decomposition 22

23. January 13, 2008 Component-Averaged Domain Decomposition 23 512,000 equations

24. January 13, 2008 Component-Averaged Domain Decomposition 24 512,000 equations

25. January 13, 2008 Component-Averaged Domain Decomposition 25 512,000 equations

26. January 13, 2008 Component-Averaged Domain Decomposition 26 512,000 equations

27. January 13, 2008 Component-Averaged Domain Decomposition 27 512,000 equations

28. January 13, 2008 Component-Averaged Domain Decomposition 28 CARP-CG: ET results

29. January 13, 2008 Component-Averaged Domain Decomposition 29 Future research on CADD  Different internal solvers: 1. Guaranteed convergence? 2. Symmetrizability?  Combine CADD with multigrid  CFD applications  Additional biomedical applications  CADD with preconditioners  Compare CADD with other DD methods, such as additive Schwarz

30. January 13, 2008 Component-Averaged Domain Decomposition 30 CARP Publications  CARP: SIAM J. Sci. Comput. 27 (2005) 1092-1117. http://cs.haifa.ac.il/~gordon/carp.pdf  Electron tomography: J. Par. & Dist. Comput. (2007), in press. http://cs.haifa.ac.il/~gordon/eltom.pdf  CARP-CG: Submitted for publication.