CADD: Component-Averaged Domain Decomposition

1
CADDComponent-AveragedDomain Decomposition

• Dan Gordon
• Computer Science
• University of Haifa

Rachel Gordon Aerospace Engg. Technion
2
Outline of Talk

• CADD Algebraic explanation
• CADD DD explanation
• The Kaczmarz algorithm (KACZ)
• KACZ ?? CARP (a CADD method)
• Applications of CARP
• CARP-CG CG acceleration of CARP
• Sample results

3
CADD Algebraic Explanation 1

• The equations are divided into blocks (not
  necessarily disjoint)
• Initial estimate vector x(x1,,xn)
• Suppose x1 is a variable (component of x) that
  appears in 3 blocks
• x1 is cloned as y1 , z1 , t1 in the different
  blocks.
• Perform one (or more) iterative operation(s) on
  each block (independently, in parallel)

4
CADD Algebraic Explanation 2

• The internal iterations in each block produce 3
  new values for the clones of x1 y1 , z1 ,
  t1
• The next iterative value of x1 is
• x1 (y1 z1 t1)/3
• The next iterate is
• x (x1 , ... , xn)
• Repeat iterations as needed for convergence

5
CADD non-overlapping domains

clone of x1
y
1
x
x
1
0
external grid point of A
domain A
domain B
6
CADD overlapping domains
domain B

clone of x1
y
x
x
1
0
1
external grid point of A
AnB
domain A
7
CADD Parallel Implementation

• Every processor is in charge of one domain
• Mode BLOCK-PARALLEL
• All processors operate in parallel (each on its
  domain)
• Processors exchange clone values
• All values are updated
• New values average of clones

8
CADD Vs. Standard DD Methods

• Difference is in handling external grid points
• CADD A clone of the external grid point is
  modified by the domain's equations and
  contributes to new value of grid point.
• Standard DD Value of external grid point is
  fixed (contributes to RHS)

9
KACZ The Kaczmarz algorithm

• Iterative method, due to Kaczmarz (1937).
  Rediscovered for CT as ART
• Basic idea Consider the hyperplane defined by
  each equation
• Start from an arbitrary initial point
• Successively project current point onto the next
  hyperplane, in cyclic order

10
KACZ Geometric Description
initial point
eq. 1
eq. 2
eq. 3
11
KACZ with Relaxation Parameter

• KACZ can be used with a relaxation parameter ?
• ?1 project exactly on the hyperplane
• ?lt1 project in front of hyperplane
• ?gt1 project beyond the hyperplane
• Cyclic relaxation Eq. i is assigned a relaxation
  parameter ?i

12
Convergence Properties of KACZ

• KACZ with relaxation (0lt ? lt2) converges for
  consistent systems
• Herman, Lent Lutz, 1978
• Trummer, 1981
• For inconsistent systems, KACZ converges
  cyclically
• Tanabe, 1971
• Eggermont, Herman Lent, 1981 (for cyclic
  relaxation parameters).

13
Another view of KACZ

• Given the system Ax b
• Consider the "normal equations" system AATy b,
  xATy
• Well-known KACZ is simply SOR applied to the
  normal equations
• The relaxation parameter of KACZ is the usual
  relax. par. of SOR

14
CARP Component-Averaged Row Projections

• CARP is a CADD method with KACZ iterations in
  each domain
• The system Axb is divided into blocks B1,,Bn
  (need not be disjoint)
• Each processor is assigned one (or more) block
• All the blocks are processed in parallel
• Results from blocks are merged to form the next
  iterate
• Merging is done by CADD averaging

15
Overview of CARP
domain A
domain B

averaging
KACZ iterations
KACZ iterations
cloning
KACZ in superspace (with cyclic relaxation)
16
Convergence of CARP

• Averaging Lemma the component-averaging and
  cloning operations of CARP are equivalent to KACZ
  row-projections in a certain superspace (with ?
  1)
• ? CARP is equivalent to KACZ in the superspace,
  with cyclic relaxation parameters known to
  converge

17
"Historical" Note

• The term "component averaging" was first used by
  Censor, Gordon Gordon (2001) for CAV and BICAV,
  used for image reconstruction in CT
• These are Cimmino-type algorithms, with weights
  related to the sparsity of the system matrix
• CADD, CARP use this concept in a different sense

18
CARP Application Solution of stiff linear
systems from PDEs

• Elliptic PDEs w/large convection term result in
  stiff linear systems (large off-diagonal
  elements)
• CARP is very robust on these systems, as compared
  to leading solver/preconditioner combinations
• Downside Not always efficient

19
CARP Application Electron Tomography(joint work
with J.-J. Fernández)

• 3D reconstructions Each processor is assigned a
  block of consecutive slices. Data is in
  overlapping blobs.
• The blocks are processed in parallel.
• The values of shared variables are transmitted
  between the processors which share them,
  averaged, and redestributed.

20
CARP-CG CG acceleration of CARP

• CARP is KACZ in some superspace (with cyclic
  relaxation parameter)
• Björck Elfving (BIT 79) developed CGMN, which
  is a (sequential) CG-acceleration of KACZ (double
  sweep, fixed relax. parameter)
• We extended this result to allow cyclic
  relaxation parameters
• Result CARP-CG

21
CARP-CG Properties

• Same robustness as CARP
• Very significant improvement in performance on
  stiff linear systems derived from elliptic PDEs
• Very competitive runtime compared to leading
  solver/preconditioner combinations on systems
  derived from convection-dominated PDEs
• Improved performance in ET

22
CARP-CG on PDEs

• CG acceleration of projection methods was done
  before, but with block projections, requiring
  multi-coloring. CARP-CG avoids this.
• Tests were run on 9 convection-dominated PDEs,
  comparing CARP-CG with restarted GMRES, CGS and
  Bi-CGSTAB, with and without various
  preconditioners. Also tested CGNR.
• Domain unit cube 80x80x80.

23
512,000 equations
24
512,000 equations
25
512,000 equations
26
512,000 equations
27
512,000 equations
28
CARP-CG ET results
29
Future research on CADD

• Different internal solvers
• 1. Guaranteed convergence?
• 2. Symmetrizability?
• Combine CADD with multigrid
• CFD applications
• Additional biomedical applications
• CADD with preconditioners
• Compare CADD with other DD methods, such as
  additive Schwarz

30
CARP Publications

• CARP SIAM J. Sci. Comput. 27 (2005) 1092-1117.
• http//cs.haifa.ac.il/gordon/carp.pdf
• Electron tomography J. Par. Dist. Comput.
  (2007), in press.
• http//cs.haifa.ac.il/gordon/eltom.pdf
• CARP-CG Submitted for publication.