Parallel solution of the Helmholtz equation with high frequency

1
Parallel solution of the Helmholtz equation with
high frequency

• Dan Gordon
• Computer Science
• University of Haifa

Rachel Gordon Aerospace Eng. Technion
2
OUTLINE

• The wave equation
• The Kaczmarz algorithm (KACZ)
• KACZ ?? CARP (Component-Averaged Row Projections)
• CARP-CG CG acceleration of CARP
• Sample results with the Helmholtz equation

3
The Helmholtz Equation

• speed c frequency ?
• wave length l c/?
• wave number k 2p/l 2p?/c
• wave eqn -?u - k2u f
• Discretization with uniform grid size h
• No. of grid pts per l Ng l/h 2p/kh
• Considered desirable Ng 8, but Ng 6 also
  gave good results
• Linear system is complex and strongly indefinite

4
The Helmholtz Equation

• Challenging problem when ? is high
• Shifted Laplacian approach
• Bayliss, Goldstein Turkel, 1983
• introduced a shift into the Laplacian
• Erlangga, Vuik Oosterlee, 2004/06
• complex shift -?u - (1 - i b) k2 u f
• Uses multigrid to solve the preconditioner
• Not simple for irregular grids

5
The Helmholtz Equation

• Bollhöfer, Grote Schenk, 2009
• Introduced algebraic multilevel preconditioner
• Use symmetric max weight matchings and
  inverse-based pivoting
• Applied to heterogeneous 2D and 3D domains
• Can be parallelized in principle
• Apologies to many others!

6
The Kaczmarz algorithm (KACZ)
initial point
eq. 1
eq. 2
eq. 3
7
KACZ with Relaxation Parameter

• KACZ can be used with a relaxation parameter w
• w1 project exactly on the hyperplane
• wlt1 project in front of hyperplane
• wgt1 project beyond the hyperplane
• Cyclic relaxation eq. i is assigned a relaxation
  parameter wi

8
Algebraic formulation of KACZ

• Given the system Ax b
• Consider the "normal equations" system AATy b,
  x ATy
• Well-known fact KACZ is SOR applied to the
  normal equations
• The relaxation parameter of KACZ is the usual
  relax. par. of SOR

9
CARP Component-Averaged Row Projections

• A block-parallel version of KACZ
• The equations are divided into blocks (not
  necessarily disjoint)
• A variable shared by 2 or more blocks is "cloned"
  into its neighboring blocks.
• For each block (in parallel) do KACZ iterations
• Every shared variable becomes the average of its
  values in the different blocks
• Repeat until convergence

10
CARP-CG CG acceleration of CARP

• CARP is KACZ in some superspace (with cyclic
  relaxation parameters)
• Björck Elfving (BIT '79) developed CGMN, which
  is a (sequential) CG-acceleration of KACZ (double
  sweep, fixed relax. parameter)
• We extended this result to allow cyclic
  relaxation parameters
• Result CARP-CG

11
CARP-CG Properties

• On one processor, CARP-CG is identical to CGMN
• Particularly useful on systems with large
  off-diagonal elements
• example convection-dominated PDEs
• Discontinuous coefficients are handled without
  requiring domain decomposition (DD)

12
Robustness of CARP-CG

• KACZ inherently normalizes the eqns
• After normalization, the diagonal elements of
  AAT are larger than the off-diagonal ones (in
  each row)
• This is not diagonal dominance, but it makes the
  normal eqns manageable
• Normalization was also found to be useful for
  discontinuous coefficients

13
Application of CARP-CG to Helmholtz equation

• A fixed relaxation parameter of 1.5 was used in
  all cases
• Domain mostly unit square or unit cube
• 2nd order central difference scheme

14
Homogeneous 2D problem

• Based on Erlangga et al. '04, 6.2
• Eqn ?u k2u 0
• Domain unit square 0,1 ? 0,1
• Dirichlet bndry cond. on one side, with a
  discontinuity at midpoint
• 1st-order absorbing bndry cond. on other sides
  (Sommerfeld radiation condition)
• Grid points per l Ng 6, 8, 10
• No. of processors 1 32
• k (75), 150, 300, 450, 600

15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
Heterogeneous 2D problem

• 3-layer heterogeneous problem
• Based on Erlangga et al. '04, 6.3
• Everything is identical to previous problem
• EXCEPT

k600
k450
k300
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
The Marmousi Model

• Well-known benchmark for solvers of the Helmholtz
  equation
• 6000m x 1600m vertical slice of earth surface,
  disturbance at top center
• Highly heterogeneous and irregular
• Speed of sound 1500 m/s to 4000 m/s
• Tested on 12 node infiniband machine
• Each node 2 quad CPUs

24
Time (s) for rel-reslt10-7, Ng 7.5
grid freq. 1 proc 4 proc 8 proc 12 proc 16 proc 24 proc 32 proc
751 x 401 ?25 97 35 22 18 20 19 18
1501 x 401 ?50 786 262 144 106 107 85 77
2001 x 534 ?65 1868 627 334 237 253 190 158
25
(No Transcript)
26
(No Transcript)
27
3D heterogeneous problem

• Domain 0,1 ? 0,1 ? 0,1
• Divided into 3 layers with k60, 72, 90
• Point source in middle of one side
• Sommerfeld radiation condition on bndry
• Also tested with k60, 90, 145
• on the infiniband machine

28
(No Transcript)
29
(No Transcript)
30
Time (s) for 3D hom. het. problems,4 conv.
goals, on 1 xeon E5520 proc.
k Ng Grid 60 6.5 633 90 6.6 953 145 6.5 1513 60/90/145 6.5 15.7 1513
104 16 77 465 533
107 33 168 1024 1347
1010 51 258 1607 2159
1013 69 351 2209 2967
31
Summary serial and parallel machines

• When the frequency increases
• Faster convergence (on a fixed grid)
• Improved scalability (on a fixed grid)
• Improved speedup (with a fixed Ng)
• For fixed Ng no. of iterations is linear in k
• Homogeneous heterogeneous problems
• Simple to implement
• Generally useful for various problems with large
  off-diagonal elements and discontinuous
  coefficients

32
Other Potential Applications

• Higher order schemes for the Helmholtz equation
  (good initial results)
• Maxwell equations?
• Saddle-point problems?
• Circuit problems?
• Linear solvers in some eigenvalue methods?
• Suggestions are welcome!

33
Relevant Publications

• http//cs.haifa.ac.il/gordon/pub.html
• CARP SIAM J Sci Comp 2005
• CGMN ACM Trans Math Software 2008
• Microscopy J Parallel Distr Comp 2008
• Large convection discontin coef CMES 2009
• CARP-CG Parallel Comp 2010
• Scaling for discont coef J Comp Appl Math
  2010
• Helmholtz equation tech rept
  http//cs.haifa.ac.il/gordon/helm.pdf
• CARP-CG SOFTWARE AVAILABLE ON REQUEST
• THANK YOU!