Parallel Preconditioners for the Incompressible Navier-Stokes Equations

1
Parallel Preconditioners for the Incompressible
Navier-Stokes Equations

• Robert Shuttleworth
• Applied Math Scientific Computation (AMSC)
• University of Maryland

2
Outline

• Background
• Incompressible Navier-Stokes Equations
• Discretization/Linearization
• Preconditioning the N-S Equations
• General Preconditioners
• N-S Problem Specific
• Pressure Correction Methods
• Pressure Convection-Diffusion
• High Performance Computing Issues
• Preliminary Results
• Lid driven cavity problem
• Flow over a diamond obstruction
• Conclusions

3
Motivation/Focus

• Motivation Efficient and robust solution of
  steady state and transient flow problems
• Develop fully implicit solution methods to the
  incompressible Navier-Stokes
• Solving the linear systems that arise can take
  upwards of 70 of the CPU time of a given
  simulation
• Linear Solvers Operator Based Block
  Preconditioning
• Focus Adapt block preconditioners to the linear
  subproblems that arise in realistic fluid flow
  problems

4
Introduction
Given the Incompressible Navier-Stokes Equations
Nonlinear Term Oseen Newton Jacobian of
Momentum Eq.
Discretization and Linearization
5
Discretization and Linearization
6
Complete Algorithm

• u(0) initial condition or initial guess
• p(0) initial condition or initial guess
• for m 1Ntimesteps
• u(m) u(m-1) , p(m) p(m-1)
• while F (u(m) ,u(m-1) ,p(m) ,u(m) ) gt
  ?nonlin
• ulag u(m)
• /
  /
• / Set up linear subproblem
  /
• /
  /
• / corresponding to F (u(m) ,u(m-1) ,p(m) ,ulag )
  0. /
• Iterate on A u(m) b until rk / r0 lt
  ?saddle
• Block Precondition

time loop
nonlinear loop
linear solver
7
General Preconditioning Premise

• Preconditioning needed in solution of any large
  scale PDE
• Bottleneck of solving N-S is the iterative
  solution of the linear systems
• Given
• Preconditioning speeds up convergence by
  improving the spectral properties of a matrix

Good
Cheaper
8
Types of Preconditioners

• General (Algebraic) Preconditioners
• Incomplete LU Factorization (ILU)
• Sparse Approximate Inverses (SPAI)
• Multigrid
• Domain Decomposition
• N-S Problem Specific Preconditioners
• Pressure Correction
• Pressure Convection-Diffusion

9
Incomplete LU Factorization (ILU)

• Factoring a sparse matrix by Gaussian Elimination
  generates fill-in. So, the L and U factors are
  less sparse than the original matrix.
• By ignoring the fill-in that occurs within a
  certain tolerance, approximate factors to L and U
  are available.
• Advantages simple to implement, inexpensive,
  good for certain problems
• Disadvantages potential instabilities, lack of
  scalability, not good for CFD applications, and
  difficulty in parallelization

10
Block Preconditioners

• Discretization
• Consider
• Optimal preconditioner is when X is the Schur
  Complement,
• Question How to approximate the Schur complement?

11
Pressure Correction (LD)U
So, we can apply a preconditioner to the saddle
point matrix of the form
12
Pressure Correction

• SIMPLER

Projection Matrix Enforces Incompressibility
13
Pressure Correction

• Advantages
• Used in both transient and steady state
• Easy to implement and parallelize
• Disadvantages
• Slower convergence coupling of physics is
  violated
• Choosing a relaxation parameter
• Inefficient for convection dominated flows

14
Pressure Convection-Diffusion L(DU)
Therefore, a right oriented preconditioner can be
applied to this problem
15
Pressure Convection Diffusion - Fp
Suppose
Suppose the velocity and pressure
convection-diffusion terms commute with one
another
Then,
16
Pressure Convection Diffusion - BFB
17
Pressure Convection-Diffusion

• Advantages
• Insensitive to mesh size, time step, and CFL
  number
• Minor Reynolds number dependence
• Solves coupled system
• Disadvantages
• Applications do not supply Fp
• Designed for Oseen iterations
• Equations for Stabilized FEM are not developed
• Boundary Conditions

18
Packages
Time Loop
Package
Methods
Component
Nonlinear Loop
Fluid Flow
Finite Element
MPSalsa (Epetra)
Linear Solver
Nonlinear Solver
NOX
Newton-Krylov Methods
Block Precond
Linear Solver
GMRESR
Aztec00 (Epetra, TSF)
block precondition
Meros (TSF)
End NonLin Loop
End Time Loop
F-1 GMRES/AMG X-1 CG/AMG
Aztec00, ML Epetra
19
Epetra Sparse Matrix Package

• Facilitates sparse matrix construction on both
  parallel and serial machines
• Compressed Row Storage (CRS)
• Double precision nonzero values are stored in
  contiguous memory locations
• Builds a map (graph) an array of integers
  corresponding to nonzero row/column entries
• Rows are stored in consecutive order

20
Epetra - Sparse Matrix Library

• Serial interfaces the BLAS
• Parallel - handles distributed matrix details
• Local versus global indices
• Global Map (graph) details which processor owns
  which entries
• Local Map (graph) details how local data (and
  ghosts) is represented
• Matrix-vector products
• Wrappers to Message Passing Interface (MPI)
• (Note The global map is normally determined by
  the user or other library.)

21
TSF Properties

• What is TSF?
• TSF Trilinos Solver Framework
• High level matrix/block matrix manipulation
  language
• Provides framework for integrating different
  solvers/preconditioners
• Interface for representation-independent solvers
• Why TSF?
• Abstract interfaces for vectors and operators
• Composable Block operators
• Deferred inverse and transpose
• Overloaded operators (matlab like syntax)
• Matlab-like simplicity, running on a
  supercomputer
• Transparent memory management

22
Benchmark Problems

• Lid Driven Cavity
• Contains many features of harder flows
• Steady and Unsteady Solutions
• Flow over a diamond obstruction
• Inflow/Outflow boundary conditions
• Harder flow
• MPSalsa
• Realistic massively parallel, chemically reactive
  fluid flow code

23
MPSalsa Steady Problem Results 2D Lid driven
cavity on a 64 x 64 grid
Re ILU Simplec Simple Fp
10 88.0 52.5 46.8 25.4
50 92.8 56.6 50.2 30.8
100 95.7 59.2 53.0 40.8
200 95.9 70.2 61.3 56.6
The values in each column represent the average
number of Outer Saddle Point Solves per Newton
Step.
Residual reduction for each of the
preconditioners.
24
MPSalsa Steady Problem Results 2D Lid driven
cavity
Re Mesh ILU Fp Proc
10 64 x 64 88.0 25.4 4
10 128 x 128 194.2 23.2 16
10 256 x 256 gt1200 23.4 64
100 64 x 64 95.7 40.8 4
100 128 x 128 335.3 40.7 16
100 256 x 256 gt1200 41.3 64
500 64 x 64 94.9 98.3 4
500 128 x 128 350.2 91.4 16
500 256 x 256 gt1200 92.2 64
Preliminary Time Comparison
Mesh Independence
25
Implementation Challenge

• Timings are not very good
• After profiling, upwards of 50 of the CPU time
  is spent in an inefficient memory allocation
  routine
• A multigrid smoother is inefficiently implemented
  

26
MPSalsa Steady Problem Results 2D Flow over a
Diamond Obstruction
Re Unknowns ILU Fp
10 16,000 45.0 36.0
64,000 110.8 37.8
250,000 332.2 36.2
25 16,000 45.5 48.8
64,000 101.7 51.5
250,000 297.2 47.0
The values in each column represent the average
number of Outer Saddle Point Solves per Newton
Step.
27
Future Work

• Sparse Approximate Commutator (SPAC) for Fp
• Compare/Optimize CPU Time amongst methods
• Tests on higher Re for both steady/time
  dependent problems
• More realistic problems
• 3D Problems
• Chemically reacting flow
• Turbulent flows

28
Conclusions

• Incompressible Navier-Stokes Equations
• Preconditioning the N-S Equations
• General Preconditioners
• Problem Specific
• Pressure Correction Methods
• Pressure Convection-Diffusion
• Preliminary Remarks
• ILU preconditioner does not scale well
• Fp preconditioner is mesh independent and
  competitive in CPU time

29
References

• A.J. Chorin, A numerical method for solving
  incompressible viscous problems, Journal of
  Computational Physics, 212,1967.
• H. C. Elman, D. J. Silvester and A. J. Wathen,
  Finite Elements and Fast Iterative Solvers,
  Oxford University Press, 2005. Howard Elman, V.
  E. Howle, John Shadid and Ray Tuminaro, A
  Parallel Block Multi-level Preconditioner for the
  3D Incompressible Navier-Stokes Equations.
  Journal of Computational Physics 187504-523,
  2003.
• D. Kay, D. Loghin, and A. J. Wathen, 2002, A
  preconditioner for the steady-state Navier-Stokes
  equations. SIAM J. Sci. Comput. 24, pp. 237-256.
• M. Pernice and M.D. Tocci, A multigrid-preconditio
  ned Newton Krylov method for the incompressible
  Navier-Stokes equations., SIAM J. Sci. Comput.
  123, pp. 398-418.
• S.V. Patankar, Numerical heat transfer and fluid
  flow, Hemisphere Pub. Corp, New York, 1980.