Adaptive Simulation Loop for Omega3P

1
Adaptive Simulation Loop for Omega3P
X. Li, M.S. Shephard, E. Seol, A.C. Bauer, Y. Luo
Scientific Computation Research Center,
Rensselaer Polytechnic Institute
L. Ge, I. Malik, K. Ko, Z. Li
Stanford Linear Accelerator Center (SLAC), DOE
and Stanford University
A Collaboration under DOEs SciDAC program

• Outline
• Issues, goals and overall approach
• Ingredients to constructing a serial adaptive
  loop
• Results to date
• Efforts on the parallel adaptive loop

TSTT
2
Constructing Adaptive Solution Procedures

• Components of an an adaptive mesh control loop
• High level problem definition
• Spatial discretization (mesh generation)
• Equation discretization
• Solvers to deal with the large systems of
  equations
• Error estimators
• Correction indicators
• Discretization enrichment methods
• Implementation
• Tightly coupled using a single set of structures
• Advantage is more optimal and computationally
  efficient if done well
• Disadvantage is complex algorithm and code
  development
• More important with frequent adaptation steps
• Loosely coupled building on existing components
• Advantage is ability to take advantage of
  existing analysis codes and adaptive tools and to
  mix and match tools
• Disadvantage is the overhead of multiple
  structures and data conversion

3
Mesh/Model Relationship

• Critical to all aspects of the simulation process
• Relationship is termed classification
• Mesh Classification Unique association of a
  mesh entity, Midi, to a geometric model entity,
  Gjdj, where diltdj is denoted by
• Midi Gjdj
• indicates the left-hand entity (or set)
  represents a portion of the right-hand entity in
  the discretization
• Multiple Midi classified on a Gjdj
• Boundary mesh entities are identifiedin terms of
  their classifications
• Classification critical to supporting adaptive
  simulations and high level problem definitions

4
Adaptive Mesh Control

• Two approaches
• Re-meshing
• Define new mesh size field
• Provide mesh generator with domain definition and
  mesh size field
• Map any required history dependent solution
  fields for the old to new mesh
• Local mesh modifications
• Determine local operations needed to perform
  desired modifications
• Execute the local modifications
• Apply incremental history dependent solution
  field updates
• Used generalized mesh modification

5
Muzzle Blast

t0.0
t2e-4
t5e-4
6
Mesh Adapted Using Mesh Modification Tool

inlet
initial mesh
B
B
adapted mesh
section B-B
interior mesh on xz plane
velocity contours on xz plane
7
Field Interface Procedures

• Fields are tensors defined in terms of basis and
  multipliers defined over mesh entities
• Solution information needed for queries and
  operations - in this case error estimation
• Maintain representation over the problem domain
  in a manner that is consistent with the solution
  process
• Operators needed to access fields at the model
  and mesh
• Used to support error estimation
• Used in other projects to support global and
  local solution transfer

8
Adaptive Loop for Accelerator Design

• Complex geometry defined in CAD modeler
• Physics modeling by the Omega3P code from SLAC
• Solves Maxwells equations in frequency domain
• Addresses Electric and magnetic fields
• Determine power loss for specific modes
• High level modeling accuracy needed
• E.g., 0.01 error in frequency predictions
• Adaptive mesh control needed to provide accuracy
  needed
• Serial adaptive loop constructed using unmodified
  Omega3P
• Parallel adaptive loop may require adding a
  parallel file

TSTT
9
Providing the Adaptive Loop Components

• Components added to construct adaptive loop
• Automatic mesh generation directly from CAD
• a-posteriori error indication procedures to
  define mesh size field
• General mesh modification procedures to construct
  requested mesh size field accounting for the CAD
  geometry
• Approach to the coupling of the components
• Interoperability procedures to provide
  information needed by components
• Geometry - support the integration to multiple
  CAD systems
• Mesh - support the integration to various mesh
  generation and modification procedures
• Field - support the integration with different
  analysis procedures
• Interoperability procedures used in this example
  from
• TSTT
• SCOREC
• Simmetrix

10
Integration of Components
11
Operator Interfaces
Using geometry operatorsmeans alternate solid
modelers can be inserted
Using TSTT mesh operatorsmeans alternate mesh
generatorsand mesh adaptation procedurescan be
inserted
Mesh adaptation based on local modification
linkeddirectly to CAD
Unaltered SLAC code
Projection-based error estimatorused to
construct new mesh sizefield given to mesh
modification
Error estimatorsfrom RPI and SLAC
12
Error Indicator/Estimator
Projection-based error Indicator
Total error Where error in element
FE solution recovered gradient
of FE solution

• indicates where to adapt
• provides a stopping criterion

13
Convert Error Field to Size Field
Size field is obtained by minimizing and
utilizing a priori error estimate
Where Lagrange multiplier desired
error tolerance problem
dimension error in element current size of
element
The number of elements is minimized for a given
error tolerance by equally distributing the
error over the mesh
14
Adaptive Results for Trispal Model
Distribution of Wall-loss
Level 1
Level 2
Level 0
Frequency Convergence
Q Convergence
15
Need for Parallel Adaptive - Need 100M Elements
Initial Mesh Generation
Mesh Distribution
Next step in TSTT/SLAC collaboration - parallel
adaptive loop
16
Parallel Adaptive Solutions

• Need distributed data structures to support
  parallel processing
• Distributed mesh topologically
• PAOMD
• Hierarchical mesh structure used to distribute
  computation and data
• Partition boundary tools
• Update, traversal
• Dynamic load balancing
• Zoltan can meet the need
• Mesh and data migration
• PAOMD supports
• Need to deal with distributed files
• This need work

17
Mesh Mesh Modification

• Refinement
• Can be done on processor with diagonal selection
  made consistent
• Snapping to boundary for curved domains -can
  require swaps, etc.
• Synchronize the partition data at end
• Coarsening and swapping
• Parallization
• On processor - performed
• Off processor - held
• Migration to get operation on processor

18
Initial Test of Component Tools

Initial Mesh - 8,000 tets
Adapted Mesh - 700,000 tets
19
Closing Remarks

• Adaptive loop constructed using available
  components and interoperability operators
• Software for multiple sources combined to create
  a adaptive loop
• Adaptive loop constructed and demonstrated in
  less that 3 months
• By people with little background on interface
  methods or the meshing components
• Included development of error estimation and
  correction indication
• Steps underway
• Installing and testing serial version on SLAC
  systems
• Parallization of the tools
• Putting components on SLAC machine
• Initial version did not resole most effective I/O
  issues
• Will work on the I/O issues
• Parallel adaptive planned to be part of the TSTT
  tools to be put together to support the design
  optimization activity
• Remeshing can adapt for both discretization
  control and to deal will bad elements when total
  shape changes are too large for mesh motion only