Adaptive Mesh Refinement

1
Adaptive Mesh Refinement

• Used in various engineering applications where
  there are regions of greater interest
• e.g. applications are http//www.damtp.cam.ac.uk/u
  ser/sdh20/amr/amr.html
• Global Atmospheric modeling
• Numerical Cosmology
• Hyperbolic partial differential equations (M.J.
  Berger and J. Oliger)
• Problems with uniformly refined meshes for above
  applications
• Grid is too fine grained thus wasting resources
• Grid is too coarse thus the results are not
  accurate

2
AMR Library

• Implements the distributed grid which can be
  dynamically adapted at runtime
• Uses the arbitrary bit indexing of arrays
• Requires synchronization only before refinement
  or coarsening
• Interoperability because of Charm
• Uses the dynamic load balancing capability of the
  chare arrays

3
Indexing of array elements

• Case of 2D mesh

4
Indexing of array elements (contd.)

• Mathematicaly (for 2D)
• if parent is x,y using n bits then,
• child1 2x , 2y using n2 bits
• child2 2x ,2y1 using n2 bits
• child3 2x1, 2y using n2 bits
• child4 2x1,2y1 using n2 bits

5
Alternate representation

0,0,4
6
Communication with Nbors

• In dimension x the two nbors can be obtained by
• - nbor --- x-1 where x is not equal to 0
• nbor --- x1 where x is not equal to 2n
• In dimension y the two nbors can be obtained by
• - nbor --- y-1 where y is not equal to 0
• nbor--- y1 where y is not equal to 2n

7

8
Communication (contd.)

• Assumption The level of refinement of adjacent
  cells differs at maximum by one (requirement of
  the indexing scheme used)
• Indexing scheme is similar for 1D and 3D cases

9
AMR Interface

• Library Tasks
• - Creation of Tree
• - Creation of Data at cells
• - Communication between cells
• - Calling the appropriate funcs for
  computation in each iteration
• - Refining Autorefine Refine on Criteria
  (specified by user)
• User Tasks
• - Coding the main chare to start the
  AmrCoordinator
• - Writing the user data structure to be kept by
  each cell
• - Fragmenting Combining of data for the
  Neighbors
• - Fragmenting of the data of the cell for refine

10
Basic Class Diagram
AmrCoordinator AmrCoordinator(DMsg) AmrCoordinato
r(StartUpMsg) createTree() Synchronise()
Main chare Main(CkArgMsg)
Jacobi Jacobi() voidfragNborData(void,int) voi
d getNborMsgArray(int ) void
store(void,int,int) void combineAndStore(void,
void,int,int ) bool refineCriterion(void) voidf
ragmentForRefine(int) void doComputation(void)
Cell AmrUserData userdata Int iterations Bitvec
Parent children CkChareId
coordHandle Refine() Synchronise() NborComm()
doIteration()
AmrUserData AmrUserData() AmrUserDatacreateData()
AmrUserData createData(void data, int size)
Cell2D Cell2D(_ArrInitMsg) createTree(_ArrInitMsg)
11
Main Chare

• Main(CkArgMsg)
• CProxy_AmrCoordinatorckNew() /1st Method/
• /2nd Method/
• StartUpMsg startMsg new StartUpMsg(dep,synchIn
  t, dim,totIterations)
• CProxy_AmrCoordinatorckNew(startMsg,0)
• /

12
AmrUserData

• Factory Methods that User has to Implement
• AmrUserData createData(void)
• Jacobi instance new Jacobi(size)
• return (AmrUserData) instance
• AmrUserData createData(void data, int
  dataSize)
• Jacobi instance new Jacobi(data, dataSize)
• return (AmrUserData ) instance

13
AmrUserData

• Template Methods
• NeighborMsg fragment(NeighborMsg msg)
• void combineAndStore(NeighborMsg
  msg1,NeighborMsg msg2)
• void store(NeighborMsg msg)
• Virtual Methods
• void fragmentNborData(void data, int
  sizePtr)
• void getNborMsgArray(int sizePtr)
• void store(void data , int dataSize, int
  neighborSide)
• void combineAndStore(void data1, void data2,
  int dataSize,int neighborSide)
• bool refineCriterion(void)
• void fragmentForRefine(int sizePtr)
• void reg_nbor_msg(int neighbor_side, NeighborMsg
  msg)

14
Library--Cell

• Methods
• void init_cell(_ArrInitMsg msg) void
  treeSetup(_ArrInitMsg msg)
• void check_queue(void)
• int sendInDimension(int dim,int side,NeighborMsg
  msg)
• void neighbor_data(NeighborMsg msg)
• void doComputation(void)
• //Refinement and synchronisation
• int sendInDimension(int dim,int side)
• void refine(_RefineMsg msg) void
  change_to_leaf(ChildInitMsg msg)
• void refine_confirmed(_DMsg msg) void
  resume(_DMsg msg)
• void synchronise(_RedMsg msg) void
  refineExec(_DMsg msg)
• void checkRefine(_RefineChkMsg msg) void
  refineReady(bitvec retidx)
• //Virtual Methods
• void create_children(_ArrInitMsg cmsg) void
  doIterations(void)
• void forwardSplitMsg(NeighborMsg msg ,int
  neighbor_side)

15
Pros and Cons of the User Interface

• Pros
• The user has to implement code for one class
  which is very similar to the sequential code
• The control is with the library
• No assumption about the data structure
• All the communication is taken care of for the
  user
• The user does not have to implement a parallel
  tree
• Cons
• The control is with the library
• The user has to do the fragmentation and
  combination of Data This can possibly be a pain
  if the users understanding of the AMR technique
  is not complete

16
AMR Libraries

• PARAMESH Peter MacNeice et al.
• http//sdcd.gsfc.nasa.gov/RIB/repositories/inhouse
  _gsfc/Users_manual/amr.htm
• This library is implemented in Fortran 90
• Does not provide an Object Oriented Interface
• User is required to implement code in a number of
  files
• Cartesian grids is the only data structure that
  is allowed in each cell
• Only supported on CrayT3E and SGIs

17
AMR Libraries (contd.)

• Parallel Algorithms for Adaptive Mesh
  Refinement, Mark T. Jones and Paul E. Plassmann,
  SIAM J. on Scientific Computing, 18,(1997) pp.
  686-708. (Also MSC Preprint p 421-0394. )
• http//www-unix.mcs.anl.gov/sumaa3d/Papers/papers.
  html
• Uses triangles instead of rectangular cells
• Checks for refinement after every timestep
• Two adjacent triangles cannot refine
  simultaneously
• It is required to communicate who the neighbors
  are after every refinement

18
AMR Libraries (contd.)

• DAGH-Dynamic Adaptive Grid Hierarchies
• By Manish Parashar James C. Browne
• In C using MPI
• Implements the distributed grid (SDDG)
• Maps the multidimensional grid via space
  filling curves to a linear structure
• The linear structure is repartitioned at run
  time to dynamically load balance
• Requires implementation of the driver by the
  user for
• - Initialization of MPI environment
• - I/p parameters for the problem
• - Setting up of the grid structure and grid
  functions

19
contd.

• - User interaction or control
• - Update grid functions
• - Communicate result of updating with other
  grids
• - Shut down environment
• Requires synchronization after every fixed
  number of steps
• Only cartesian grids are allowed

20
Future Work

• The developer should have the option of taking
  the control from the library
• Visualization tools
• Determine which strategy works best for the load
  balancing and if needed implement others
• Integration with the FEM framework
• Checkpointing