PARAMESH: A PARALLEL, ADAPTIVE GRID TOOL FOR THE SPACE SCIENCES Kevin Olson Drexel University Philad

1
PARAMESH A PARALLEL, ADAPTIVE GRID TOOL FOR THE
SPACE SCIENCESKevin OlsonDrexel University
Philadelphia, PAPresented, AISR MeetingMay
2008University of Maryland
2
COLLABORATORS

• Peter MacNeice (NASA/GSFC)
• Joan Centrella (NASA/GSFC)
• Don Lamb (U. of Chicago)
• other collaborators include
• C. Mobarry, R. DeFainchtein, M. Gehmeyer, M.
  Bhat,
• C. Packer, M. Rilee, J. VanMetre, D. Choi
  (NASA/GSFC)
• R. Devore (NRL), D. Swesty, A. Calder, M. Zingale
  (SUNY/SB), J. Dursi, K. Riley, A. Siegel, T.
  Linde, D. Sheeler, A. Dubey, K. Weide (U.
  Chicago)
• Initial funding provided by NASA/ESTO-CT

3
TALK OUTLINE

• AN OVERVIEW OF PARAMESH
• AISRP GOALS and ACCOMPLISHMENTS
• SOME APPLICATIONS and SCIENCE RESULTS USING
  PARAMESH

4
An Quick Overview of PARAMESH
5
PARAMESH what is it ?

• A package designed to ease the task of adding
  parallelization and dynamic, adaptive mesh
  refinement (AMR) to an already existing uniform
  mesh, serial code
• A library of subroutines and accessible data
  structures
• Written in Fortran90 and C (NAG, Lahey, Intel,
  Portland Group, HP-Compaq, IBM, SGI, g95)
• Interprocessor communication using MPI
• Version 4.1 released March 2008.
• PARAMESH WEB site
• http//www.physics.drexel.edu/olson/paramesh-doc
  /Users_manual/amr.html

6

• A subset of Berger-Oliger, block-adaptive scheme
• Computational Volume is recursively bisected into
  blocks, forming a tree data structure.

7

• Blocks are ordered and distributed to processors
  using a space filling curve.

8

• Each Block is a logically cartesian, uniform mesh
  of cells.
• Each cell in a block can store user specified
  data at cell centers, corners, edges or faces

A 2-D Block of Cells
A Single Mesh Cell in 3-D
9

• Support for consistent fluxes and ensuring
  conservation for finite volume schemes
• Support for averaging data at cell edges to
  ensure consistent circulation integrals around
  cell faces

10
ACCOMPLISHMENTS UNDER AISR
11
AISR ACCOMPLISHMENTS

• Extend and improve PARAMESH
• Parallel I/O (HDF5, MPIIO), I/O formats for the
  graphics packages ChomboVis and Visit.
• C Interface
• Improved support for multigrid solvers
• Improve divergence of B control (Balsara
  algorithm supported, used in FLASH 3.0)
• Improved support for non-cartesian coordinate
  systems (cylindrical and spherical coordinates
  supported)
• Many performance enhancements.

12
AISR ACCOMPLISHMENTS

• Effective Open Source Development
• Code managed using Sourceforge
• Developed coding standards and a developers
  guide was posted to the WEB site.
• Improved automatic testing procedure.
• Self-documenting comments using Robodoc.
• Hard to find other developers. Users make
  feature requests, but are not interested in
  developing code (exception was FLASH code team).
• Integrated new versions of PARAMESH into actual,
  working, space science applications (ARMS,
  HAHNDOL, FLASH)

13
SOME SPACE SCIENCE APPLICATIONS and RESULTS
USING PARAMESH(ARMS, HAHNDOL, AND FLASH)
14
Other Space Science Applications

• CASIM (M. Benna and P. Mahaffy at GSFC)
• MHD application for modeling comet-solar wind
  interaction
• YDFCT (D. Odstrcil at NOAA)
• MHD application for modeling multiple interacting
  CMEs, integrated into CCMC
• ZeusAMR (W. Abbett et al. at U.C. Berkeley)
• Combination of Zeus MHD code and PARAMESH for
  modeling magnetic flux emergence from the sun
• IBEAM (D. Swesty et al. SUNY-SB)
• Modern Astrophysics framework, radiation
  hydrodynamics for modeling gamma ray burst
  fireballs
• Plus others, the list continues to grow, hundreds
  of users worldwide.

15
ARMSR. DeVore (NRL)P. MacNeice, K. Olson
(NASA/GSFC)

• Solves the equations of MHD (DeVore, 1991)
• Code for which PARAMESH was originally developed
• Used for solar physics applications
• Numerical schemes FCT with constrained transport
  for MHD and multigrid for implicit formulation of
  non-linear thermal conductivity

16
Coronal Mass Ejection S. Antiochos (NASA/GSFC)
et al.
17
Solar Polar Jet FormationS. Antiochos
(NASA/GSFC) et al.
18
HAHNDOLJ. Centrella, D. Choi, B. Imbiriba, J.
Baker, D. Fiske, J. Van Meter (NASA/GSFC), D.
Brown, L. Lowe (N.C. State)

• General Relativity code, Solves Einstein
  Equations
• Goal To simulate gravitational waves resulting
  from the collision of super-massive black holes
  in order to help interpret data from LISA mission
  (to be launched someday).
• Numerical Schemes Multigrid, Finite Difference
• Incorporates latest version of PARAMESH.
• Used to perform one of the largest simulations on
  Columbia System at NASA/AMES

19
Gravitational Wave Propagation(J. Centrella et
al., NASA/GSFC)
20
FLASH ASTROPHYSICS CODEFLASH code team,
University of Chicago

• Fryxell et al., 2000, ApJS, 131, 273.
• WEB site www.flash.uchicago.edu
• Implements various CFD Schemes, MHD, Nuclear
  Reactions, Stellar Equations of State, and
  self-gravity using multigrid.
• Designed to model Astrophysical thermonuclear
  flashes (X-ray bursts, Novae, and Type 1a
  Supernovae).
• FLASH 3.0 recently released. Incorporates
  version 4.0 of PARAMESH.

21
X-ray Burst (M. Zingale, SUNY/SB)
22
FLASH Validation Experiments
23
FLASH Validation Experiments
24
Type Ia SupernovaD. Lamb, et al., U. of Chicago
25
CONCLUSIONS

• Parallel, Adaptive Mesh Refinement has wide
  applicability in the space sciences
• PARAMESH provides a useful and flexible tool for
  adding parallel AMR to a wide variety of
  applications, allowing the efficient solution of
  problems in the Space Sciences.
• The major goals that were originally proposed to
  AISR were accomplished.
• Thank you AISR !