HandsOn Flash

1
Hands-On Flash

• Setting up Your Own Simulations

2
Outline

• Setting up an out of the box simulation
• Customizing Flash for a new simulation (i.e., how
  to set up your problem)
• Simple visualization and analysis with idl using
  fidlr3.0

3

• Setting up an out of the box problem

4
Flash Source to Executable

• Flash comes as source code consisting of source
  code components known as units
• Going from Flash source code to executable
  consists of
• The setup step specifying which source units to
  include in the application and linking together
  the source files
• The compilation step running make to compile the
  flash executable

5
The Setup Script

• The primary job of the setup script is to
• Traverse the FLASH source tree and link necessary
  files for a given application to the object
  directory
• Find the target Makefile.h for a given machine.
• Generate the Makefile to make the FLASH
  executable.
• Generate files needed to add runtime parameters
  to a given simulation.
• Generate files needed to parse the runtime
  parameter file.

6
Example Makefile.h

• Located in FLASH3/sites/ltmachine namegt

MPI_PATH /usr/local/mpich-intelHDF5_PATH
/usr/local/hdf5-iccNCMPI_PATH
/usr/local/pnetcdf-icc FCOMP
MPI_PATH/bin/mpif90CCOMP
MPI_PATH/bin/mpiccCPPCOMP
MPI_PATH/bin/mpiCCLINK
MPI_PATH/bin/mpif90 FFLAGS_OPT -c -r8 -i4
-O3 -real_size 64 -unroll -align -prefetch -pad
-ip FFLAGS_DEBUG -c -g -r8 -i4 -check bounds
-check format -check output_conversion -warn all
-real_size 64FFLAGS_TEST -c -r8 -i4 -O2
-real_size 64CFLAGS_OPT -c -O3
-D_LARGEFILE64_SOURCECFLAGS_DEBUG -c -g -debug
extended -D_LARGEFILE64_SOURCECFLAGS_TEST -c
-O2 -D_LARGEFILE64_SOURCECFLAGS_HDF5 -I
(HDF5_PATH)/includeCFLAGS_NCMPI -I
(NCMPI_PATH)/includeCFLAGS_MPI
-I(MPI_PATH)/includeLFLAGS_OPT -r8 -i4
-Vaxlib -lsvml -Ur -oLFLAGS_DEBUG -r8 -i4
-Vaxlib -g -oLFLAGS_TEST -r8 -i4 -Vaxlib
-oLIB_HDF5 -L (HDF5_PATH)/lib -lhdf5
-lzLIB_MPI -L(MPI_PATH)/lib -lfmpich
-lmpichLIB_NCMPI -L(NCMPI_PATH)/lib -lpnetcdf
7
First Setup Example

• From the main flash directory, FLASH3/, type

./setup Sedov -2d -auto
command
problem name
dimension
Setup script determines which units to include
based on the problem name (rather than manually)
To run setup you shouldnt need to modify the
setup script (FLASH3/bin/setup.py)
8
Setup Options
-auto setup script will automatically generate
the Units file based on the problem you specify
Sample Units File
INCLUDE Driver/DriverMain/TimeDep INCLUDE
Grid/GridMain/paramesh/Paramesh3/PM3_package/heade
rs INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_pa
ckage/mpi_source INCLUDE Grid/GridMain/paramesh/Pa
ramesh3/PM3_package/source INCLUDE
Grid/localAPI INCLUDE IO/IOMain/hdf5/serial/PM INC
LUDE PhysicalConstants/PhysicalConstantsMain INCLU
DE RuntimeParameters/RuntimeParametersMain INCLUDE
Simulation/SimulationMain/Sedov INCLUDE
flashUtilities/general INCLUDE physics/Eos/EosMain
/Gamma INCLUDE physics/Hydro/HydroMain/split/PPM/P
PMKernel INCLUDE physics/Hydro/HydroMain/utilities

Try manually changing IO/IOMain/hdf5/serial/PM to
IO/IOMain/hdf5/parallel/PM - Then run setup
without the -auto flag
If you dont use the -auto flag, you must have a
valid Units file in the top FLASH directory
(FLASH3/Units)
9
Setup Options

• Another way to use a specific unit in setup is
  with the -unit option

-unitltunitgt force a specific unit to be used
in setup
./setup Sedov -2d -auto -unitIO/IOMain/hdf5/par
allel/PM
10
More Setup Options
11
Even More Setup Options
12
Setup Shortcuts

• The Flash3 setup script includes a shortcuts
  feature
• Allows many setup options to be included with one
  keyword
• Defined in Flash3/bin/setup_shortcuts.txt
• To use a shortcut, add shortcut to your setup
  line
• Example
• The shortcut ug is defined as
• ug--with-unitGrid/GridMain/GridUG
• ./setup Sod -auto ug
• this is equivalent to typing in unit options with
  
• -unitGrid/GridMain/UG
• -unitIO/IOMain/hdf5/serial/UG (because the
  appropriate IO is included by default)
• Look in Flash3/bin/setup_shortcuts.txt for more
  examples and to define your own

13
setup Generated Files
contains the options with which setup was called
and the command line resulting after shortcut
expansion
setup_call
contains the list of libraries and their
arguments (if any) which was linked in to
generate the executable
setup_libraries
contains the list of all units which were
included in the current setup
setup_units
contains a list of all pre-process symbols passed
to the compiler invocation directly
setup_defines
contains the exact compiler and linker flags
setup_flags
contains the list of runtime parameters defined
in the Config files processed by setup
setup_params
setup_vars
contains the list of variables, fluxes, species,
particle properties, and mass scalars used in the
current setup, together with their descriptions
14

• Customizing Flash for a new simulation (i.e.,
  how to set up your problem)

15
Adding a New Simulation

• New Simulations are added by creating a directory
  in FLASH3/source/Simulation/SimulationMain
• The Simulation directory gives people working on
  a particular problem a place to put problem
  specific code that replaces the default
  functionality in the main body of the code
• Its also a place to tell the setup script which
  units this problem will need from the rest of the
  code
• In the Simulation directory, one needs to have
• A Config file to at least describe the units and
  runtime parameters needed by this problem
• One or more flash.par files which each give a set
  of runtime parameter values for running the
  problem in different ways
• Code specifying initial conditions and code for
  the simulation as a unit (e.g., initialization of
  the unit)

16
Config File Syntax
17
More Config File Syntax
18
Even More Config File Syntax
19
Required Code for a New Simulation

• There are certain pieces of code that all
  simulations must implement
• Simulation_initBlock.F90 Sets initial
  conditions in a single block.
• Simulation_init.F90 Reads the runtime
  parameters, and performs other necessary unit
  initializations.
• Simulation_data.F90 A Fortran module which
  stores data and parameters specific to the
  Simulation.
• Optionally, a sumulation could implement
• Simulation_initSpecies.F90 To give the
  properties of the species involved in a
  multispecies simulation

20
Customized Code for a new Simulation

• In a FLASH simulation directory, you can place
  code that overrides the functionality you would
  pick up from other code units
• In the custom code you can modify
• Boundary conditions (Grid_applyBCEdge.F90)
• Refinement criterion (Grid_markRefineDerefine.F90)
  
• Diagnostic integrated quanties for output (in the
  flash.dat file), e.g., total mass (a default) or
  vorticity (IO_writeIntegralQuantities.F90)
• Diagnostics to compute new grid scope variables
  (Grid_computeUserVars.F90)
• In general, this is a place to hack the code in
  ways specific to your problem, and you can hack
  basically anything

21
Example Non-trivial Setup TwoGamma

• TwoGamma is for a simple test of advecting two
  fluids having different gammas to investigate
  whether an instability develops at the interface
  between the two fluids.
• This Simulation implements
• Simulation_initSpecies.F90, because it has
  multiple fluids
• and Grid_applyBCEdge.F90, because it needs custom
  boundary conditions on the lower x edge of the
  domain

22
TwoGamma Config File
configuration file for the TwoGamma target
problem REQUIRES Driver REQUIRES
physics/Hydro REQUIRES physics/Eos/EosMain/Multiga
mma REQUIRES Multispecies REQUESTS IO
Parameters D sim_p0 constant
pressure PARAMETER sim_p0 REAL
2.5e-0 D sim_rho1 density of the first
fluid PARAMETER sim_rho1 REAL
1.0e-0 D sim_rho2 density of the second
fluid PARAMETER sim_rho2 REAL
1.0e-0 D sim_cvelx initial velocity PARAMETER
sim_cvelx REAL 0.1e-0 SPECIES
FLD1 SPECIES FLD2
23
TwoGamma flash.par
AMR parameters lrefine_max 4 lrefine_min
4 simulation parameters basenm
"twogamma_" restart .false. plotFileIntervalTim
e 0.1 checkpointFileIntervalTime
0.5 nend 15000 tmax
10.0 checkpointFileNumber 0 plotFileNumber
0 dtini 1.e-10 dtmin 1.e-10 cfl .5 cvisc
.1
24
TwoGamma flash.par (Cont.)
smlrho 1.e-10 smallt 1.e-10 xmin
0.0e0 xmax 1.0 ymin 0.0e0 ymax
1.0e0 geometry "cartesian" variables for
plotting plot_var_1 "dens" plot_var_2
"temp" plot_var_3 "ener" plot_var_4
"pres" plot_var_5 "velx" plot_var_6
"fld1" plot_var_7 "fld2" xl_boundary_type
"user" xr_boundary_type "outflow" yl_boundary_ty
pe "periodic" yr_boundary_type "periodic"
25
TwoGamma Simulation_initSpecies.F90

• subroutine Simulation_initSpecies()
• implicit none
• include "Multispecies.h"
• include "Flash.h"
• include "Multispecies.h"
• include "Multispecies_interface.h"
• call Multispecies_setProperty(FLD1_SPEC, A, 1.)
• call Multispecies_setProperty(FLD1_SPEC, Z, 1.)
• call Multispecies_setProperty(FLD1_SPEC, GAMMA,
  1.66666666667e0)
• call Multispecies_setProperty(FLD2_SPEC, A,
  4.0)
• call Multispecies_setProperty(FLD2_SPEC, Z,
  2.0)
• call Multispecies_setProperty(FLD2_SPEC, GAMMA,
  2.0)
• end subroutine Simulation_initSpecies

26
TwoGamma Grid_applyBCEdge.F90

• if(faceLOW) then
• select case (bcType)
• case(OUTFLOW)
• do i 1,guard
• dataRow(i) dataRow(guard1)
• end do
• case(USER_DEFINED)
• select case(var)
• case(GAMC_VAR)
• dataRow(1guard)sim_gammac1
• case(DENS_VAR)
• dataRow(1guard)sim_rho1
• case(PRES_VAR)
• dataRow(1guard)sim_p0
• case(VELX_VAR)
• dataRow(1guard)sim_cvelx
• case(VELY_VAR)
• dataRow(1guard)0.0
• case(VELZ_VAR)

27

• Simple visualization and analysis with idl using
  fidlr3.0

28
Fidlr (Flash idl Routines) 3.0

• 1 and 2d visualization
• 2d slices of 3d data
• Has idl routines to read the HDF5 and NetCDF
  Flash checkpoint and plotfiles.
• xflash3 Gui frontend
• Can call load routines from idl command line and
  manipulate checkpoint data interactively
• Lives in tools/fidlr3.0 in your source

29
Setting up and Running idl for fidlr

• Setting up environment correctly very important
• To get color, run idl with start_linux
• idl start_linux
• IDLgt xflash3

30
xflash3 the fidlr GUI
31
fidlr Interactive Examples

• Read in density from file, put it on a uniform
  grid, and make a contour plot
• IDLgt dens loaddata(sedov_2d_6lev_chk_0001,
  dens, XCOORDS x, YCOORDSy)
• IDLgt contour, dens, x, y
• Read in a dataset, with full AMR details
• IDLgt read_amr, sedov_2d_6lev_hdf5_chk_0001,
• TREEtree, PARAMETERSparams, DATAdata,
  STORED_VARSvars