FACETS A Multiphysics Parallel Component Framework

1
FACETS A Multiphysics Parallel Component
Framework

• Svetlana Shasharina, John Cary, Ammar Hakim,
  Gregory Werner, Scott Kruger, Alexander
  Pletzer
• Tech-X Corporation, Boulder, CO
• University of Colorado, Boulder, CO

Presented by Nanbor Wang
Workshop on Component-Based HPC October 16, 2008
Karlsruhe, Germany
Work partially funded by the US Department of
Energy, Office of Science, Grant
DE-FC02-07ER54907
2
Outline

• FACETS specifics
• FACETS requirements superstructure and
  infrastructure
• External components
• Internal components
• Input language
• Status and future directions

3
ITER is an Experimental Device for Studying
Commercial Use of Fusion
4
FACETS is a tightly coupled and parallel
application

• FACETS is one of 3 fusion integrated modeling
  efforts supported by SciDAC - 10 institutions,
  2M/year
• FACETS (Framework Application of Core-Edge
  Transport Simulations) integrates core, edge and
  wall plasmas -all with disparate time scales much
  shorter than discharge durations
• Hence the need for implicit solvers (meaning
  that during 1 time step components might exchange
  data more than 1 time)
• Hence the need of tight coupling MPI based
  coupling in memory (1 mpi_world)

5
Coupling framework needs infrastructure and
superstructure

• Infrastructure is services to build new
  components
• Data and algorithms organization
• Messaging
• Meshing
• Domain decomposition
• I/O
• Superstructure is for interaction and composition
  of components
• Registries of
• interfaces
• Implementations
• instances
• Composition of simulations
• Separation of super and infra is not clean

6
FACETS uses well-tested elements for
infrastructure and builds own superstructure

• Outside elements are used for infrastructure
• MPI
• TAU (performance measurement and tuning)
• PETSc (solvers)
• VisIt (visualization)
• Optional Babel (for language interoperability)
• Superstructure and components organization - we
  build our own (with reuse from VORPAL plasma
  physics code, CU/Tech-X)
• Why not existing components frameworks
• Need to learn a big and new thing and adopt all
  or nothing
• FACETS components have the same interface so are
  not distinguished by ports (details follow)
• Performance concerns (now we have direct memory
  access to all internal data that needs to talk to
  each other)
• Petascale platforms issues (being resolved,
  though by adding static builds to CCA)

7
FACETS development is driven by

• Ability to incorporate legacy codes
• Ability to build new high-performance and modular
  components
• Multi-stage construction process that multiple
  simulations without recompiling and direct memory
  access for speed
• Separation of data from algorithms for
  flexibility and reuse
• Language for defining a simulation
• Run-time discovery of implementations and
  instances

8
External components are whole legacy codes (not
split into modules)

• Startup interface
• Read input file
• Set up processors to run on
• Choose processors to expose itself to other
  components (N processor to expose Core to Edge
  and M processors to expose Core to Wall, for
  example)
• Allocate memory
• Build internal data from
• Default
• Saved data from another run
• Dump/Restore interface
• Dump all needed for restore data into a file
• Restore data from a file
• Advance interface
• Advance in time
• Set and get scalars by name (scalars are just for
  now as coupling between core and edge is at
  particular radius - 1D)
• We have transport modules (glf23, mmm95), UEDGE
  and WallPsi external components now

9
Internal components common services

• Internal components have services for
• Messaging
• Reading/writing data
• Setting
• Data structures
• Grids
• Domains
• Data interpolation
• Internal components can be nested
• Parents and children
• Communication is mediated by a parent - not
  directly between coupled components

10
Internal components multi-stage construction for
flexibility and efficiency

• Empty constructors
• Method buildData memory allocation and default
  values
• Method buildSolver acquiring handles to all
  needed data across objects
• These methods are called while the input file is
  being read
• Multiple simulations using different input files
  can be created without recompilation
• That is how a new scalable core component was
  created - trying out many algorithms

11
Internal components separation of data and
algorithms

• Advance method is delegated to updaters
• Each component can have many updaters and they
  can be chained
• Updaters acquire access to internal component
  data when buildSolver() method is called
• Updaters can be changed at run time if
  convergence is slow or not reached
• Coupling is performed by a special kind of
  updater and directed by a parent
• Coupling updater lives in a parent component
  containing children being coupled
• Data exchange rate is prescribed by parent (at
  every step by default)
• Coupling updater specifies the exchanged data by
  name and uses get/set methods of components
• Execution of coupled components is concurrent,
  with the static load split imposed by parent

12
Input language describes composition of
components, their updaters and updatable data

• This is a top composite component
• ltComponent facetsgt
• kind updaterComponent
• First child component
• ltComponent coregt
• kind coreComponent
• Data structs of core
• ltDataStruct qOldgt
• kind distArray1D
• onGrid coreGrid
• lt/DataStructgt
• Similar qNew is skipped
• Updater calculation qNew from qOld
• ltUpdater acceptgt
• kind linCombinerUpdater
• in qOld
• out qNew
• lt/Updatergt
• lt/Componentgt

13
Input language describes coupling

• Updater coupling core and edge
• ltUpdater myCoreEdgeUpdatergt
• kind explicitCoreEdgeUpdater
• coreName core
• edgeName edge
• variables to pass from core to edge
• core2EdgeVars "heatFlux"
• variables to pass from edge to core
• edge2CoreVars "temperature"
• lt/Updatergt
• lt/Componentgt

14
Input language drives registration of
implementations and instances

• ltComponent coregt
• kind coreComponent
• //Define grid
• //Define data structures (distributed arrays)
• //Define updaters
• lt/Componentgt
• Actions
• Create an instance of FcCoreComponent named
  core.
• Call buildData and buildSolver using domain, data
  structures and updaters specified further
• The is complete and now registered by name to be
  discovered by name at run time

15
Composition needs programmatic restrictions

• All components communicate using generic
  interfaces (set and get by name)
• Generality all hooked by names
• Danger compose a system that is non-physical
  (edge containing core, or core talking directly
  to the wall)
• Suggested solution
• Concrete classes derive from interfaces
  corresponding the interface describing allowed
  coupling
• These interfaces have additional pure virtual
  methods describing data exchange particular to
  this coupling
• Concrete classes derive from these interfaces,
  define the methods and register them by the name
  of the exchanged parameter to map set/get to a
  particular method
• Base set and get methods are dispatched to
  concrete methods by the variable name

16
Generic set/get methods are dispatched to
functions making sense (constraining coupling)
and derived lower in the hierarchy

• FcCoreEdgeIfc public FcUpdaterComponent
• FcCoreEdgeIfc()FcUpdaterComponent
• this-gtregisterSetMethod("EnergyFlux",
  FcCoreEdgeIfcsetEnergyFlux)
• this-gtregisterGetMethod("EnergyFlux",
  FcCoreEdgeIfcgetEnergyFlux)
• virtual double getEnergyFlux() 0
• virtual setEnergyFlux(const double) 0
• FxEdgeWallIfc public FcUpdaterComponent
• this-gtregisterSetMethod("ParticleFlux",
  FcEdgeWallIfcsetParticleFlux)
• this-gtregisterGetMethod("ParticleFlux",
  FcEdgeWallIfcgetParticleFlux)
• virtual double getParticlesFlux() 0
• virtual setParticlesFlux(const double) 0
• FcUpdaterComponent
• setDouble(const string nm, conts double val)
  
• // uses nm to find the method in the
  registry and call the appropriate

17
Alternative way to impose correct composition is
to use CCA, but the number of ports is larger
than the number of actual interfaces

• CoreComp
• provides_port CoreToEdgeIfc
• uses_port EdgeToCoreIfc
• EdgeComp
• provides_port EdgeToCoreIfc
• uses_port CoreToEdgeIfc
• FacetsComp //Parent
• provides_port CoreToEdgeIfc
• provides_port EdgeToCoreIfc
• uses_port EdgeToCoreIfc
• uses_port CoreToEdgeIfc
• Where the ports have identical methods as the
  data going back and forth is the same (gripes
  from FACETS)

18
FACETS has core and edge coupled and has more to
come

• Internal Core component is developed
• Wrapped UEDGE and WallPsi as external components
• Coupled core and edge
• Uses Babel to bring in F90 transport modules and
  F90/Python UEDGE
• Next
• Robust core-edge coupling
• Dynamic load balancing
• Imposing interfaces decided upon
• Bringing in the wall
• Implementing correct composition mechanisms
• Investigate MCMD ideas
• More information can be found at
  https//www.facetsproject.org/facets