Design Requirements and Realization

1
Design Requirements and Realization

• The Cactus Team
• Albert Einstein Institute
• cactus_at_cactuscode.org

2
Motivation Users

• Cactus is primarily a code for users and
  application developers so it must
• be easy and flexible to use
• be portable, and work in the same way on all
  architectures
• not involve learning new coding paradigms
• allow coding in Fortran!
• support collaborative development and use
• have confidence tests
• hide all the clever computer science stuff
  parallelization, IO, overloading, configuring,
• have documentation and examples
• be stable ...
• and yet must be able to do whatever the users
  need to do

3
Motivation Developers

• And, since computer scientists supply the
  infrastructure, it must
• support and encourage development and
  incorporation of new technologies
• allow different programming languages and
  paradigms
• provide rich low level APIs and services
• have fixed and extensible interfaces
• allow thorns to customise many different areas of
  the flesh (main routine, scheduling tree, )
• make new infrastructure thorns immediately
  available to application thorns for testing and
  using

4
Motivation Cactus Team

• And since we have to support it, it must
• make use of and exploit existing technologies
  where possible
• not use non-portable languages or features, for
  ease of porting and maintaining
• be as modular as possible
• be well documented
• adhere to coding guidelines and practises
• provide us with a good excuse to play with the
  very latest hardware and software ...

5
Design Issues For Users
Easy access to other packages
Modular
Portable develop on laptops run on
supercomputers
Easy (hidden) Parallelism
Help from mailing lists and support desk
Encapsulation to protect from other thorns
Easy and flexible IO
Reliable checkpointing
Self configuring on all machines
Easy extension to AMR
Program in Fortran, C, C with same API
Facilitate collaborative work
Testsuites
Hidden optimisation
GUIs for running, compiling, debugging
Documentation, tutorials, examples
6
Abstract Design Goals

• Generalization
• Meta-code that can be applied to any numerical
  system
• Abstraction
• Identify key concepts that can be abstracted
• Encapsulation
• Modularize, protect thorn developers from each
  other
• Extension
• Prepare for future concepts and technologies

7
Generalization
Ray tracing
Regular/irregular grids
Finite differences
Finite volumes
Unstructured grids
Meta-code for any numerical system
Finite elements
Multi-patch
N-body
Cartesian
Monte-Carlo
Polar
Hyperbolic
Adaptive mesh refinement
Elliptic
Parabolic
8
Abstraction
Convergence
Evolution skeleton
Input/output
Checkpointing
Grid variables
Abstract Key Concepts
Reduction operators
Coordinates
Interpolation operators
Parallel operators
Elliptic solvers
Data types
9
Encapsulation
Thorns and Thorn Arrangements
ActiveThorns
CVS Versioning
Private parameters
Protection for thorn developers
Parameter ranges
Private grid variables
Parameter checking
Implementations
Inheritance
10
Extension
Capability browsing
Communication layer
Java/Perl thorns
GUIs and Portals
Scripting language
Be ready for new concepts
Adaptive meshes
Unstructured grids
Simulation steering
New hardware architectures
New things with havent been named yet
Interactive scheduling
11
Cactus Architecture
Cactus
Thorns
Computational Toolkit
Toolkit
Toolkit
Flesh
Configure
CST
12
Application View
13
Thorn Architecture
Thorn
Configuration Files
Parameter Files and Testsuites
Source Code
????
????
Fortran Routines
C Routines
C Routines
Make Information
Documentation!
14
Thorn Collections

• For organizational convenience, thorns are
  grouped into arrangements
• may have related functionality (e.g. IO or
  Maxwell solvers)
• may have the same author
• may contain everything needed for one problem
• e.g. CactusBase, CactusPUGH
• We call a collection of arrangements, a toolkit
  e.g.
• Cactus Computational Toolkit
• Cactus Relativity Toolkit

15
Realization - Flesh API

• Abstract Flesh API for
• Driver functions (storage, communication)
• Interpolation
• Reduction
• IO, checkpointing
• Coordinates
• etc, etc
• In general, thorns overload or register their
  capabilities with the Flesh, agreeing to provide
  a function with the correct interface
• e.g. CCTK_SyncGroup (overloaded)
• e.g. CCTK_OutputVar(variable,IOASCII) (registe
  red)

16
Realization - Parallelization

• Want to be able to make use of different
  communication protocols
• MPI, PVM, SHMEM, pthreads, OpenMP, Corba, RDMA,
• portability across platforms
• choose best option for a given architecture
• compare and different protocols
• Thorns dont call communication functions
  directly, but through the Cactus API
• CCTK_SyncGroup, CCTK_Barrier, CCTK_Reduce,
• Thorns overload or register functions to provide
  these capabilities
• e.g. thorn CactusPUGH/PUGH (parallel MPI driver)
• CCTK_Barrier gt MPI_Barrier

17
Realization - Configuration files

• Each thorn provides 3 configuration files,
  detailing its interface with the Flesh and with
  other thorns
• CCL Cactus Configuration Language
• interface.ccl
• implementation, this thorns variables and
  variables used from other thorns (coming soon
  functions provided or used from other thorns)
• param.ccl
• this thorns parameters, parameters used and
  extended from other thorns
• schedule.ccl
• when and how this thorns routines should be
  executed, optionally with respect to routines
  from other thorns

18
Realization - Portability

• Cactus data types for portability
• CCTK_REAL, CCTK_INT, CCTK_COMPLEX,
• Used for Grid Variables and Parameters
• GNU Autoconf
• Known Architectures files
• Thorns call Cactus API rather than
  system/software dependent packages
• CCTK_Abort gt MPI_Abort (or something else)

19
Realization - Encapsulation

• Thorns
• A thorn usually implements a single specific task
• Object Orientated Features
• Grid Variables, Functions
• private, protected, public
• Parameters
• private, restricted, global
• Implements
• what does this thorn do? Provides relationships
  between thorns.
• Inherits
• get all public variables, functions from an
  implementation and its ancestors

20
Realization - Scheduling

• The Cactus Flesh contains a flexible rule based
  scheduler which controls the program flow.
• The scheduler calls routines from thorns, the
  order in which the routines are called is
  prescribed in a thorn configuration file.
• The main calling routines in the Flesh are also
  overloadable, providing a flexible and
  configurable mechanism for customising program
  flow.

21
Realization - Implements

• Each thorn declares what it does or what it
  implements
• interpolate
• maxwell_initial_data
• extract_multipoles
• Defines relationships between thorns
• Thorns providing same implementation must have
  common public and protected variables
• Can compile many different thorns all providing
  same implementation
• pick which one to use in the parameter file at
  run time!

22
Realization - Parameters

• Range checking and validation
• combats old problem of setting parameters to
  values which didnt exist
• evolution_method superstable fast 10th order
  shock capturing
• thorn writers must now specify ranges and
  descriptions for all parameters
• checked at run time
• Steerable
• new implementation of steerable/changeable
  parameters for remote steering
• must define steerable only if it makes sense to
  do so

23
Realization - Compilation

• Autoconf based configuration process for
  different architectures, compilers and libraries
• CST (Cactus Specification Tool) Perl based
  parsing of files written in Cactus Configuration
  Language (CCL) generates thorn wrappers, argument
  lists (Fortran!), parameters, variables,
  scheduling, etc.
• Cactus creates different configurations from the
  same source code
• different architectures on common file system
• use different compiler options, debug-modes,
  communication protocols
• different thorn lists

24
Realization - Testing

• Test Suite Checking Technology
• Thorns can contain parameter files and their
  output
• Running Cactus test suite program re-executes
  these parameter files checking against the
  original output.
• RemoteTest
• Advantages
• Checks development work on your own thorns
• Protects against changes in other thorns
• Compares e.g multiprocesser runs against single
  processor runs, or the use of new thorns
• Compares different architectures
• Warning Levels
• CCTK_Warn
• Warning level can be chosen at run time