Overview of the ACTS Toolkit For NERSC Users

1
Overview of the ACTS ToolkitFor NERSC Users

• Brent Milne
• John Wu
• acts-support_at_nersc.gov

2
What is the ACTS Toolkit?

• Advanced Computational Testing and Simulation
• Part of the DOE 2000 Project
• Will likely shift to base DOE funding
• Tools (roughly 20) for developing parallel
  applications
• Developed (primarily) at DOE Labs
• Separate projects originally
• ACTS is an umbrella project
• Collecting tools
• Leveraging numerous independently funded projects

3
ACTS Project Goals

• Bringing tools together into a Toolkit
• Making the tools interoperable
• Providing consistent application interfaces
• Promoting general (not application specific)
  solutions to parallel programming needs
• Encouraging code re-use
• Impacting DOE science
• ASCI, IT2(SSP)
• Enabling large scale applications
• example follows Oil Reservoir Simulation

4
Prometheus

• Multigrid solver
• Unstructured meshes in solid mechanics
• Modified matrix graphs
• 26 million DOF, 640 PEs
• Use PETSc

http//www.cs.berkeley.edu/madams/prometheus
5
NERSC Activities

• Make ACTS tools available on NERSC platforms
• Provide technical support
• Find new ER users who can benefit from the tools
• Work with users to integrate tools into
  applications
• Evaluate the tools
• Create ACTS information center (online)

6
ACTS Information Online
http//acts.nersc.gov/

• One-stop Shopping
• But coordinated and integrated with developer
  resources.
• User-oriented Information
• Which tool should I use to solve my problem?
• Which tools are ready for prime time?
• Reviews strengths, weaknesses, caveats, etc.

7
ACTS Support at NERSC
acts-support_at_nersc.gov

• Support for application development
• Technical support
• Tool installation (by request)
• Leverage with developers
• Support is also available through traditional
  NERSC channels (e.g. consult_at_nersc.gov).

8
Risk

• ACTS Tools are still research projects
• Cant guarantee indefinite support
• Cant guarantee fixed interface
• But commercial software doesnt either
• A NERSC goal is to minimize risk to users
• Provide unbiased advice
• Promote tools that work
• Support tools with the best chance of surviving
• Be an advocate for users

9
Tool Catagorization

• Numerical
• Libraries/data structures that implement
  numerical algorithms
• I.e. your classic numerical packages
• PETSc, Aztec, Hypre, ScaLAPACK, SuperLU, PVODE,
  Opt
• Structural (Frameworks)
• Libraries/data structures that manage data,
  communicate
• Global Arrays, Overture, POOMA , Cumulvs, PAWS,
  InDEPS
• Infrastructural
• TAU (performance analysis)
• Globus (meta-computing)
• PADRE, Nexus, Tulip (middleware -- mostly used by
  other tools)

10
Tool Status at NERSC

• Installed on the T3E and fully supported by
  NERSC
• Aztec
• PETSc
• ScaLAPACK
• TAU
• Known to have been successfully used on NERSCs
  T3E
• CUMULVS
• Global Arrays
• Globus
• POOMA
• PVODE

11
Tools available on T3E
12
ScaLAPACK

• Installed on the T3E
• Parallel version of LAPACK
• Well known, trusted
• Basic dense linear algebra (BLAS)
• Direct solution of linear systems / Factorization
• General, tri-diagonal, banded
• Dense matrix eigensolvers
• Sustained 605 GFLOPS on ASCI Red in materials
  simulation
• Get started man scalapack or module help
  scalapack

13
PETSc

• Installed on the T3E
• Functionality for solving PDEs in parallel
• The most widely used and well supported member of
  the ACTS toolkit
• Runs on serial and parallel machines (even NT)
• C and Fortran Programming Interfaces
• Modular, objected design methodology
• Methodology extensible to User code
• Get started module help petsc or man petsc

14
Aztec

• Installed on T3E
• Solving sparse linear systems on distributed
  machines
• Highly efficient and scalable
• Some Apps have successfully scaled to 1000s of
  processors
• Krylov iterative methods
• CG, CGS, Bi-CG-Stab, GMRES, TFQMR
• Large preconditioning suite including
• Jacobi, Gauss-Seidel, overlapping domain decomps
  (ILU et al)
• Get started module help aztec or man aztec

15
Aztec

• User input matrix in DMSR or DVBR format
• Aztec sets up its own data structure for solving
  linear systems
• Used by groups of dedicated users
• Salinas
• MPSalsa
• COYOTE II
• TOUGH
• ...

http//endo.sandia.gov/BB/comp_str_dyn/mp-fema.htm
l
16
Using Aztec basic steps

• Prepare your linear system
• distribute the matrix
• call AZ_transform
• set up right-hand side and initial guess
• call AZ_reorder_vec on initial guess and
  right-hand side
• selective an iterative solver and a
  preconditioner
• call AZ_solve
• call AZ_invorder_vec on solution

17
Using Aztec DMSR

• local index to global index map
• ja
• ja0nloc - starting positions
• janloc1 - global column indices
• val
• val0nloc-1 diagonal elements
• valnloc not used
• valnloc1 off-diagonal nonzeros

18
Using Aztec

• Two re-ordering
• global re-ordering to distribute the matrix (done
  by user)
• local re-ordering to allow maximum overlapping of
  communication and computation (done by
  AZ_transform)
• Part of the right-hand side (elements beyond
  nloc) is used as scratch space internally
• more information at http//acts.nersc.gov/aztec/ev
  aluation.html
• examples to look at
• /usr/local/pkg/Aztec/Aztec-2.0/app
• /u1/kewu/azmm.tar

19
TAU

• Currently installed on the T3E
• Profiling of C/C and Fortran programs
• Detailed information - much more than prof/gprof
• For C per-class and per-instance profiling
• Graphical display of profiling results
• Built-in viewers, interface to VAMPIR
• Automatic instrumentation in the future
• Get started module help tau or man tau

20
TAU Screen Shot
21
Using TAU instrumentation

• Template ltclass Tgt
• class Templated
• private
• T Data
• public
• T SetData (T d)
• Data d
• return Data

22
Using TAU compilation

• Load module tau to define TAUROOTDIR
• include one of the Makefiles from
  TAUROOTDIR/t3e/lib to define TAU_DEFS and
  TAULIBS
• Add TAULIBS to link command
• Examples at TAUROOTDIR/examples
• More help at http//acts.nersc.gov/tau/at-nersc.ht
  ml

23
Using TAU view traces

• Must use VAMPIR on T3E
• To turn TAU trace files to VAMPIR trace file
• merge all TAU trace files into one (tau_merge)
• convert merged file with tau_convert -vampir
• see Makefile of examples in TAUROOTDIR/examples
  
• To use VAMPIR
• module load vampir
• vampir tracefile.pv

24
Tools not on T3E or under development
25
SuperLU

• Direct solution of large sparse linear systems
• Single processor and multithread versions are
  available http//www.nersc.gov/xiaoye
• Distributed memory (MPI) version under
  development
• Attained 10 GFLOPS (speedup 100) on 512 node T3E
• 2-D matrix distribution
• static pivoting (replace small pivot with larger
  value)
• iterative refinement to ensure accuracy

26
SuperLU
http//www.nersc.gov/xiaoye/SuperLU
27
Hypre

• Family of problem specific preconditioners
• Physics and grid based
• Algebraic and structured multigrids
• Sparse approximate inverse
• Will be becoming available for use soon
• Should be useful stand-alone
• and in the context of other ACTS tools
• http//www.llnl.gov/CASC/linear_solvers/

28
PVODE

• Solves ODE DAE
• Adams multi-step for non-stiff problems
• BDF multi-step for stiff problems
• Can use linear solvers from ACTS
• Integration with PETSc planned
• Now includes non-linear solvers (KINSOL)
• Newton based solvers
• Globalization strategies (line search, trust
  region)

Http//www.llnl.gov/CASC/PVODE
29
ATLAS and PHiPAC

• Automatic generation of machine optimized BLAS
• Can beat hand-tuned performance

30
Global Arrays

• Shared memory programming interface on
  distributed memory computers
• But doesnt hide remote access characteristics
• Global array object abstraction
• One-sided communication
• Used as infrastructure for several computational
  chemistry packages (e.g. NWChem on the NERSC T3E)

31
Global Arrays
http//www.emsl.pnl.gov/pub/proj/tms/hpcc_actinide
s
32
Overture

• C Class Library
• Overlapping composite grids

• Complex geometry

33
POOMA

• C framework (class library) for high
  performance parallel computing
• Abstractions for arrays, grids, particles
• Allows application to be written with high level
  data-parallel style
• T3E specific difficulties have apparently been
  overcome
• Extensive expression templates overwhelmed
  compilers

34
CUMULVS

• Simple mechanisms for
• Computational steering
• Interactive visualization
• Fault tolerance
• Handles, for multiple attached viewers,
• Data collection, distribution
• Computational parameter control (steering)
• Can be used in heterogeneous environment (layered
  on PVM)

35
Globus

• Provides many services to enable development of
  meta-computing applications
• Resource discovery, monitoring, allocation
• Process (and IO) staging, management
• Communication - varied protocols - Nexus
• User and resource authentication, security (RSA)
• Hides implementation details from application
• Provides common middleware layer of interfaces
• Transparent remote access to resources
• User authentication, File staging, Batch queuing,
  etc., through common interfaces
• Perhaps of greatest immediate interest to
  application programmer

36
PAWS

• Distributing data between separate parallel
  applications
• Coupling parallel simulation to parallel
  visualization tools that
• require different parallel data distributions
• run on same machine
• ...or run across heterogeneous machines
• Minimal application coding requirements
• PAWS Controller handles interactions, data
  mapping

37
PAWS Screen Shot
Controller
Sim. Code
Viz. Code
38
Future Directions

• Comparisons of parallel linear system solvers
• AZTEC
• BLOCKSOLVE
• ISIS
• PETSc
• PSPARSLIB
• Test problems, anyone ?

39
Future Directions

• CCA (Common Component Architecture)
• Developing standardized ways of managing
  numerical components to allow
  mixing-and-matching
• Frameworks for gluing components together
• Similarities to CORBA, DCOM, Java Beans
• But
• Scientific Interface Description Language
  (allowing Fortran)
• Designed for zero-copy interconnections
• ESI (Equation Solver Interface)
• Developing standardized interfaces for scalable
  linear algebra
• Specific test case for CCA component design

40
Conclusion

• DOE has funded a nice set of tools
• We are here to maintain them for you
• Any questions?

http//acts.nersc.gov/
acts-support_at_nersc.gov