Allen D. Malony, Sameer S. Shende, Robert Bell Kai Li, Li Li, Kevin Huck, Nick Trebon

1
TAU Parallel Performance System

• Allen D. Malony, Sameer S. Shende, Robert
  BellKai Li, Li Li, Kevin Huck, Nick Trebon
• malony,sameer,bertie,likai,lili,khuck_at_cs.uorego
  n.edu
• Department of Computer and Information Science
• Performance Research Laboratory
• University of Oregon

2
Outline

• Motivation
• TAU architecture and toolkit
• Instrumentation
• Measurement
• Analysis
• Example applications
• Users of TAU
• Conclusion

3
Problem Domain

• ASCI defines leading edge parallel systems and
  software
• Large-scale systems and heterogeneous platforms
• Multi-model simulation
• Complex, multi-layered software integration
• Multi-language programming
• Mixed-model parallelism
• Complexity challenges performance analysis tools
• System diversity demands tool portability
• Need for cross- and multi-language support
• Coverage of alternative parallel computation
  models
• Operate at scale

4
Tools for Performance Problem Solving

• Empirical-based performance optimization process
• Understand performance technology concerns

5
TAU Performance System

• Tuning and Analysis Utilities (11 year project
  effort)
• Performance system framework for scalable
  parallel and distributed high-performance
  computing
• Targets a general complex system computation
  model
• Entities nodes / contexts / threads
• Multi-level system / software / parallelism
• Measurement and analysis abstraction
• Integrated toolkit for performance
  instrumentation, measurement, analysis, and
  visualization
• Portable performance profiling and tracing
  facility
• Open software approach with technology
  integration
• University of Oregon , Forschungszentrum Jülich,
  LANL

6
TAU Performance Systems Goals

• Multi-level performance instrumentation
• Multi-language automatic source instrumentation
• Flexible and configurable performance measurement
• Widely-ported parallel performance profiling
  system
• Computer system architectures and operating
  systems
• Different programming languages and compilers
• Support for multiple parallel programming
  paradigms
• Multi-threading, message passing, mixed-mode,
  hybrid
• Support for performance mapping
• Support for object-oriented and generic
  programming
• Integration in complex software systems and
  applications

7
General Complex System Computation Model

• Node physically distinct shared memory machine
• Message passing node interconnection network
• Context distinct virtual memory space within
  node
• Thread execution threads (user/system) in context

Interconnection Network
Inter-node messagecommunication


Node
Node
Node
node memory
memory
memory
SMP
physicalview
VM space

modelview

Context
Threads
8
TAU Performance System Architecture
ParaProf
9
TAU Instrumentation Approach

• Support for standard program events
• Routines
• Classes and templates
• Statement-level blocks
• Support for user-defined events
• Begin/End events (user-defined timers)
• Atomic events
• Selection of event statistics
• Support definition of semantic entities for
  mapping
• Support for event groups
• Instrumentation optimization

10
TAU Instrumentation

• Flexible instrumentation mechanisms at multiple
  levels
• Source code
• manual
• automatic
• C, C, F77/90/95 (Program Database Toolkit
  (PDT))
• OpenMP (directive rewriting (Opari), POMP spec)
• Object code
• pre-instrumented libraries (e.g., MPI using PMPI)
• statically-linked and dynamically-loaded (e.g.,
  Python)
• Executable code
• dynamic instrumentation (pre-execution)
  (DynInstAPI)
• virtual machine instrumentation (e.g., Java using
  JVMPI)

11
TAU Source Instrumentation

• Automatic source instrumentation (TAUinstr)
• Routine entry/exit and class method entry/exit
• Block entry/exit and statement level (to be
  added)
• Uses an instrumentation specification file
• Include/exclude list for events and files
• Uses command line options for group selection
• Instrumentation event selection (TAUselect)
• Automatic generation of instrumentation
  specification file
• Instrumentation language to describe event
  constraints
• Event identity and location
• Event performance properties (e.g., overhead
  analysis)
• Create TAUselect scripts for performance
  experiments

12
Program Database Toolkit (PDT)

• Program code analysis framework
• develop source-based tools
• High-level interface to source code information
• Integrated toolkit for source code parsing,
  database creation, and database query
• Commercial grade front-end parsers
• Portable IL analyzer, database format, and access
  API
• Open software approach for tool development
• Multiple source languages
• Implement automatic performance instrumentation
  tools
• tau_instrumentor

13
Program Database Toolkit (PDT)
Application / Library
C / C parser
Fortran parser F77/90/95
Program documentation
PDBhtml
Application component glue
IL
IL
SILOON
C / C IL analyzer
Fortran IL analyzer
C / F90/95 interoperability
CHASM
Program Database Files
Automatic source instrumentation
TAU_instr
DUCTAPE
14
PDT 3.0 Functionality

• C statement-level information implementation
• for, while loops, declarations, initialization,
  assignment
• PDB records defined for most constructs
• DUCTAPE
• Processes PDB 1.x, 2.x, 3.x uniformly
• PDT applications
• XMLgen
• PDB to XML converter (Sottile)
• Used for CHASM and CCA tools
• PDBstmt
• Statement callgraph display tool

15
PDT 3.0 Functionality (continued)

• Cleanscape Flint parser fully integrated for
  F90/95
• Flint parser is very robust
• Produces PDB records for TAU instrumentation
  (stage 1)
• Linux x86, HP Tru64, IBM AIX
• Tested on SAGE, POP, ESMF, PET benchmarking codes
• Full PDB 2.0 specification (stage 2) Q1 04
• Statement level support (stage 3) Q3 04
• PDT 3.0 release at SC2003

16
TAU Performance Measurement

• TAU supports profiling and tracing measurement
• Robust timing and hardware performance support
• Support for online performance monitoring
• Profile and trace performance data export to file
  system
• Selective exporting
• Extension of TAU measurement for multiple
  counters
• Creation of user-defined TAU counters
• Access to system-level metrics
• Support for callpath measurement
• Integration with system-level performance data
• Linux MAGNET/MUSE (Wu Feng, LANL)

17
TAU Measurement with Multiple Counters

• Extend event measurement to capture multiple
  metrics
• Begin/end (interval) events
• User-defined (atomic) events
• Multiple performance data sources can be queried
• Associate counter function list to event
• Defined statically or dynamically
• Different counter sources
• Timers and hardware counters
• User-defined counters (application specified)
• System-level counters
• Monotonically increasing required for begin/end
  events
• Extend user-defined counters to system-level
  counter

18
Performance Analysis and Visualization

• Analysis of parallel profile and trace
  measurement
• Parallel profile analysis
• ParaProf
• ParaVis
• Profile generation from trace data
• Performance database framework (PerfDBF)
• Parallel trace analysis
• Translation to VTF 3.0 and EPILOG
• Integration with VNG (Technical University of
  Dresden)
• Online parallel analysis and visualization

19
ParaProf Framework Architecture

• Portable, extensible, and scalable tool for
  profile analysis
• Try to offer best of breed capabilities to
  analysts
• Build as profile analysis framework for
  extensibility

20
Profile Manager Window
21
Pprof Output (NAS Parallel Benchmark LU)

• Intel QuadPIII Xeon
• F90 MPICH
• Profile - Node - Context - Thread
• Events - code - MPI

22
ParaProf (NAS Parallel Benchmark LU)
Routine profile across all nodes
node,context, thread
Global profiles
Event legend
Individual profile
23
TAU Vampir (NAS Parallel Benchmark LU)
Callgraph display
Timeline display
Parallelism display
Communications display
24
Case Study SAMRAI (LLNL)

• Structured Adaptive Mesh Refinement Application
  Infrastructure (SAMRAI)
• Programming
• C and MPI
• SPMD
• Instrumentation
• PDT for automatic instrumentation of routines
• MPI interposition wrappers
• SAMRAI timers for interesting code segments
• timers classified in groups (apps, mesh, )
• timer groups are managed by TAU groups

25
Full Profile Window (Exclusive Time)
512 processes
26
Node / Context / Thread Profile Window
27
Derived Metrics
28
Full Profile Window (Metric-specific)
512 processes
29
ParaProf Enhancements

• Readers completely separated from the GUI
• Access to performance profile database
  
  
• Profile translators
• mpiP, papiprof, dynaprof
• Callgraph display
• prof/gprof style with hyperlinks
• Integration of 3D performance plotting library
• Scalable profile analysis
• Statistical histograms, cluster analysis,
• Generalized programmable analysis engine
• Cross-experiment analysis

30
ParaVis

• Scalable parallel profile analysis
• Scalable performance displays
• 3D graphics
• Analysis across profile samples
• Allow for runtime use
• Animated / interactive visualization
• Initially develop with SCIRun
• Computational environment
• Performance graphics toolkit
• Portable plotting library
• OpenGL

Performance Visualizer
Performance Analyzer
Performance Data Reader
31
Performance Visualization in SCIRun
SCIRun program
EVH1, IBM
EVH1, Linux IA-32
32
Terrain Visualization (Full profile)
F
Uintah
33
Scatterplot Visualization

• Each pointcoordinatedeterminedby threevalues
• MPI_Reduce
• MPI_Recv
• MPI_Waitsome
• Min/Maxvalue range
• Effective forclusteranalysis

Uintah
34
Bargraph Visualization (MPI routines)
Uintah, 512 processes, ASCI Blue Pacific
35
TAU Performance Database Framework

• profile data only
• XML representation
• project / experiment / trial

36
PerfDBF Browser
37
PerfDBF Cross-Trial Analysis
38
TAU Performance System Status

• Computing platforms (selected)
• IBM SP / pSeries, SGI Origin 2K/3K, Cray T3E /
  SV-1 / X1, HP (Compaq) SC (Tru64), Sun, Hitachi
  SR8000, NEC SX-5/6, Linux clusters (IA-32/64,
  Alpha, PPC, PA-RISC, Power, Opteron), Apple
  (G4/5, OS X), Windows
• Programming languages
• C, C, Fortran 77/90/95, HPF, Java, OpenMP,
  Python
• Thread libraries
• pthreads, SGI sproc, Java,Windows, OpenMP
• Compilers (selected)
• Intel KAI (KCC, KAP/Pro), PGI, GNU, Fujitsu, Sun,
  Microsoft, SGI, Cray, IBM (xlc, xlf), Compaq,
  NEC, Intel

39
Selected Applications of TAU

• Center for Simulation of Accidental Fires and
  Explosion
• University of Utah, ASCI ASAP Center, C-SAFE
• Uintah Computational Framework (UCF) (C)
• Center for Simulation of Dynamic Response of
  Materials
• California Institute of Technology, ASCI ASAP
  Center
• Virtual Testshock Facility (VTF) (Python, Fortran
  90)
• Los Alamos National Lab
• Monte Carlo transport (MCNP) (Susan Post)
• Full code automatic instrumentation and profiling
• ASCI Q validation and scaling
• SAICs Adaptive Grid Eulerian (SAGE) (Jack
  Horner)
• Fortran 90 automatic instrumentation and profiling

40
Selected Applications of TAU (continued)

• Lawrence Livermore National Lab
• Overturen
• Radiation diffusion (KULL)
• C automatic instrumentation, callpath profiling
• Sandia National Lab
• DOE CCTTSS SciDAC project
• Common component architecture (CCA) integration
• Combustion code (C, Fortran 90, GrACE, MPI)
• Center for Astrophysical Thermonuclear Flashes
• University of Chicago / Argonne, ASCI ASAP Center
• FLASH code (C, Fortran 90, MPI)

41
Concluding Remarks

• Complex ASCI parallel systems and software pose
  challenging performance analysis problems that
  require robust methodologies and tools
• To build more sophisticated performance tools,
  existing proven performance technology must be
  utilized
• Performance tools must be integrated with
  software and systems models and technology
• Performance engineered software
• Function consistently and coherently in software
  and system environments
• TAU performance system offers robust performance
  technology that can be broadly integrated

42
Acknowledgements

• Department of Energy (DOE)
• MICS office
• DOE 2000 ACTS contract
• Performance Technology for Tera-class Parallel
  Computer Systems Evolution of the TAU
  Performance System
• Performance Analysis of Parallel Component
  Software
• University of Utah, DOE ASCI Level 1 sub-contract
• DOE ASCI Level 3 (LANL, LLNL)
• NSF National Young Investigator (NYI) award
• Research Centre Juelich
• John von Neumann Institute for Computing
• Dr. Bernd Mohr
• Los Alamos National Laboratory