Allen D. Malony

1
TAU Performance DataBaseFramework (PerfDBF)

• Allen D. Malony
• malony_at_cs.uoregon.edu
• Department of Computer and Information Science
• Computational Science Institute
• University of Oregon

2
Outline

• Motivation for performance databases
• TAU performance system
• TAU Performanc DataBase Framework
• Architecture
• XML profile data representation
• Example
• Performance engineering in software engineering
• X-PARE (eXPeriment Alerting and Reporting)
• Concluding remarks

3
Why Performance Databases?

• Focus on empirical performance optimization
  process
• Necessary for multi-results performance analysis
• Multiple experiments (codes, versions, platforms,
  )
• Historical performance comparison
• Integral component of performance analysis
  framework
• Improved performance analysis architecture design
• More flexible and open tool interfaces
• Supports extensibility and foreign tool
  interaction
• Performance analysis collaboration
• Performance tool sharing
• Performance data sharing and knowledge base

4
Empirical-Based Performance Optimization
Process
5
TAU Performance System Framework

• Tuning and Analysis Utilities (aka Tools Are Us)
• Performance system framework for scalable
  parallel and distributed high-performance
  computing
• Targets a general complex system computation
  model
• nodes / contexts / threads
• Multi-level system / software / parallelism
• Measurement and analysis abstraction
• Integrated toolkit for performance
  instrumentation, measurement, analysis, and
  visualization
• Portable performance profiling/tracing facility
• Open software approach

6
TAU Performance System Architecture
Paraver
EPILOG
7
TAU Performance Database Framework

• profile data only
• XML representation
• project / experiment / trial

8
PerfDBF Components

• Performance Data Meta Language (PerfDML)
• Common performance data representation
• Performance meta-data description
• PerfDML translators to common data representation
• Performance DataBase (PerfDB)
• Standard database technology (SQL)
• Free, robust database software (PostgresSQL)
• Commonly available APIs
• Performance DataBase Toolkit (PerfDBT)
• Commonly used modules for query and analysis
• Facility analysis tool development

9
Common and Extensible Profile Data Format

• Goals
• Capture data from profile tools in common
  representation
• Implement representation in a standard format
• Allow for extension of format for new profile
  data objects
• Base on XML (obvious choice)
• Leverage XML tools and APIs
• XML parsers, Suns Java SDK,
• XML verification systems (DTD and schemas)
• Target for profile data translation tools
• eXtensibile Stylesheet Language Transformations
  (XSLT)
• Which performance profile data are of interest?
• Focus on TAU and consider other profiling tools

10
Performance Profiling

• Performance data about program entities and
  behaviors
• Code regions functions, loops, basic blocks
• Actions or states
• Statistics data
• Execution time, number of calls, number of FLOPS
  ...
• Characterization data
• Parallel profiles
• Captured per process and/or per thread
• Program-level summaries
• Profiling tools
• prof/gprof, ssrun, uprofile/dpci, cprof/vprof,

11
TAU Parallel Performance Profiles
12
PerfDBF Example

• NAS Parallel Benchmark LU
• configure -mpiinc/usr/include
  -mpilib/usr/lib64 -archsgi64-fortransgi
  -SGITIMERS -useropt-O2

NPB profiled With TAU
Standard TAU Output Data
TAU XML Format
TAU to XML Converter
Database Loader
SQL Database
AnalysisTool
13
Scalability Analysis Process

• Scalability study on LU
• suite.def of procs -gt 1, 2, 4, and 8
• mpirun -np 1 lu.W1
• mpirun -np 2 lu.W2
• mpirun -np 4 lu.W4
• mpirun -np 8 lu.W8
• populateDatabase.sh
• run Java translator to translate profiles into
  XML
• run Java XML reader to write XML profiles to
  database
• Read times for routines and program from
  experiments
• Calculate scalability metrics

14
Raw TAU Profile Data

• Raw data output
• One processor
• "applu 1 15 2939.096923828125 248744666.5830078
  0 GROUP"applu
• Four processors
• "applu 1 15 2227.343994140625 51691412.17797852
  0 GROUP"applu
• "applu 1 15 2227.343994140625 51691412.17797852
  0 GROUP"applu
• "applu " 1 14 596.568115234375 51691519.34106445
  0 GROUP"applu
• "applu " 1 14 616.833251953125 51691377.21313477
  0 GROUP"applu"

group name
profile calls
exclusive time
inclusive time
name
subs
calls
15
XML Profile Representation

• One processor
• ltinstrumentedobjgt
• ltfuncnamegt 'applu 'lt/funcnamegt
• ltfuncIDgt8lt/funcIDgt
• ltinclpercgt100.0lt/inclpercgt
• ltinclutimegt2.487446665830078E8lt/inclutimegt
  
• ltexclpercgt0.0lt/exclpercgt
• ltexclutimegt2939.096923828125 lt/exclutimegt
• ltcallgt1lt/callgt
• ltsubrsgt15lt/subrsgt
• ltinclutimePcallgt2.487446665830078E8lt/inclu
  timePcallgt
• lt/instrumentedobjgt

16
XML Representation

• Four processor mean
• ltmeanfunctiongt
• ltfuncnamegt'applu 'lt/funcnamegt
• ltfuncIDgt12lt/funcIDgt
• ltinclpercgt100.0lt/inclpercgt
• ltinclutimegt5.169148940026855E7lt/inclutimegt
  
• ltexclpercgt0.0lt/exclpercgt
• ltexclutimegt1044.487548828125lt/exclutimegt
• ltcallgt1lt/callgt
• ltsubrsgt14.25lt/subrsgt
• ltinclutimePcallgt5.1691489E7lt/inclutimePcal
  lgt
• lt/meanfunctiongt

17
Contents of Performance Database
18
Scalability Analysis Results

• Scalability of LU performance experiments
• Four trial runs
• Funname processors meanspeedup
• .
• applu 2 2.0896117809566
• applu 4 4.812100975788783
• applu 8 8.168409581149514
• exact 2 1.95853126762839071803
• exact 4 4.03622321124616535446
• exact 8 7.193812137750623668346

19
Current Status and Future

• PerfDBF prototype
• TAU profile to XML translator
• XML to PerfDB populator
• PostgresSQL database
• Java-based PostgresSQL query module
• Use as a layer to support performance analysis
  tools
• Make accessing the Performance Database quicker
• Continue development
• XML parallel profile representation
• Basic specification
• Opportunity for APART to define a common format

20
Performance Tracking and Reporting

• Integrated performance measurement allows
  performance analysis throughout development
  lifetime
• Applied performance engineering in software
  design and development (software engineering)
  process
• Create performance portfolio from regular
  performance experimentation (couple with software
  testing)
• Use performance knowledge in making key software
  design decision, prior to major development
  stages
• Use performance benchmarking and regression
  testing to identify irregularities
• Support automatic reporting of performance bugs
• Enable cross-platform (cross-generation)
  evaluation

21
XPARE - eXPeriment Alerting and REporting

• Experiment launcher automates measurement /
  analysis
• Configuration and compilation of performance
  tools
• Instrumentation control for Uintah experiment
  type
• Execution of multiple performance experiments
• Performance data collection, analysis, and
  storage
• Integrated in Uintah software testing harness
• Reporting system conducts performance regression
  tests
• Apply performance difference thresholds (alert
  ruleset)
• Alerts users via email if thresholds have been
  exceeded
• Web alerting setup and full performance data
  reporting
• Historical performance data analysis

22
XPARE System Architecture
Experiment Launch
Performance Database
Performance Reporter
Comparison Tool
Regression Analyzer
Alerting Setup
23
Experiment Results Viewing Selection
24
Web-Based Experiment Reporting
25
Web-Based Experiment Reporting (continued)
26
Alerting Setup
27
Other Performance Database Projects

• HPM Toolkit (DeRose IBM)
• PPerfDB (Karavanic Portland State University)
• HPCView (Mellor-Crummey, Fowler Rice)
• SCALEA (Fahringer, Truong University of Vienna)
• EXPERT (Mohr, Wolf Research Center Juelich)

28
Acknowledgements

• University of Oregon
• Li Li, Robert Bell, Sameer Shende
• University of Utah
• Alan Morris, Steve Parker, Dav St. Germain
• Department of Energy (DOE), ASCI
  AcademicStrategic Alliances Program (ASAP)
• Computational Science Institute, ASCI/ASAPLevel
  3 projects with LLNL / LANL,University of
  Oregonhttp//www.csi.uoregon.edu
• Center for the Simulation of Accidental Fires
  andExplosions (C-SAFE), ASCI/ASAP Level 1
  center, University of Utahhttp//www.csafe.utah.
  edu