PerfTrack: Scalable Performance Diagnosis of High End Systems

1
PerfTrack Scalable Performance Diagnosis of
High End Systems

• Karen L. Karavanic
• Portland State University
• Dr. John May
• Center for Advanced Scientific Computing
• Lawrence Livermore National Laboratoryss

2
High End Computing The Cutting Edge

• Blue Gene/L beta-System
• IBM/DOE, Rochester, U.S.
• Processor 0.7 GHz PowerPC 440
• Achieved performance 70,720/91,750 Gflops
• Number of Processors 32,768
• SGI Altix 1.5GHz at NASA/Ames Research Center
• Achieved performance 51,870/60,960 Gflops
• Number of Processors 10,160
• Coming Soon petaflop systems

3
PetaFLOPS Computing WhattaFLOPS ???

• Floating Point Operations Per Second
• MFLOPS 10 6
• GFLOPS 10 9
• TFLOPS 10 12
• PFLOPS 10 15 1,000,000,000,000,000A million
  billion

4
Why do we need PetaFLOPS??

• Understanding global climate change
• Applying genomics /proteomics to human health
• Predicting natural disasters - earthquakes,
  tsunamis, etc.
• Address fundamental intellectual questions (the
  formation of the universe, the fundamental
  character of matter)
• Simulation
• Time to answer can save lives
• One story U.S. National Atmospheric Release
  Advisory Center (narac.llnl.gov)

5
Performance Diagnosis

• What is the scaling behavior of my code?
• How do these results compare to other
  platforms?
• Did performance of foo improve when we compiled
  with -O3? By how much?
• Are the performance requirements being met?
• Whats the max I/O wait time weve seen for runs
  on more than 16 nodes?
• How accurate was my predictive model?

6
Performance Diagnosis

• "Why did an application's performance change
  unexpectedly?"
• "How do different versions of compilers, runtime
  libraries, hardware, and system software affect a
  specific application's performance?"
• "What general correlations can we observe between
  these external influences and the performance of
  a suite of applications?"

7
Performance Diagnosis

• Performance diagnosis must include data from a
  number of different executions of the same
  application
• different platforms
• different OS and library versions
• code enhancements
• different data sets
• Performance Tuning as Scientific Experimentation
• An experiment is an instrumented program run
• Name, store, analyze, visualize data from many
  experiments

8
Our Solution PerfTrack

• Guiding Vision
• Provide developers with a better understanding of
  application and system performance
• Our Approach
• Integrate database technology into a performance
  analysis tool
• Collect and store as much information as possible
  about each run of an application
• Integrate and develop performance analysis
  techniques
• Collaborate with application developers through
  release of a working prototype

9
Our Solution PerfTrack

• Track performance results from large application
  runs
• Database stores performance information and data
  about how each application was built and executed
• Observe affect on performance over time of
  changes in the application, its compilation
  environment, runtime libraries, and hardware and
  software environment
• Variety of performance tools, benchmark output
  files, etc.

10
Our Solution PerfTrack

• Research Challenges
• Scalability
• Heterogeneous data
• Extensibility
• Ease of Use
• Detecting Relationships Data Mining and Machine
  Learning in the Parallel Performance Domain
• Diagnosing Performance Identifying the root
  cause OS, middleware, application

11
PerfTrack Design

• Design goal No knowledge of database technology
  or vocabulary should be required to use PerfTrack
• Design goal PerfTrack must be scalable to 1000s
  of program runs and 100,000s of performance
  results
• Design goal PerfTrack must be flexible enough
  to store data from different measurement tools
  and different types of performance studies
• Design goal PerfTrack must be extensible to
  accommodate future tool innovations
• Design goal PerfTrack can be used with a
  variety of DBMS packages

12
PerfTrack Prototype

• Data Collection
• Build environment
• Run environment
• Performance Data
• PT data format (PTdf)
• ResourceType, Application, Resource,
  ResourceAttribute, PerfResult
• PTdataStore interface
• Shelters PerfTrack user from the DBMS
• DBMS approach
• Support for Oracle or PostgreSQL DBMS
• generic schema

13
Project PerfTrack-A

• Project Title Automated Collection of Data to
  Describe Code Build Environment
• Develop automated techniques for gathering
  information about the software and hardware
  environment related to each particular code
  build, and for updating this information as
  things change.
• A robust solution is required for direct use in a
  national science lab environment -- characterized
  by rapid pace of updates and frequent arrival of
  experimental and novel architectures
• Data may be located in web pages, email updates,
  or may be gathered directly from the host systems
• Code to be written in python for use in Linux,
  Solaris, and MacOSX environments

14
Project PerfTrack-B

• Project Title Automated Comparative Analysis of
  Parallel Program and Machine Performance
• Develop techniques for analyzing performance data
  from runs of large-scale parallel and distributed
  programs. The code to be developed is a front
  end that uses a well-defined interface to an
  underlying data store (a PostgreSQL database in
  our PSU prototype).
• Starting point existing GUI functionality
• Innovation new functionality to select subsets
  of the large data for analysis sessions, using
  simple statistical or data mining approaches
• Code to be written in python for use in Linux,
  Solaris, and MacOSX environments

15
Acknowledgments

• This research supported in part by UC/LLNL
  subcontract B539302.
• Portions of this work were performed under the
  auspices of the U.S. Department of Energy by the
  University of California Lawrence Livermore
  National Laboratory under contract No.
  W-7405-Eng-48.
• PerfTrack contributors include John May, Brian
  Miller (Lawrence Livermore National Laboratory)
  Kevin Huck (U of Oregon) Rashawn Knapp, Kathryn
  Mohror, Brian Pugh, Travis Spencer, Eric Wilson
  (Portland State)
• http //www.cs.pdx.edu/karavan
• karavan_at_cs.pdx.edu

16
For More Information

• Karavanic, May, Mohror, Miller, Huck, Knapp,
  Pugh, "Integrating Database Technology with
  Comparison-based Parallel Performance Diagnosis
  The PerfTrack Performance Experiment Management
  Tool," SC2005, November 2005, Seattle, WA
  available at www.supercomp.org
• karavan_at_cs.pdx.edu
• My office hour Wed 430 - 6pm