Parallel File System Benchmarking Project

1
Parallel File SystemBenchmarking Project
Tracey Wilson DICE, Program Manager
CSC twilso23_at_csc.com
2
Outline

• DICE Overview
• What is the DICE TAP?
• The File System Comparison Issue and Why?
• DICE Plan for Framework
• Normalization
• Summary

3
Nationwide Test Environment for Technology
Innovations
4
Testing Validation ServicesIndependent 3rd
Party Evaluations
Evaluating HPC, Storage, Networking and Data
Management Technologies
5
DICE Architecture

• DICE utilizes a geographically distributed test
  bed
• Real world research networks
• DICE sites indicative of HPC Data Centers

6
Quite SimplyA Test Bed for Advancing Innovation
Independent, Unbiased 3rd Party Evaluations
Genesis of a Vision by Government, Industry
Academia to Address Critical HPC Technology
Challenges
7
Community Input is Key

• The DICE Technical Advisory Panel
• Advises DICE on
• Technology directions
• Project concepts
• Testing methodologies
• Utilizes working groups to focus on more specific
  issues
• Current members include

Kevin Regimbal, PNNL Chris Jordan, TACC
Lee Ward, Sandia Steve Conway, IDC
Kevin Wilson, PG Dan Duffy, NASA
Ron Bewtra, NOAA Lloyd Slonaker, DoD
Matt Leininger, LLNL
8
What is the File System Issue?

• Lack of standardized metrics for performance
• Need exists for file system performance
  comparison
• Need exists for defined benchmarks for local and
  remote file systems
• Different HPC systems and their file system
  architectures have no direct correlations
• Scaling
• Scaling of current file systems has unpredictable
  performance impacts
• Load
• Performance varies with file systems with loads
  of 50-70
• Fragmented data is not accounted for

9
Why do HPC Centers Care?

• Centers need input for
• New procurements
• User requirements
• Application performance
• File systems are a critical piece of an HPC
  solution
• How do you as a Center Director make an informed
  decision?

Can you prove or rather disprove the performance
or value of one file system compared to another?
10
How did the Effort Begin?

• In 2007, DICE received 3 proposals for evaluation
  projects
• Panasas
• DoD
• Microsoft
• DICE Advisory Panel offered new solution
• Develop a standard benchmark framework for file
  system evaluation

1. Make the framework expandable to use
2. Make it fully available for the HPC community to
   use
3. Effort was to include normalization of results

11
Benefits of Standardization

• Standardized Metric
• One set of benchmarks for all parallel file
  systems
• Results will be normalized for comparison
• HPC Community needs non-biased benchmark(s)
• Increased understanding of the impacts on current
  and proposed upgrades to a centers file system
• The storage subsystem performance
• Scaling
• Load performance

12
Current Ideas

• Parameterized Benchmark Suite
• Simulate many different workloads
• Expose more tuning/configuration options in the
  benchmarks
• Suite Extensible so users can add new benchmarks
• Have unique benchmarks for synthetic workloads
• Transactions
• Streaming
• Random I/O
• Various Read/Write Ratios

13
What is the plan?

• Currently surveying the HPC benchmark and file
  system community including
• Carnegie Melon
• University of California/Santa Cruz
• ORNL
• Argonne National Labs
• PNNL
• TACC
• LLNL
• And many more

14
Application I/O Patterns

• Many different I/O intensive applications
• Investigate the development of an I/O analyzer
• Recognize I/O utilization of applications
• Similar to trace and trace replay
• Add this functionality to the framework to
  simulate application I/O patterns
• Verification of performance
• Can be applied on different architectures

15
Normalization

• Normalization is needed between differing
  architectures
• Need a formula
• Performance
• Cost
• Additional Criteria will be evaluated
• Aging and fragmentation of data should be
  considered
• Difficult to develop
• Need to develop on a small scale first
• Will the normalization hold as you scale systems?
• Plan is to create the framework first then
  normalize

16
Summary

• Lot of collaborative effort from the Team and the
  Community is required
• Full project plan is in draft
• Survey underway to identify current efforts
• Plan to have initial framework for file system
  testing by early 2010
• Plan to use this on other file system types

17
How To Get Involved

• Collaboration is key
• Developing a DICE Forum for this project
• Will be collecting traces
• DICE TAP will be forming specific working group
  for this effort
• Interested parties should send email to
  twilson_at_avetec.org
• DICE Website www.diceprogram.org