Evaluation and Verification Engine EVE

1
Evaluation and Verification Engine (EVE)

• Dan Dietz
• ddietz_at_purdue.edu
• Michael Oberg - Mentor
• SIParCS Internship Program 2009
• National Center for Atmospheric Research

2
Outline

• Motivation
• Evaluation and verification
• Benchmarking and analysis
• Design
• Overview
• Implementation
• Results
• Examples
• Conclusion
• Future work

3
Background

• Research Systems Evaluation Team
• Evaluates emerging architectures
• Receives 6 to 12 new systems a year
• Maintains existing computer systems
• TeraGrid computational resource Frost
• Maelstrom storage cluster
• Improves computer systems
• Research to enhance infrastructure

4
Motivation

• Evaluation
• Steady influx of new systems
• Simplify new system evaluation
• Verification
• Monitor performance over time
• Verify changes made to systems

5
Motivation (continued)

• Benchmark
• Large variety of benchmarks
• Time consuming to configure and run
• Analysis
• Results are in various formats
• Need to unify data into central database
• Re-output and visualize data

6
Design

• Technologies used
• Python
• PostgreSQL
• Shell scripting
• Ploticus
• PHP

7
Design (continued...)

• Database

Set
Group
Group
IOR 16,32,64,128
popio 16,32,64,128
8
Design (continued)

• Benchmark

CLI
Cmd file
Resource
EVE
DB
Web
9
Design (continued)

• Analysis

CLI
Text
Cmd file
EVE
EVE (benchmark stage)
Plots
Web
10
Results
11
Results (continued)
12
Results (continued)
13
Results (continued)
14
Results (continued)
15
Conclusions

• EVE
• reduces steps to run benchmarks
• organizes data in central location
• simplifies analysis of data
• provides consistency and repeatability of
  benchmarking and analysis

16
Future Work

• EVE is a test suite for performance
• Currently focuses on storage
• CPU architecture
• Network
• Application performance
• Expand supported systems and benchmarks within
  ReSET environ
• Web interface
• Alerts

17

• Thank you
• Questions?
• Dan Dietz
• ddietz_at_purdue.edu