Benchmarking Benchmarks

1
Benchmarking Benchmarks

• Project 3
• cs533

2
Introduction

• Benchmarks are uniform way of comparing
  performance on different computer systems
• However, if benchmark is not setup properly, it
  will not adequately reflect actual application
  (workload) performance
• You are to test the effectiveness of a benchmark
  of your choosing with reflecting the application
  workload of your creation

3
Overview

• Select benchmark
• Install benchmark
• Run benchmark
• Create application-level benchmark
• Report results

4
Select Benchmark

• Well-known
• described peer-reviewed forum
• published results for computer systems for
  comparison
• Possibilities
• HINT
• SPEC CPU2000
• Linpack
• BYTEMark
• Quake 3
• Bonnie

5
Install Benchmark

• Download
• Read documentation
• Configure
• Run validation tests (as appropriate)
• Figure out how to run it
• Figure out output generated

6
Run Benchmark

• Run benchmark
• Appropriate flags, number of iterations
• At least two different systems
• Systems should be selected so as to get
  different results
• Ex CPU speeds, memory sizes, architectures

7
Create Application Level Benchmark

• Create your own workload
• Design so can compare with benchmark
• Examples
• Compile large software system
• Run large simulation
• Perform numerous database transactions

8
Report Results (1 of 2)

• Background on well-known benchmark
• who, where, when invented
• where published
• what the intent of the benchmark is
• measures of performance and some published
  performance results.
• Descriptions of the systems you run
• CPU type and speed
• memory size
• cache levels and sizes
• operating system

9
Report Results (2 of 2)

• Details on your application level benchmark
• Type of workload
• Length of run
• Number of iterations
• Measure of performance
• Results and analysis depicting results clearly
• tables and graphs
• appropriate stats and presentation techniques
• Conclusions on the meaning of your data
• data that supports your conclusions
• subjective opinions you may have