Formal Verification of Pipelined Processors

1
Thoughts on Metrics Benchmarking
Randal E. Bryant
2
Program Complexity

• Complexity Sources
• Control
• Concurrency
• Error handling mitigation
• Data
• Complex data structures
• Data structure invariants

Embedded Systems
Control
My Systems
Data
3
Role of Benchmarks

• Good
• Provides objective standard for comparison
• Competition drives progress
• Bad
• Can become goal rather than means to achieve goal
• Poorly selected set can thwart progress
• E.g., DIMACS SAT benchmarks

4
Benchmarking Desiderata

• Organic Process
• E.g., SPEC, SAT
• Continually update
• Large community involved
• Selection
• Range of sizes
• From different sources (applications,
  organizations)
• Desirable, but Uncommon
• Parameterized
• Access to originating application

5
Some Ideas

• Select Small Set of Driving Applications
• Control intensive embedded system(s)
• Data intensive
• Create Supporting Benchmarks
• Scaled down versions or components of driving
  applications
• Issues
• Language
• Ancilliary code Libraries, OS
• Specs.