The UNIX Benchmark Process

1
The UNIX Benchmark Process
Principal Center For Workgroup Hardware and
Software
May 2001
2
Background

• UNIX benchmark process has deteriorated since it
  was first established
• Experience dictated that modifications needed to
  be made and the process more closely monitored.
• Alterion and NASA have worked closely to develop
  a new process which alleviates all currently
  identified problems.
• Others will probably be found

3
Existing Benchmark Process Had Significant
Problems

• Rankings included server class machines not
  appropriate for use as NASA desktop systems
• Spec rate benchmarks were used to develop UNIX
  ratings
• Weighted high volume systems (lots of RAM, lots
  of processors) higher than typical desktop
  platforms
• Introduction of Spec2000 resulted in existing
  architectures being excluded from listings

4
Current Process Introduces Solutions

• Server class systems have been eliminated from
  the listings by developing Criteria For A Desktop
  System
• Spec baseline performance benchmarks will be used
  instead of rate benchmarks to more closely
  approximate NASAs UNIX desktop usage.
• Alterion will use Spec2000 numbers exclusively.
  
• NASA has joined Spec and will conduct certified
  benchmark testing on key desktop configurations
• Results will be submitted directly to Spec

5
Criteria For A Desktop System

• Standard Office Power (110V), not 220V
• No Special Environmental Accommodations
• Not Excessively Loud or Hot
• Not Excessively Heavy
• Supports Common Input Devices (keyboard, mouse,
  dials, tablet, etc.)
• Supports Attachment of a Monitor
• Supports 24-bit color graphics and 3D
  acceleration

6
Features of The New Benchmark Process

• ODIN Program Office and Delivery Order Managers
  involved
• NASA PC/WHS helps complete available Spec
  performance ratings
• ODIN Vendors and UNIX hardware manufactures will
  be engaged
• Alterion will provide benchmark weightings and
  publish rankings bi-annually, rather than
  quarterly
• Alterion will certify ODIN proposed Attachment R
  UNIX offerings
• NASA will sponsor forums for process improvement

7
Available Documentation

• A detailed discussion of the problems with the
  previous benchmark process and proposed (now
  accepted) solutions.
• The documented process for coordinating the
  efforts of the ODIN delivery order vendors, the
  PC/WHS Spec Testing Lab, the ODIN Program Office
  and DOCOTRs and Alterion.