TPC Benchmarks

1
TPC Benchmarks

• Sandeep Gonsalves
• CSE 8330 Project 1
• SMU
• May 1, 2004

2
Overview

• Benchmarks
• Benchmark Wars
• Establishment of the TPC
• TPC Benchmarks A, B, C, D, E, H, R, S, W
• Conclusion

3
Benchmarks

• Standard for comparison of various systems
  performing similar operations
• Set of programs that simulate a typical workload
  on a given system
• Used to measure system performance
• Metrics such as speed, performance, price etc.
  are recorded
• Used to determine the optimal system

4
Domain-specific benchmarks

• No single metric can measure the performance of
  computer systems on all applications 1
• Performance of a system varies tremendously from
  one application domain to another 1

5
Domain-specific benchmarks

• Key criteria for a domain-specific benchmark to
  be useful
• Relevant
• Portable
• Scaleable
• Simple 1

6
Early Benchmarks

• TP1 benchmark
• Wisconsin benchmark

7
Benchmark Wars

• Benchmark wars happen when one vendor publishes
  superior results for an important benchmark
  evaluation and the other vendors or individuals
  try to get back by improving their numbers 2
• Consequences
• Abnormal results
• Waste of resources
• Confusion !

8
Initial Efforts

• Working Group on Performance Measurement
  Standards (WG-PMS) 2
• Anon et.al. 1985
• Three standard performance tests for OLTP systems
• Most popular DebitCredit 2

9
Establishment of the TPC

• Motivation
• Need for the competition to get civilized
• Put an end to manipulations
• Stop the confusion in the industry
• Form an industry forum to enforce OLTP
  performance measurement standards 2

10
TPC

• TPC is the acronym for the Transaction Processing
  Performance Council
• Established on 10th August, 1988
• Founder Omri Serlin
• Co-founder Tom Sawyer
• Initial 8 members Control Data Corp, Digital
  Equipment Corp, ICL, Pyramid Technology, Stratus
  Computer, Sybase, Tandem Computers and Wang
  Laboratories 2

11
TPC

• First standard TPC-A published in November, 1989
• TPC-A Councils version of the DebitCredit
  benchmark test 2
• Published the second standard TPC-B in August
  1990
• TPC-B Councils version of the TP1 benchmark
  test 2

12
TPC

• Major changes brought about by the TPC 2
• Submission of a Full Disclosure Report (FDR)
• Results must be audited by a TPC certified
  individual

13
TPC Members - 2004
14
TPC Auditors - 2004
15
TPC Benchmark A

• Issued in November 1989
• Obsolete as of 6th June, 1995
• Measure performance of update-intensive database
  environments 5
• e.g.. OLTP applications with characteristics
  like
• Multiple on-line terminal sessions
• Significant disk input/output
• Moderate system and application execution time
• Transaction integrity 5

16
TPC Benchmark A

• Measures the transactions per second (tps) of a
  system as a measurement of performance 5
• Measure the performance of systems in a wide area
  or local area network configuration 5
• Metrics are TPC-A local Throughput and TPC-A
  wide Throughput measured in tps 6
• System configurations, details of the benchmark
  run, total cost mandatory part of FDR 6

17
TPC-A Specifications 5

• Specification consists of 11 clauses
• TPC-A is stated in terms of a hypothetical bank
  that has one or more branches and each branch has
  multiple tellers
• Each customer of the bank has an account
• Transactions denote customer operations such as
  withdrawals, deposits etc. performed by a teller
  at a branch

18
TPC-A Specifications

• Fig 4 ER Diagram of the components of the TPC
  Benchmark A database 5

19
TPC-A Specifications 5

• Any commercially available database management
  system (DBMS), database server, file system, etc
  can be used to implement the database
• The system under test must support the ACID
  properties of transaction processing systems
• Horizontal partitioning of files/tables is
  permitted
• Vertical partitioning of files/tables is not
  permitted

20
TPC-A Results 5

• Throughput of the system in units of transactions
  per second is
• tpsA-Local - local area networks
• tpsA-Wide - wide area networks
• Cost is given as price/tpsA
• what is tested and/or emulated is priced and
  what is priced is tested and/or emulated
• 5 year maintenance pricing must be included

21
TPC-A FDR Requirements 5

• Identification of sponsor of the benchmark
• Participating companies
• Program listings
• List of settings for parameters and options that
  are tunable by a customer and have been altered
  from the defaults in actual products
• Auditors name, address, phone number along with
  a copy of the auditors attestation letter
• Etc

22
TPC-A FDR Requirements 5

• In short
• All that a customer would need to replicate the
  results !
• The FDR document must be available to the public
  at a reasonable cost
• Official language is English

23
TPC Benchmark B

• Approved in August 1990
• Obsolete as of 6th June, 1995
• Quite different from TPC-A
• Database Stress Test
• Not OLTP oriented
• It focuses on database management systems (DBMS)
  applications and on the back-end database server
  8

24
TPC-B Specifications 8

• Specified in terms of a hypothetical bank
• Measures the total number of simultaneous
  transactions that a system can carry out
• No users, communication lines or terminals used
• Equivalent to electronic data processing (EDP)
  batch processing applications
• Metrics Throughput tpsB and the associated
  price-per-TPS

25
TPC Benchmark C

• Approved on July 23, 1992
• Currently in use with its latest version 5.2
• It is an OLTP benchmark
• It is centered on the transactions of a wholesale
  supplier managing orders which is an order-entry
  environment 14
• TPC-C metrics are new-order txn rate (tpmC) and
  price/performance (/tpmC) 14

26
TPC Benchmark C

• It measures the entire business operation and is
  a more extensive and complex yardstick for
  measuring the performance of an OLTP system 14
• Current version is the 20th revision since 1992
  12

27
TPC Benchmark D

• Approved on April 5, 1995
• Obsolete as of April 6, 1999
• Decision support benchmark
• Metrics are
• TPC-D Composite Query-per-Hour Metric (QphD_at_Size)
• TPC-D Price/Performance (/QphD_at_Size)
• Systems Availability Date 18

28
TPC Benchmark D

• It illustrates decision support systems that
• Examine large volumes of data
• Execute queries with a high degree of complexity
  
• Give answers to critical, frequently-asked
  business questions 18
• Its successors are TPC-H and TPC-R

29
TPC Benchmark H

• Approved on February 26, 1999
• Current version is 2.1.0 which was specified on
  August 14, 2003
• Decision support benchmark
• It executes a set of queries against a standard
  database under controlled conditions to evaluate
  the performance of various decision support
  systems 20

30
TPC Benchmark H

• Metrics are
• TPC-H Composite Query-per-Hour Metric (QphH_at_Size)
  which is the performance metric
• TPC-H Price/Performance (/QphH) which is the
  price-performance metric
• The Systems Availability Date 20

31
TPC Benchmark R

• Approved on February 26, 1999
• Current version is 2.1.0 which was specified on
  August 14, 2003
• Decision support benchmark
• Similar to TPC-H

32
TPC Benchmark R

• The TPC-R metrics are
• TPC-R Composite Query-per-Hour Metric (QphR_at_Size)
  which is the performance metric
• TPC-R Price/Performance (/QphR) which is the
  price-performance metric
• System Availability date 23

33
TPC Benchmark W

• TPCs latest benchmark
• First version 1.0 was approved on December 9,
  1999
• Transactional web e-Commerce benchmark that
  models a retail store on the internet like an
  online bookstore. 24
• TPC-W enables the testing of environments where a
  host of servers perform different functions 25

34
TPC Benchmark W

• TPC-W metrics are
• Number of web interactions processed per second
  (WIPS) which is the performance metric.
• The associated price per WIPS (/WIPS)
• The availability date of the priced configuration
  26
• It can verify the performance measurements of a
  variety of e-commerce servers in a real world
  internet environment 25

35
Aborted benchmark efforts

• TPC S
• Server version of the TPC-C
• Did not receive sufficient Council support
• Potential for abnormally high performance ratings
  22

36
Aborted benchmark efforts

• TPC E
• Enterprise benchmark
• No complex benchmark available that could stress
  large enterprise systems
• Could cause confusion
• Would cause vendors to spend additional resources
  
• Was germane to only a very small number of
  vendors competing in that arena 22

37
What's up with the TPC

• As of March 2004
• Working on a new TPC-E specification
• Working on a new decision support benchmark
  TPC-DS
• Carrying out revisions on the TPC-H, TPC-R, TPC-W

38
Conclusion - TPC

• Fair-competition in the industry
• End to bench marketing wars
• Widely used by all vendors
• Continuous evolution of its benchmarks

39
Conclusion - TPC Log
40
References

• 1 Jim Gray Database and Transaction
  Processing Performance Handbook. The Benchmark
  Handbook 1993. Online edition. http//www.benchmar
  kresources.com/handbook/index.html
• 2 Omri Serlin The History of DebitCredit and
  the TPC. The Benchmark Handbook 1993. Online
  edition.
• http//www.benchmarkresources.com/handbook/index.
  html
• 3 Anon, et al, A Measure of Transaction
  Processing Power, Datamation, V. 31.7, April
  1985, pp. 112-118.
• 4 Gray, J.N., Reuter, A., Transaction
  Processing Concepts and Techniques, Morgan
  Kaufmann, San Mateo, CA, 1993, pp. 11-12, 168.

41
References

• 5 TPC BENCHMARK A Standard Specification
  Revision 2.0 7June 1994
• http//www.tpc.org/tpca/spec/tpca_current.pdf
• 6 TPC-A
• http//www.tpc.org/tpca/default.asp
• 7 Hanson, Robert J., TPC Benchmark B - What It
  Means and How to Use It, ATT Global
  Information Solutions
• http//www.tpc.org/tpcb/default.asp
• 8 TPC BENCHMARK B Standard Specification
  Revision 2.0 7June 1994 http//www.tpc.org/tpcb/sp
  ec/tpcb_current.pdf

42
References

• 9 Bramer, Brian. System benchmarks, DeMontfort
  University, UK. http//www.cse.dmu.ac.uk/bb/Teach
  ing/ComputerSystems/SystemBenchmarks/BenchMarks.ht
  mlintroduction
• 10 Levine, Charles.,SIGMOD '97 Industrial
  Session 5 - 5/29/97. http//www.tpc.org/informatio
  n/sessions/sigmod/indexc.htm
• 11 Ozsu, T., P. Valderezs, Principles of
  Distributed Database Systems, Second Edition.
  Prentice Hall, Englewood Cliffs, NJ., 1999.
• 12 TPC BENCHMARK C Standard Specification
  Revision 5.2 December 2003 http//www.tpc.org/tpcc
  /spec/tpcc_current.pdf

43
References

• 13 Patterson, D. A., J. L. Hennessy, Computer
  Architecture, a Quantitative Approach, Morgan
  Kaufmann, San Mateo, CA, 1990. Chapter 1.
• 14 Raab, Francois et.al. Overview of the TPC
  Benchmark C The Order-Entry Benchmark.
• http//www.tpc.org/tpcc/detail.asp
• 15 TPC-C
• http//www.tpc.org/tpcc/default.asp
• 16 TPC-D
• http//www.tpc.org/tpcd/default.asp

44
References

• 17 Stephens, Jack. TPC-D. The Industry Standard
  Decision Support Benchmark
• http//www.tpc.org/information/sessions/sigmod/in
  dexc.htm
• 18 TPC BENCHMARKTM D (Decision Support)
  Standard Specification Revision 2.1
• http//www.tpc.org/tpcd/spec/tpcd_current.pdf
• 19 TPC-D
• http//www.tpc.org/tpcd/default.asp
• 20 TPC BENCHMARK H (Decision Support) Standard
  Specification Revision 2.1.0
• http//www.tpc.org/tpch/spec/tpch2.1.0.pdf

45
References

• 21 Levine, Charles. TPC Benchmarks, Microsoft.
  http//research.microsoft.com/gray/WICS_96_TP/
• 22 Draft White Paper. August 2, 1999. Object
  Management Group
• www.omg.org/docs/bench/99-08-02.ps
• 23 TPC-R
• http//www.tpc.org/tpcr/default.asp
• 24 TPC Benchmark W (Web Commerce)
  SpecificationVersion 1.8. Feb 19, 2002.
• http//www.tpc.org/tpcw/spec/tpcw_V1.8.pdf

46
References

• 25 Smith, Wayne D. TPC-W Benchmarking An
  Ecommerce Solution. Revision 1.2. Intel
  Corporation.
• http//www.tpc.org/tpcw/TPC-W_wh.pdf