1. An Intro. to Database Benchmarks
June 2001
Prof. Sang Ho Lee
Soongsil University

2. Benchmarks: What and Why
To measure the performance of a hardware and software system in a controlled environment using a standard methodology
Performance tests by which different database hardware/software platforms are compared in terms of price and performance
Motivated and driven by database customer demand (usually, to make acquisition decisions easy)

3. Performance Assurance
Once you build the system you must do
Quality assurance: Functionality validation
Performance assurance: benchmark
find the performance bugs
looking for bottlenecks (queues)
When performance assurance is complete, model can be used to predict performance on higher loads, new hardware. This is called “capacity planning”

4. Custom Benchmark vs. Generic Benchmark
Created by a particular customer based on a specific application
Can measure precisely capabilities that the customer wants
Very cost and hard to maintain
Generic benchmark
Created to be generally useful for a large group of potential customers with relatively standard applications across an application domain
Also known as domain-specific benchmark
Hybrid benchmark
generic benchmark + custom benchmark

5. Popular Benchmarks Wisconsin benchmark (1983)
TP1/DebitCredit benchmark (1985)
Transaction Processing Council formed (Aug. 1988)
TPC-A (Nov. 1989)
TPC-B (Aug. 1990)
AS3AP (1991)
SUN benchmark (001, object operations version benchmark) (1992)
TPC-C (Jul. 1992)
007 Benchmark (1993)
Sequoia 2000 Storage Benchmark (1993)
TPC-D (Apr. 1995)
BUCKY (1997)
TPC-W (1998)
BORD (2000)

6. How to Normalize Results
Difficult to compare the performance of the wide variety of database software/hardware platforms (from IBM mainframe to PC)
Current practice
Requires domain-specific qualifications that must be met
Example: ACID properties of transaction for OLTP benchmarks
Reports two specific measures
a measure of peak performance, say transactions per second (tps)
a measure of price/performance, say dollars per tps

7. Scalar measures vs. Vector measures
Vector measures allow users to distinguish different performance effects in terms of functional origins, namely separability
Think about pros and cons !

8. Key criteria for domain-specific benchmarks
Relevance: performs typical operations within the problem domain
Portability: easy to implement on many different systems and architectures
Scalability: applicable small and large computer systems
Simplicity: easily understandable otherwise it would lack credibility
Vendor neutrality: must not be biased in favor of a particular implementation

9. Brief history on OLTP benchmarks
DebitCredit benchmark (1985)
Anon. et al., “A Measure of Transaction Processing Power”, Datamation
TP1 benchmark
an implementation by vendors that typically departs from the DebitCredit specifications one way or another
Aug. 1988: Transaction Processing Council formed
Originally eight vendors, currently 35 vendors joined
Nov. 1989: TPC-A, OLTP with LAN or WAN
Aug. 1990: TPC-B, OLTP with no network
Jul. 1992: TPC-C, On-Line Business Transaction Processing
Jun. 1995: TPC-A/B are frozen
Feb. 2001: TPC-C version 5.0 release

10. Decision support system (DSS) benchmarks
relatively small number of users
mostly read-only queries, seldom update operation
Queries involve a great many data and indexes, thus use significantly more resources than OLTP transactions
data warehouse application
Wisconsin
AS3AP
Set Query
TPC-D benchmark – obsolete as of June
Separated into TPC-R benchmark and TPC-H benchmark

11. Engineering database benchmarks
Sun benchmark (001 benchmark)
W. Rubenstein, M. Kubicar and R. Cattle (ACM SIGMOD 1987)
R. Cattle, J. Skeen (ACM TODS 1992)
HyperModel Benchmark
T. Anderson, et al. (EDBT 1990)
007 Benchmark
M. Carley, D. DeWitt, J. Naughton (ACM SIGMOD 1993)
A Complex Object Benchmark (ACOB)
D. DeWitt, P. Futtersack, D. Maier and F. Velez
VLDB conf. 1990

12. How benchmarks are used: Good stories
Benchmark defines the fields
Accelerates progress
Designer can compare approaches
Users can compare approaches
Give performance agenda
Measures release-to-release progress
Set goals
Something managers can understand
Benchmark definition forms a community/vocabulary
Cross fertilization of people and ideas
Sharing of knowledge

13. Research issues What are good criteria for domain-specific benchmarks
Currently, relevance, portability, scalability, simplicity
What about scalar versus vector measures
How to incorporate actual costs of system ownership
Multi-user benchmark
How to achieve saparability of performance effects
How to develop an analytic framework
New applications or models
BLOBs, GIS, IR, Object-relational data model, etc.

14. Comments on benchmarks (1)
Benchmarking is hard
Tools are a big help, but each experiment takes a day
Always do an atomic model first, check reality against the model
Keep careful notes
Change only one thing at a time (controlled experiments)
Use all the tools you can have: Good luck!

15. Comments on benchmarks (2)
Watch out (unauthorized) vendor’s claims
Watch out benchmarketing
Benchmark result is certainly one thing, but not everything
Understanding what results say EXACTLY is very important
Must be aware of underlying platforms, networks and hardware, even though we are talking about DBMS (or OLTP) performance
Remember, vendor’s technologies and benchmark specifications continue to evolve over time

16. References (1)
T. L. Anderson, A. J. Berre, M. Mallision, H. Poster, and B. Schneider, The HyperModel Benchmark, Proc. of the Int. Conference on Extending Database Technology, 1990.
Anon, et al., A Measure of Transaction Processing Power, Datamation, 1985.
D. Bitton, D. J. DeWitt, and C. Turbyfill, Benchmarking Database Systems: A Systematic Approach, Proc. of the Ninth Int. Conference on Very Large Data Bases, 1983.
D. Bitton and C. Turbyfill, A Retrospective on The Wisconsin Benchmark, In: Readings in Database Systems, M. Stonebraker ed., Morgan Kaufman, 1988.
M. J. Carey, D. J. DeWitt, and J. F. Naughton, The OO7 Benchmark, Proc. of the ACM SIGMOD Conference, 1993.

17. References (2)
M. J. Carey, D. J. DeWitt, J. F. Naughton, M. Asgarian, P. Brown, J. E. gehrke, and D. N. Shah, The BUCKY Object-Relational Benchmark, Proc. of the ACM SIGMOD Conference, 1997.
M. J. Carey, D. J. DeWitt, C. Kant, and J. F. Naughton, A Status Report on the OO7 OODBMS Benchmarking Effort, Proc. of the ACM OOPSLA Conference, 1994.
R. G. G. Cattell, An Engineering Database Benchmark, In: The Benchmark Handbook for Database and Transaction Processing Systems 2nd ed., J. Gray ed., Morgan Kaufmann, 1993.
R. G. G. Cattell and J. Skeen, Object Operations benchmark, ACM Transactions on Database Systems, 1992.
D. J. DeWitt, The Wisconsin Benchmark: Past, Present, and Future, In: The Benchmark Handbook for Database and Transaction Processing Systems 2nd ed., J. Gray ed., Morgan Kaufmann, 1993.

18. References (3)
J. M. Hellerstein and M. Stonebraker, Predicate Migration: Optimizing Queries with Expensive Predicates, Proc. of the ACM SIGMOD Conference, 1993.
W. Kim, Completeness Criteria for Object-Relational Database Systems, UniSQL Inc, 1996.
S. H. Lee, S. J. Kim, and W. Kim, The BORD Benchmark for Object-Relational Databases, Springer-Verlag Lecture Notes in Computer Science, 2000.
P. O’Neil, The Set Query Benchmark, In: The Benchmark Handbook for Database Transaction Processing Systems 2nd ed., J. Gray ed., Morgan Kaufmann, 1993.
P. O’Neil, Database Performance Measurement, In: The Computer Science and Engineering Handbook, A Tucker ed., CRC Press, 1997.

19. References (4)
W. B. Rubenstein, M. S. Kubicar, and R. G. G. Cattell, Benchmarking Simple Database Operations, Proc. of the ACM SIGMOD Conference, 1987.
H.Schreiber, Justitia – A Generic Benchmark for the OODBMS Selection, Proc. of the Fourth International Conference of Data and Knowledge Systems for Manufacturing and Engineering, 1994.
K. Shanley, History and Overview of the TPC, Tranasaction Processing Performance Council,
M. Stonebraker, J. Frew, K. Gardels, and J. Meredith, The SEQUOIA 2000 Storage Benchmark, Proc. of the ACM SIGMOD Conference, 1993.
M. Stonebraker, Object-Relational DBMSs, Morgan Kaufmann, 1996.

20. References (5)
Transaction Processing Performance Council, TPC Benchmark C standard specification version 5.0,
Transaction Processing Performance Council, TPC Benchmark H standard specification version 1.3.0,
Transaction Processing Performance Council, TPC Benchmark R standard specification version 1.2.0,
Transaction Processing Performance Council, TPC Benchmark W version 1.4,
C. Turbyfill, C. Orji, and D. Bitton, AS3AP: An ANSI SQL Standard Scaleable and Portable Benchmark for Relational Database Systems, In: The Benchmark Handbook second edition, J. Gray ed., Morgan Kaufmann, 1993.