1. TPC Benchmarks: TPC-A and TPC-B
May 2002
Prof. Sang Ho Lee
Soongsil University

2. Who/What is the TPC
Founded 1988 by 8 vendors (lead by Omri Serlin), now over 40 vendors/users
Mission: to define transaction processing and database benchmarks and to disseminate objective and verifiable TPC performance data to the industry
De facto industry standards body for OLTP performance
Administrated by
Shenleey Public Relations
777 N. First St., Suite 600
San Jose, CA
Tel:
Most TPC specs, information, results are on the web page:

3. TPC Current Benchmarks
Benchmark C
The current OLTP benchmark
Two metrics: Transactions-per-minute-C (tpmC) and price-per-tpm-C ($/tpmC)
Benchmark H
Ad-hoc benchmark
Two metrics : Query-per-hour (QphH) and price-per-QphH ($/QphH)
Benchmark R
Business reporting benchmark
Two metrics : Query-per-hour (QphR) and price-per-QphR ($/QphR)
Benchmark W
Web commerce benchmark
Two metrics : web-interactions-per-second (WIPS) and price-per-WIPS ($/WIPS)

4. TPC Obsolete Benchmarks
Benchmark A
Based on Debit/Credit benchmark
Jun obsolete
Benchmark B
Database half of Debit/Credit benchmark
No network environment
Benchmark D
Complex decision support
Multitable join, sorting, aggregation, ...
Two metrics: Power metric Throughput
Apr obsolete

5. Aborted TPC benchmark efforts
TPC Server benchmark (TPC-S)
To create a server version of TPC-C
Remove TPC-C’s front-end and remote terminal emulation (RTE) requirement
Dec resignation
TPC-E, the Enterprise benchmark
TPC-C is significantly more complex and robust than TPC-A, but still is not complex enough to stress very large, enterprise-class systems
1996 resignation
TPC Client/Server
Adding 3 transactions to TPC-C benchmark
WWW changes the computing paradigm, so go to TPC-W
mid 1995 resignation

6. TPC-A and TPC-B History
1985: DebitCredit benchmark
Anon. et al,. A measure of transaction processing power, Datamation
TP1 benchmark
an implementation by vendors that typically departs from the DebitCredit specifications one way or another
Aug. 1988: Transaction Processing Council formed
Nov. 1989: TPC-A, OLTP with LAN or WAN
Aug. 1990: TPC-B, OLTP with no network
July 1992: TPC-C, On-Line Business Transaction Processing
Jan. 1995: no new result on TPC-A/B (i.e. move to TPC-C)

7. TPC-A Published in November, 1989
Council’s version of the Debit/Credit test
On-line transaction processing (OLTP) benchmark
Use context of a bank application
Metrics
throughput: tpsA-Local, tpsA-Wide
price-per-performance: dollars/tpsA-Local, dollars/tpsA-Wide
About 300 TPC-A benchmark results were published
Highest result was 3692 tpsA with a cost of $4,873 per tpsA

8. Logical database design
BRANCH
HISTORY
TELLER
ACCOUNT
1:M

9. TPC-A transaction profile
Read 100 bytes including Aid, Tid, Bid, Delta from terminal Begin transaction Update Account where Account_ID = Aid: Read Account_Balance from Account Set Account_Balance = Account_Balance + Delta
Write Account_Balance to Account
Write to History: Aid, Tid, Bid, Delta, Time_Stamp
Update Teller where Teller_ID = Tid:
Set Teller_Balance = Teller_Balance + Delta
Write Teller_Balance to Teller
Update Branch where Branch_ID = Bid:
Set Branch_Balance = Branch_Balance + Delta
Write Branch_Balance to Branch
Commit transaction
Write 200 bytes (Aid, Tid, Bid, Delta, Account_Balance) to terminal

10. Test configuration of TPC-A
System under Test (SUT)
one or more processing units (e.g., hosts, front-ends, workstations, etc.)
hardware and software components of all networks
data storage media, host system(s) supporting the database
Driver System
Remote Terminal Emulator (RTE) functionality
Generates and sends 100 byte transactional message to the SUT;
Receives 200 byte responses;
Records message response times;
Performs conversion and/or multiplexing into the communications protocol
Statistical accounting
Driver/SUT communication interface

11. SUT, Driver, and Communication of TPC-A

12. Features of TPC-A Strong ACID with tests specified
Response time constraints
90% of all transactions have a response time of less than 2 seconds
Transaction arrival distribution: random
History file: horizontal partitioning permitted
Price
all h/w and s/w over 5 year maintenance cost except physical communications media
Detailed report required
The test sponsor submits a full disclosure report (FDR)
Response time is measured at driver

13. Scaling rules
For each nominal tps configured, the test must use a minimum of
Account records/rows: 100,000
Teller records/rows: 10
Branch records/rows: 1
History records/rows: 2,592,000
(90 eight-hour days = 90 * 8 * 60 * 60)
Terminals: 10

14. Pacing of transactions by emulated terminals
Each emulated terminal, after sending a request to update the database to the SUT, must wait for a given “Think Time” after receiving that reply, before sending the next request

15. TPC-B Officially approved in August, 1990
Council’s version of the TP1 benchmark
not OLTP benchmark
No terminals with think time
Batch transaction generator
Metrics: tpsB, dollars/tpsB
About 130 TPC-B results were published
Highest: 2,025 tpsB with $254 per tpsB

16. TPC-A vs. TPC-B Similar with TPC-A Difference from TPC-A
same transaction profile, ACID requirements, and costing formula
Difference from TPC-A
Use of batch transaction generator
No terminal emulation with think time
No network configuration
Concept of a user does not exist
Response time is replaced by residence time (i.e. how long the transaction resides within the database server)