1. HPTS 99 Are TP Benchmarks Still Relevant to the TP Community?
Amen Anon et. al.
HPTS 99
Are TP Benchmarks Still Relevant to the TP Community?
10/14/2018
Bill Laing

2. Summary Benchmarks drove much TP technology
TPC benchmarks are not relevant to the rapid growth and rapid change of the Web environment
Benchmarks are not challenging to the TP technical community
10/14/2018
Bill Laing

3. TPC Galvanized the TP Community
Historically many of the base TP technology issues were developed to address high TP rates and win benchmarks
Locking strategies
Group commit
Log striping
Undo/redo strategies
Etc. etc.
See previous HPTS papers
Along time ago 1000 (debit/credit) TPS was a dream and a motivator
Now it is a common place vendor capability
So what now?
Have we lost our way?
10/14/2018
Bill Laing

4. TPC-C TPC-C is still the “fairest” benchmark for competition but it is
Losing relevance
Unrealistic configurations (RAID0 only, No failover)
Mostly a database focus
Measures price/performance and absolute performance but has no measure of:
Application Development costs
Deployment costs
Operations costs
Application Maintenance
Availability
Cost and ease of scalability
Yes there are H (1 result), R (1 result), and W is coming but
Are they relevant to TP technologists?
10/14/2018
Bill Laing

5. TPC-C results Top single system Scale out
#1 64 CPUs (Sun/Oracle)
#4 32 CPUs (HP/Sybase)
#5 64 CPUs (Sqnt/Oracle/Dynix)
#6 32 CPUs (HP/Oracle/HP-UX)
#7 24 CPUs (Sun/Sybase)
#8 16 CPUs (HP/Sybase/HP-UX)
Scale out
# x 12 CPUs (IBM/Oracle/AIX)
# x 12 CPUs (Alpha/Oracle/Unix)
# x 22 CPUs (Sun/Oracle/Solaris)
# x 8 CPUs (NT/NEC/Oracle)
# x 4 CPUs (NT/Compaq/Oracle)
# x 4 CPUs (NT/Tandem ServerNet/Oracle)
# x 8 CPUs (IBM/AIX/PPC)
10/14/2018
Bill Laing

6. The Biggest TPC-C Benchmark Sun+Oracle 115,395. 73 tpmC @ 105
The Biggest TPC-C Benchmark Sun+Oracle 115, $/tpmC == 12.2 M$ available 8/22/99
27 Sun StorageEdge A3500 with 1778 disks (15.6 TB)
Sun E Solaris+ Oracle8i
64x400 Mhz cpus + 64 GB RAM
(8,430k$ hardware + 1,890 k$ software)
Play on the heaviest,largest
Tuxedo® on 32 Sun Ultra Mhz workstations, Wyse terminals
429k$ hardware, 168k$ software
10/14/2018
Bill Laing

7. TPM-C Relevance? How much is 100 kTpmC? 100,000 users
Each submitting 2.3 transactions/minute
About 300 M transactions/day
Greater than Yahoo’s page views during Christmas last year
And remember
Over 90% of the TPC-C transactions are buy vs. browse
Average web site is less than 10% buy vs. browse
10/14/2018
Bill Laing

8. What happened to Scale Out
Most results are not scale out
Most Web sites start small and want to grow dramatically
So far scale out hasn’t won
Is it what customers want?
10/14/2018
Bill Laing

9. New Benchmark Food for discussion during the next 2 days Benchmark
Should focus on driving TP technology
Should show incremental growth
Closest to linear growth?
10/14/2018
Bill Laing

10. Summary Benchmarks drove much TP technology
TPC benchmarks are not relevant to the rapid growth and rapid change of the Web environment
Benchmarks are not challenging to the TP technical community
10/14/2018
Bill Laing