HPTS 99 Are TP Benchmarks Still Relevant to the TP Community? Amen Anon et. al. HPTS 99 Are TP Benchmarks Still Relevant to the TP Community? 10/14/2018 Bill Laing

Summary Benchmarks drove much TP technology TPC benchmarks are not relevant to the rapid growth and rapid change of the Web environment Benchmarks are not challenging to the TP technical community 10/14/2018 Bill Laing

TPC Galvanized the TP Community Historically many of the base TP technology issues were developed to address high TP rates and win benchmarks Locking strategies Group commit Log striping Undo/redo strategies Etc. etc. See previous HPTS papers Along time ago 1000 (debit/credit) TPS was a dream and a motivator Now it is a common place vendor capability So what now? Have we lost our way? 10/14/2018 Bill Laing

TPC-C TPC-C is still the “fairest” benchmark for competition but it is Losing relevance Unrealistic configurations (RAID0 only, No failover) Mostly a database focus Measures price/performance and absolute performance but has no measure of: Application Development costs Deployment costs Operations costs Application Maintenance Availability Cost and ease of scalability Yes there are H (1 result), R (1 result), and W is coming but Are they relevant to TP technologists? 10/14/2018 Bill Laing

TPC-C results Top single system Scale out #1 64 CPUs 115396 (Sun/Oracle) #4 32 CPUs 102023 (HP/Sybase) #5 64 CPUs 93901 (Sqnt/Oracle/Dynix) #6 32 CPUs 92833 (HP/Oracle/HP-UX) #7 24 CPUs 53050 (Sun/Sybase) #8 16 CPUs 52117 (HP/Sybase/HP-UX) Scale out # 2 5 x 12 CPUs 110434 (IBM/Oracle/AIX) # 3 8 x 12 CPUs 102542 (Alpha/Oracle/Unix) # 9 2 x 22 CPUs 51871 (Sun/Oracle/Solaris) #10 4 x 8 CPUs 50208 (NT/NEC/Oracle) #21 4 x 4 CPUs 33936 (NT/Compaq/Oracle) #24 6 x 4 CPUs 27383 (NT/Tandem ServerNet/Oracle) #79 4 x 8 CPUs 14286 (IBM/AIX/PPC) 10/14/2018 Bill Laing

The Biggest TPC-C Benchmark Sun+Oracle 115,395. 73 tpmC @ 105 The Biggest TPC-C Benchmark Sun+Oracle 115,395.73 tpmC @ 105.63 $/tpmC == 12.2 M$ available 8/22/99 27 Sun StorageEdge A3500 with 1778 disks (15.6 TB) Sun E10000 + Solaris+ Oracle8i 64x400 Mhz cpus + 64 GB RAM (8,430k$ hardware + 1,890 k$ software) Play on the heaviest,largest Tuxedo® on 32 Sun Ultra 10 333Mhz workstations, Wyse terminals 429k$ hardware, 168k$ software 10/14/2018 Bill Laing

TPM-C Relevance? How much is 100 kTpmC? 100,000 users Each submitting 2.3 transactions/minute About 300 M transactions/day Greater than Yahoo’s page views during Christmas last year And remember Over 90% of the TPC-C transactions are buy vs. browse Average web site is less than 10% buy vs. browse 10/14/2018 Bill Laing

What happened to Scale Out Most results are not scale out Most Web sites start small and want to grow dramatically So far scale out hasn’t won Is it what customers want? 10/14/2018 Bill Laing

New Benchmark Food for discussion during the next 2 days Benchmark Should focus on driving TP technology Should show incremental growth Closest to linear growth? 10/14/2018 Bill Laing

Summary Benchmarks drove much TP technology TPC benchmarks are not relevant to the rapid growth and rapid change of the Web environment Benchmarks are not challenging to the TP technical community 10/14/2018 Bill Laing