Performance & Benchmarking

What Matters? Which airplane has best performance:

CPU Time Time to run a program:

GHz Myth GHz measures cycles / second Faster is better, but only if architecture otherwise constant

MIPS MIPS = Millions of Instructions Per Second – Inverse of these two rates: – Can't compare different architectures Especially RISC/CISC

FLOPS FLOPS = Floating Point Operations Per Second – Same issue as MIPS… What exactly is a FLOP? How many does a program take?

Synthetic Benchmarks Standard "program" run on different machines Early programs: – Whetstone – floating point ops – Dhrystone – integer ops Issues: – Small size… don't test memory – Compiler optimizations targeted at benchmarks

Benchmark Suites Collection of large/real world programs tested

What do you care about? Some parts of suite focus on – Integer – Floating Point – Memory Generally don't test IO

SPEC Standard Performance Evaluation Corporation bin/osgresults?conf=cpu bin/osgresults?conf=cpu2006 Industry consortium, licenses test suite

Geekbench Consumer focused, cross platform

TPC Transaction Processing Council benchmarks – Business transaction oriented simulation High volume, short duration Communication Disk IO

Application Testing If one application is make or break, benchmark with it:

Benchmark Mathematics Comparisons: System A is x% faster than B

Benchmark Mathematics Car A travels 10 miles in 3 minutes, Car B 10 miles in 4 minutes. How much faster is A?

Average Performance Average performance across multiple programs may be important

Weighted Average If we know jobs have distinctly different usages, need weighted mean:

Arithmetic Mean Issues Skewed by outliers:

Arithmetic Mean Issues Normalized task performance easier to talk about – "C takes 10 times longer than A to run program V"

Arithmetic Mean Issues Speedup calculations depend on normalization target:

Average Performance Geometric mean: – Multiply n value, take the nth root

Average Performance Geometric mean: – Multiply n value, take the nth root

Geometric Mean Geometric Mean: – Less dependent on outliers

Normalized Which machine we normalize against doesn't matter…

Harmonic Mean Machine does equal reads & updates on DB – Can process 10,000 reads/second – Can process 2,000 writes/second What is the average query rate?

Harmonic Mean Machine does equal reads & updates on DB – Can process 10,000 reads/second – Can process 2,000 writes/second What is the average query rate? – NOT (10, ,000) / 2 = 6,000

A Car Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

A Car Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

A Car Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

Harmonic Mean Harmonic mean: – Averages rates

Harmonic Mean Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

Harmonic Means Machine does equal reads & updates on DB – Can process 10,000 reads/second – Can process 2,000 writes/second

Types Each mean says different things: Others: – Weighted geometric SPEC uses for categories – Weighted harmonic