1. Performance & Benchmarking

2. What Matters? Which airplane has best performance:

3. CPU Time Time to run a program:

4. GHz Myth GHz measures cycles / second Faster is better, but only if architecture otherwise constant

5. MIPS MIPS = Millions of Instructions Per Second – Inverse of these two rates: – Can't compare different architectures Especially RISC/CISC

6. FLOPS FLOPS = Floating Point Operations Per Second – Same issue as MIPS… What exactly is a FLOP? How many does a program take?

7. Synthetic Benchmarks Standard "program" run on different machines Early programs: – Whetstone – floating point ops – Dhrystone – integer ops Issues: – Small size… don't test memory – Compiler optimizations targeted at benchmarks

8. Benchmark Suites Collection of large/real world programs tested

9. What do you care about? Some parts of suite focus on – Integer – Floating Point – Memory Generally don't test IO

10. SPEC Standard Performance Evaluation Corporation https://www.spec.org/cgi- bin/osgresults?conf=cpu2006 https://www.spec.org/cgi- bin/osgresults?conf=cpu2006 Industry consortium, licenses test suite

11. Geekbench Consumer focused, cross platform http://browser.primatelabs.com/geekbench3/

12. TPC Transaction Processing Council benchmarks – Business transaction oriented simulation High volume, short duration Communication Disk IO

13. Application Testing If one application is make or break, benchmark with it:

14. Benchmark Mathematics Comparisons: System A is x% faster than B

15. Benchmark Mathematics Car A travels 10 miles in 3 minutes, Car B 10 miles in 4 minutes. How much faster is A?

16. Average Performance Average performance across multiple programs may be important

17. Weighted Average If we know jobs have distinctly different usages, need weighted mean:

18. Arithmetic Mean Issues Skewed by outliers:

19. Arithmetic Mean Issues Normalized task performance easier to talk about – "C takes 10 times longer than A to run program V"

20. Arithmetic Mean Issues Speedup calculations depend on normalization target:

21. Average Performance Geometric mean: – Multiply n value, take the nth root

22. Average Performance Geometric mean: – Multiply n value, take the nth root

23. Geometric Mean Geometric Mean: – Less dependent on outliers

24. Normalized Which machine we normalize against doesn't matter…

25. Harmonic Mean Machine does equal reads & updates on DB – Can process 10,000 reads/second – Can process 2,000 writes/second What is the average query rate?

26. Harmonic Mean Machine does equal reads & updates on DB – Can process 10,000 reads/second – Can process 2,000 writes/second What is the average query rate? – NOT (10,000 + 2,000) / 2 = 6,000

27. A Car Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

28. A Car Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

29. A Car Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

30. Harmonic Mean Harmonic mean: – Averages rates

31. Harmonic Mean Car goes 60mph for 30 miles, 30 mph for another 30 miles, what is average speed?

32. Harmonic Means Machine does equal reads & updates on DB – Can process 10,000 reads/second – Can process 2,000 writes/second

33. Types Each mean says different things: Others: – Weighted geometric SPEC uses for categories – Weighted harmonic