1. Jan. 5, 2000Systems Architecture II1 Machine Organization (CS 570) Lecture 2: Performance Evaluation and Benchmarking * Jeremy R. Johnson Wed. Oct. 4, 2000 *This lecture was derived from material in the text (Chap. 1) and material in the undergraduate text from Hennessy and Patterson. All figures from Computer Organization and Design: The Hardware/Software Approach, Second Edition, by David Patterson and John Hennessy, are copyrighted material (COPYRIGHT 1998 MORGAN KAUFMANN PUBLISHERS, INC. ALL RIGHTS RESERVED).

2. Jan. 5, 2000Systems Architecture II2 Introduction Objective: To quantify performance and relate performance to design parameters. Also to understand the role of benchmarking Execution Time (sec) = Inst/Program X Cycles/Inst (CPI) X Sec/Cycle Topics –Performance Definition –Performance parameters and equation –Benchmarking –Fallacies and Pitfalls: Amdahl’s law

3. Jan. 5, 2000Systems Architecture II3 Performance Definition Response Time Throughput Cost Example Performance X = 1/Execution Time X “X is n times faster than Y”  Performance X / Performance y = n Airplane Passengers Range (mi) Speed (mph) Throughput Boeing 7773754630610228,750 Boeing 7474704150610286,700 BAC/Sud Concorde13240001350178,200 Douglas DC-8-501468720544 79,424

4. Jan. 5, 2000Systems Architecture II4 Measuring Performance Execution time –Wallclock (elapsed time) –CPU time (system vs. user) –Limited accuracy Instruction count (simulator/hardware counters) Cycle count (simulator/hardware counters) Memory performance (simulator/hardware counters)

5. Jan. 5, 2000Systems Architecture II5 Performance Parameters and Equation Instruction count - depends on program, compiler, optimization flags, instruction set architecture Cycles Per Instruction (CPI) - depends on implementation of architecture (datapath, pipelining, parallelism, etc.) Clock rate - depends on implementation design and technology Execution Time (sec) = Inst/Program X Cycles/Inst (CPI) X Sec/Cycle

6. Jan. 5, 2000Systems Architecture II6 Benchmarking Want sample to use sample programs that approximate actual usage Beware of –small (artificial, kernel) benchmarks –synthetic benchmarks (Whetstone, Dhrystone) –Peak performance reports –use of parameters other than execution time (e.g. program size, MIPS) Make sure results are reproducible SPEC (System Performance Evalutation Corporation) –Collection of real world integer and floating point programs –http://www.specbench.org –CPU95 (SPECint95, SPECfp95) - originate in 1989 –CPU2000 just out, also graphics, web and other benchmarkds

7. Jan. 5, 2000Systems Architecture II7 SPEC95

8. Jan. 5, 2000Systems Architecture II8

9. Jan. 5, 2000Systems Architecture II9 SPEC95 Doubling clock rate does not double performance

10. Jan. 5, 2000Systems Architecture II10 SPEC89 Compiler “enhancements” and performance

11. Jan. 5, 2000Systems Architecture II11 Summarizing Results Example Perf B /Perf A = 1001/110 = 9.1 Total execution time Arithmetic mean (weighted) Geometric mean (for ratios) - used by SPEC

12. Jan. 5, 2000Systems Architecture II12 SPECint95 Geometric mean of ratios compared to SPARC 10 Model 40 Dell Computer Co Dell Precision WorkStation 410 1 13.4 13.4 Dell Computer Co Dell Precision WorkStation 410 1 15.3 15.3 Dell Computer Co Precision WorkStation 410 (450MH 1 17.6 17.6 Dell Computer Co Precision WorkStation 410 (650 M 1 31.5 31.2 Dell Computer Co Precision WorkStation 410 (700 M 1 33.7 33.4 Dell Computer Co Precision WorkStation 420 (600 M 1 30.0 29.7 Dell Computer Co Precision WorkStation 420 (733 M 1 35.8 35.3 Dell Computer Co Precision WorkStation 610 (450MH 1 18.9 18.9 Dell Computer Co Precision Workstation 410 (450MH 1 18.6 18.6 Dell Computer Co Precision Workstation 410 (500MH 1 20.4 20.4 Dell Computer Co Precision Workstation 410 (550MH 1 22.6 22.6 Dell Computer Co Precision Workstation 410 (600MH 1 24.6 24.6 Dell Computer Co Precision Workstation 610 1 16.4 16.4 Dell Computer Co Precision Workstation 610 1 16.5 16.5 Dell Computer Co Precision Workstation 610 (450MH 1 19.0 19.0 Dell Computer Co Precision Workstation 610 (500MH 1 22.1 22.1 Dell Computer Co Precision Workstation 610 (500MH 1 21.6 21.6 Dell Computer Co Precision Workstation 610 (550MH 1 24.4 24.4

13. Jan. 5, 2000Systems Architecture II13 Amdahl’s Law Execution Time After Improvement = Execution Time Unaffected + ( Execution Time Affected / Amount of Improvement ) Example: "Suppose a program runs in 100 seconds on a machine, with multiply responsible for 80 seconds of this time. How much do we have to improve the speed of multiplication if we want the program to run 4 times faster?" How about making it 5 times faster? Principle: Make common cases fast

14. Jan. 5, 2000Systems Architecture II14 Remember Performance is specific to a particular program/s –Total execution time is a consistent summary of performance For a given architecture performance increases come from: –increases in clock rate (without adverse CPI affects) –improvements in processor organization that lower CPI –compiler enhancements that lower CPI and/or instruction count Pitfall: expecting improvement in one aspect of a machine’s performance to affect the total performance

15. Jan. 5, 2000Systems Architecture II15 Principle of Locality Programs tend to reuse instructions and data that they have used recently Spatial –items whose addresses are near one another tend to be referenced close together in time Temporal –recently accessed items are likely to be accessed in the near future

16. Jan. 5, 2000Systems Architecture II16 Memory Hierarchy CPU Registers CACHECACHE Memory IO Device Size: 200 B Speed: 5 ns Size: 64 KB Speed: 10 ns Size: 32 MB Speed: 100 ns Size: 2 GB Speed: 5 ms

17. Jan. 5, 2000Systems Architecture II17 Typical Parameters Values in 1995

18. Jan. 5, 2000Systems Architecture II18 Memory Access Speed on DEC 21164 Alpha Clock Speed 500 MHz (= 2 ns clock rate) Registers (2 ns) L1 On-Chip (4 ns) L2 On-Chip (5 ns) L3 Off-Chip (30 ns) Memory (220 ns)

19. Jan. 5, 2000Systems Architecture II19 Performance Equation CPU Execution Time = (CPU clock cycles + Memory Stall Cycles) X Clock Cycle Memory Stall Cycles = =Number of misses X Miss penalty =IC X Misses per instruction X Miss penalty =IC X Memory references/instruction X Miss rate X Miss penalty Miss rate = fraction of cache accesses that result in a miss