1. June 20, 2001Systems Architecture II1 Systems Architecture II (CS 282-001) Lecture 1: Performance Evaluation and Benchmarking * Jeremy R. Johnson Wed. June 20, 2001 *This lecture was derived from material in the text (Chap. 2). All figures from Computer Organization and Design: The Hardware/Software Approach, Second Edition, by David Patterson and John Hennessy, are copyrighted material (COPYRIGHT 1998 MORGAN KAUFMANN PUBLISHERS, INC. ALL RIGHTS RESERVED).

2. June 20, 2001Systems Architecture II2 Introduction Objective: To quantify performance and relate performance to design parameters. Also to understand the role of benchmarking. Execution Time (sec) = Inst/Program X Cycles/Inst (CPI) X Sec/Cycle Topics –Performance Definition –Performance parameters and equation –Benchmarking –Fallacies and Pitfalls: Amdahl’s law

3. June 20, 2001Systems Architecture II3 Performance Definition Response Time Throughput Cost Example Performance X = 1/Execution Time X “X is n times faster than Y”  Performance X / Performance y = n Airplane Passengers Range (mi) Speed (mph) Throughput Boeing 7773754630610228,750 Boeing 7474704150610286,700 BAC/Sud Concorde13240001350178,200 Douglas DC-8-501468720544 79,424

4. June 20, 2001Systems Architecture II4 Measuring Performance Execution time –Wallclock (elapsed time) –CPU time (system vs. user) –Limited accuracy Instruction count (simulator/hardware counters) Cycle count (simulator/hardware counters) Memory performance (simulator/hardware counters)

5. June 20, 2001Systems Architecture II5 Performance Parameters and Equation Instruction count - depends on program, compiler, optimization flags, instruction set architecture Cycles Per Instruction (CPI) - depends on implementation of architecture (datapath, pipelining, parallelism, etc.) Clock rate - depends on implementation design and technology Execution Time (sec) = Inst/Program X Cycles/Inst (CPI) X Sec/Cycle

6. June 20, 2001Systems Architecture II6 Performance Equation Example Suppose we have two implementations of the same instruction set architecture Machine A has a clock cycle time of 1 ns and CPI = 2.0 Machine B has a clock cycle time of 2 ns and CPI = 1.2 Which machine is faster for this program? CPU A = I  2.0  1 ns = 2  I ns CPU B = I  1.2  2 ns = 2.4  I ns Perf A /Perf B = 2.4/2 = 1.2  A is 1.2 times faster than B

7. June 20, 2001Systems Architecture II7 Comparing Code Segments Compare efficiency of two code sequences CPU Clock Cycles 1 = 2  1 + 1  2 + 2  3 = 10 cycles CPU Clock Cycles 2 = 4  1 + 1  2 + 1  3 = 9 cycles CPI 1 = 10 cycles / 5 instructions = 2 cycles/inst CPI 2 = 9 cycles / 6 instructions = 1.5 cycles/inst Second choice is better even though there are more inst!

8. June 20, 2001Systems Architecture II8 Benchmarking Use sample programs that approximate actual usage Beware of –small (artificial, kernel) benchmarks –synthetic benchmarks (Whetstone, Dhrystone) –Peak performance reports –use of parameters other than execution time (e.g. program size, MIPS) Make sure results are reproducible SPEC (System Performance Evaluation Corporation) –Collection of real world integer and floating point programs –http://www.specbench.org –CPU95 (SPECint95, SPECfp95) - originate in 1989 –CPU2000 also graphics, web and other benchmarks

9. June 20, 2001Systems Architecture II9 SPEC95

10. June 20, 2001Systems Architecture II10

11. June 20, 2001Systems Architecture II11 SPEC95 Doubling clock rate does not double performance

12. June 20, 2001Systems Architecture II12 SPEC89 Compiler “enhancements” and performance

13. June 20, 2001Systems Architecture II13 Summarizing Results Example Perf B /Perf A = 1001/110 = 9.1 Total execution time Arithmetic mean (weighted) Geometric mean (for ratios) - used by SPEC

14. June 20, 2001Systems Architecture II14 SPECint95 Geometric mean of ratios compared to SPARC 10 Model 40 Dell Computer Co Dell Precision WorkStation 410 1 13.4 13.4 Dell Computer Co Dell Precision WorkStation 410 1 15.3 15.3 Dell Computer Co Precision WorkStation 410 (450MH 1 17.6 17.6 Dell Computer Co Precision WorkStation 410 (650 M 1 31.5 31.2 Dell Computer Co Precision WorkStation 410 (700 M 1 33.7 33.4 Dell Computer Co Precision WorkStation 420 (600 M 1 30.0 29.7 Dell Computer Co Precision WorkStation 420 (733 M 1 35.8 35.3 Dell Computer Co Precision WorkStation 610 (450MH 1 18.9 18.9 Dell Computer Co Precision Workstation 410 (450MH 1 18.6 18.6 Dell Computer Co Precision Workstation 410 (500MH 1 20.4 20.4 Dell Computer Co Precision Workstation 410 (550MH 1 22.6 22.6 Dell Computer Co Precision Workstation 410 (600MH 1 24.6 24.6 Dell Computer Co Precision Workstation 610 1 16.4 16.4 Dell Computer Co Precision Workstation 610 1 16.5 16.5 Dell Computer Co Precision Workstation 610 (450MH 1 19.0 19.0 Dell Computer Co Precision Workstation 610 (500MH 1 22.1 22.1 Dell Computer Co Precision Workstation 610 (500MH 1 21.6 21.6 Dell Computer Co Precision Workstation 610 (550MH 1 24.4 24.4

15. June 20, 2001Systems Architecture II15 Amdahl’s Law Execution Time After Improvement = Execution Time Unaffected + ( Execution Time Affected / Amount of Improvement ) Example: “ Suppose a program runs in 100 seconds on a machine, with multiply responsible for 80 seconds of this time. How much do we have to improve the speed of multiplication if we want the program to run 4 times faster?" How about making it 5 times faster? Principle: Make common cases fast

16. June 20, 2001Systems Architecture II16 Remember Performance is specific to a particular program/s –Total execution time is a consistent summary of performance For a given architecture performance increases come from: –increases in clock rate (without adverse CPI affects) –improvements in processor organization that lower CPI –compiler enhancements that lower CPI and/or instruction count Pitfall: expecting improvement in one aspect of a machine’s performance to affect the total performance