1 Measuring and Discussing Computer System Performance or “My computer is faster than your computer” Reading: 2.4, Peer Instruction Lecture Materials for Computer Architecture by Dr. Leo Porter is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. Dr. Leo Porter Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

2 Performance Metrics 1. Network Bandwidth (data/sec) 2. Network Latency (ms) 3. Frame Rate (frames/sec) 4. Throughput (ops/sec) Domains SelectionOnline WoW Crysis (FPS) Torrent Download Google Server Farm A4312 B4132 C2134 D2314 ENone of the above Match (Best) Performance Metric to Domain Jack’s car buying analogy, we care about many…. Prep with explanation of metrics and domains.

3 Measures of “Performance” Execution Time Frame Rate Throughput (operations/time) Responsiveness Performance / Cost Performance / Power Performance / Power^2

4 Recall our O(n) discussion Much of computer science focuses on execution time – and much of our class will as well. Ultimately, much of what we do we want fast (response time). So is time a reasonable metric? People sit in front of computers / iphones / etc. User time/ clock time/ CPU time CPU Time for this class – for now.

5 All Together Now CPU Execution Time Instruction Count CPIClock Cycle Time = XX

6 All Together Now CPU Execution Time Instruction Count CPIClock Cycle Time = XX instructions cycles/instruction seconds/cycle seconds

7 IC = 1 billion, 500 MHz processor, execution time of 3 seconds. What is the CPI for this program? CPU Execution Time Instruction Count CPIClock Cycle Time = XX Selectio n CPI A3 B15 C1.5 D15*10^9 ENone of the above 3 sec = 1*10^9 inst*CPI * 1sec/(5*10^8)cycles 1.5*10^9 cycles = 10^9insts*CPI 1.5 = CPI

8 Who Affects Performance? There are a number of people involved in processor / programming design Each of these elements of the performance equation can be impacted by different designer(s) Next slides will be about who can impact what. We’ll do speed voting (1 min ind, 1 min group) then discuss each slide. CPU Execution Time Instruction Count CPIClock Cycle Time = XX Individual only IC CT

9 Who Affects Performance? What can a programmer influence? CPU Execution Time Instruction Count CPIClock Cycle Time = XX 1 min ind / 1 min group SelectionImpacts AIC BIC, CPI CIC, CPI, and CT DIC and CT ENone of the above IC CT

10 Who Affects Performance? What can a compiler influence? CPU Execution Time Instruction Count CPIClock Cycle Time = XX SelectionImpacts AIC BIC, CPI CIC, CPI, and CT DCPI and CT ENone of the above IC CT 1 min ind / 1 min group

11 Who Affects Performance? What can an instruction set architect influence? CPU Execution Time Instruction Count CPIClock Cycle Time = XX SelectionImpacts AIC BIC, CPI CIC, CPI, and CT DCPI and CT ENone of the above IC CT 1 min ind / 1 min group

12 Who Affects Performance? What can an hardware designer influence? CPU Execution Time Instruction Count CPIClock Cycle Time = XX SelectionImpacts AIC BIC, CPI CIC, CPI, and CT DCPI and CT ENone of the above IC CT 1 min ind / 1 min group

13 Performance Variation CPU Execution Time Instruction Count CPIClock Cycle Time = XX Same Diff Same Diff Same Diff Sele ction Row(s) Correct A1 B1 and 3 C2 D2 and 3 ENone of the above ROW DIFF

14 Other Performance Metrics Time is useful – but how might we try to measure the “performance” of a machine -MIPS -MFLOPS

15 MIPS MIPS = Millions of Instructions Per Second = Instruction Count Execution Time * 10 6 = Clock rate CPI * 10 6 program-independent deceptive Just crank up clock rate and have it execute tons of noops. But we need to sell processors, what do we market?

16 Trying to market the “performance” of a processor. “Speed Demons” vs. “Brainiacs” If we can’t use something like raw processor speed (CT), if we want CPI - we need to look at performance on benchmarks Intel vs. Alpha, Intel wins… but ends up remarketing.

17 Benchmarks - Which Programs? peak throughput measures (simple programs)?

18 Benchmarks - Which Programs? peak throughput measures (simple programs)? synthetic benchmarks (whetstone, dhrystone,...)?

19 Benchmarks - Which Programs? peak throughput measures (simple programs)? synthetic benchmarks (whetstone, dhrystone,...)? Real applications

20 Benchmarks - Which Programs? peak throughput measures (simple programs)? synthetic benchmarks (whetstone, dhrystone,...)? Real applications SPEC (best of both worlds, but with problems of their own) -System Performance Evaluation Cooperative -Provides a common set of real applications along with strict guidelines for how to run them. -provides a relatively unbiased means to compare machines.

21 Danger in Benchmark-Specific Performance Measures measures compiler as much as architecture!

22 SPEC Performance on Pentium III and Pentium 4 Focus on clock rate relative to change in INT vs. FP performance. SSE2 on P3 was a FP stack, P4 had independent registers

23 Speedup Often want to compare performance of one machine against another Performance = 1 Execution Time Speedup (A over B) = Performance A Performance B Speedup (A over B) = ET B ET A

24 Amdahl’s Law Execution time after improvement = Execution Time Affected Amount of Improvement Execution Time Unaffected+

25 Amdahl’s Law and Parallelism Our program is 90% parallelizable (segment of code executable in parallel on multiple cores) and runs in 100 seconds with a single core. What is the execution time if you use 4 cores (assume no overhead for parallelization)? Selecti on Execution Time A25 seconds B32.5 seconds C50 seconds D92.5 seconds ENone of the above ISOMORPHIC Execution time after improvement = Execution Time Affected Amount of Improvement Execution Time Unaffected+

26 Amdahl’s Law and Parallelism Our program is 90% parallelizable (segment of code executable in parallel on multiple cores) and runs in 100 seconds with a single core. What is the execution time if you use 2 cores (assume no overhead for parallelization)? Selecti on Execution Time A55 seconds B50 seconds C100 seconds D95 seconds ENone of the above ISOMORPHIC Execution time after improvement = Execution Time Affected Amount of Improvement Execution Time Unaffected+

27 Amdahl’s Law So what does Amdalh’s Law mean at a high level? Selectio n “BEST” message from Amdahl’s Law AParallel programming is critical for improving performance BImproving serial code execution is ultimately the most important goal. CPerformance is strictly tied to the ability to determine which percentage of code is parallelizable. DThe impact of a performance improvement is limited by the percent of execution time affected by the improvement ENone of the above

28 Point out Phenom II x4 and x2 get same performance – but one has 4 cores the other 2. What does this tell us? (Note – with some slower speeds of the Phenom this isn’t the case – 4 cores help.)

29 Speedup vs. Sizeup Speedup runs into problems for parallelization because of diminishing returns and dominance of serial execution. What if time were a constant? Human perception (graphics) Earthquake prediction Weather prediction

30 Key Points Be careful how you specify “performance” Execution time = IC * CPI * CT Use real applications, if possible Use standards, if possible Make the common case fast