Measuring Performance and Benchmarks Instructor: Dr. Mike Turi Department of Computer Science and Computer Engineering Pacific Lutheran University Lecture slides adapted from Part 19, EE 334 Dr. Valeriu Beiu (Washington State University) Spring 2005

Why is Measuring Performance Important? When purchasing/upgrading a computer: ▫Best Performance? ▫Least Cost? ▫Best Performance/Cost? When a computer designer is faced with design options: ▫Best Performance Improvement? ▫Least Cost? ▫Best Performance/Cost? 2

How to Define “Performance” Which has higher performance? ▫Time to deliver 1 passenger? ▫Time to deliver 400 passengers? 3 Plane DC to Paris Top Speed Passengers Throughput (p x mph) Boeing hours 610 mph ,700 BAD/Sud Concorde 3 hours 1350 mph ,200

Definitions of Performance Execution Time/Response Time ▫Time between the start and completion of a task Throughput/Bandwidth ▫Total amount of work/tasks done in a given time 4

Relative Performance Performance = 1 / Execution_Time ▫Execution_Time = seconds per program A is n times faster than B ▫A is n times as fast as B ▫PerformanceA / PerformanceB = n If A is n times faster than B, then the execution time on B is n times longer than it is on A ▫Execution_TimeB / Execution_TimeA = n 5

CPU Execution Time/CPU Time The time the CPU spends computing a task – Does not include time waiting for I/O or running other programs User CPU Time – CPU time spent in program System CPU Time – CPU time spent in OS performing tasks on behalf of the program Differentiating User and System CPU Time can be difficult 6

How to Measure Time? Seconds, minutes, hours, etc. Count the number of clock cycles that elapse ▫Computers constructed using a clock that runs at a constant rate 7

The Clock Clock Cycle/Clocks/Cycles – Discrete time intervals that determine when events take place in the hardware Clock Period/Clock Cycle Time – The time for a complete clock cycle (e.g. 400 ps) Clock Rate/Clock Frequency – Inverse of Clock Period (e.g. 2.5 GHz) CPU Execution Time for a program = – CPU clock cycles for a program × Clock cycle time – CPU clock cycles for a program / Clock rate 8

Clock Cycles per Instruction (CPI) Average clock cycles per instruction CPU Clock Cycles = Instr. for a program × CPI Instruction Count ▫The number of instructions executed by a program ▫CPU Time = Instruction count × CPI × Clock cycle time ▫CPU Time = Instruction count × CPI / Clock rate 9

Performance is impacted by… Performance is: ▫Instruction count ▫CPI ▫Clock rate Algorithm Programming language Compiler ISA 10

Performance using MIPS MIPS = Million Instructions per Second ▫MIPS = Instruction Count / (Execution Time × 10 6 ) Problems ▫Cannot compare computers with different instruction sets ▫MIPS varies between programs on the same computer ▫MIPS can vary independently of performance  If a computer executes more instructions, but each instruction is faster 11

Amdahl’s Law The performance enhancement possible with a given improvement is limited by the amount that the improved feature is used ▫Diminishing returns Make the common case fast 12

Speedup Speedup due to enhancement “E” ▫Speedup =Exec_Time (old) Exec_Time (w/o E) = Exec_Time (new) Exec_Time (w/E) ▫Speedup = Performance (new) Performance (w/E) = Performance (old) Performance (w/o E) 13

Choosing which Programs to use to Measure Performance Ideally run typical programs with typical input before purchase or building the machine ▫Called a workload; For example: ▫Engineer uses compiler, spreadsheet ▫Author uses word processor, drawing program, compression software In some situations it’s hard to do ▫Don’t have access to machine to benchmark before purchase ▫Don’t know workload in future 14

Benchmarks Apparent speed of processor depends on code used to test it Need industry standards so that different processors can be fairly compared Companies exist that create these benchmarks: “typical” code used to evaluate systems Need to be changed every 2 or 3 years since designers could (and do!) target for these standard benchmarks Also have benchmarks for power (active and idle) 15

Standardized Benchmarks Example: Standard Performance Evaluation Corporation (SPEC) One benchmark set: SPEC CPU ▫Benchmark suites for wide range of hardware ▫Workloads from real user applications ▫12 integer benchmark tests ▫19 floating point benchmark tests ▫History of SPEC CPU:  CPU92, CPU95, CPU2000, CPU

CINT2006 (Integer Component) (programs written in C or C++) Perl Bzip (compression) GCC (C compiler) mcf (schedules public transportation) gobmk (AI plays a game of Go) hmmer (search gene sequence) sjeng (AI plays chess) libquantum (simulates a quantum computer) H.264/AVC (video compression) OMNet++ (model large Ethernet campus network) astar (path-finding algorithms) XML processing 17

CFP2006 (Floating Point Component) (programs written in Fortran or C/C++) Fluid Dynamics Quantum Chemistry Physics Molecular Dynamics General Relativity Finite Element Analysis Linear Programming Image Ray-Tracing Structural Mechanics Computational Electromagnetics Weather Speech Recognition 18

Test Your Knowledge (True/False) 1.Clock rate does not equal performance 2.A program runs in 100 sec. on a machine, mult accounts for 80 sec. of that. If we want to make the program run 6 times faster, we need to increase the speed of mult’s by AT LEAST 6. 3.Benchmarks cannot change since they are used to measure and compare computer performance among many manufacturers 19

The Megahertz Myth Apple G4 vs. Intel Pentium 4 ▫867 MHz vs. 1.7 GHz ▫Which performs better? See link below: ▫ qchttp:// qc 20