Lecture 2b: Performance Metrics

Performance Metrics Measurable characteristics of a computer system: Count of an event Duration of a time interval Size of a parameter Rate: Operations executed per second

Performance Mertrics Clock Speed Clock speed/frequency (f): the rate of clock pulses (ex: 1GHz) Cycle time (T c ): time between two clock pulses (T c = 1/f) TcTc

Performance Mertrics Instruction Execution Rate Cycles per Instruction (CPI): is an average depends on the design of micro-architecture (hardwired/microprogrammed, pipelined) Number of instructions: is the number of instructions executed at runtime Depends on instruction set architecture (ISA) compiler CPI = CPI i : number of cycles required for instruction i I i : number of executed instructions of type i

Performance Metrics CPU Performance CPU time of a program (T) = instructions x cycles x time program instruction cycle CPI (cycles per instruction) T = instruction count x CPI x 1 f

Performance Metrics CPU Performance Drawbacks: In modern computers, no program runs without some operating system running on the hardware Comparing performance between machines with different operating systems will be unfair

Performance Metrics Execution time Elapsed time / wall-clock time / response time Latency to complete a task, including disk access, memory access, I/O, operating system overhead, and everything (includes time consumed by other programs in a time-sharing system) CPU time The time CPU is computing, not including I/O time or waiting time User time / user CPU time CPU time spent in the program System time / system CPU time CPU time spent in the operating system performing tasks requested by the program

Performance Metrics Performance Comparison Relative performance Performance x = 1. Execution time X Performance Ratio = Performance X = Execution time Y Performance Y Execution time X

Performance Metrics Relative Performance If workload consists of more than one program, total execution time may be used. If there are more than one machine to be compared, one of them must be selected as a reference.

Performance Metrics Throughput Total amount of work done in a given time Measured in tasks per time unit Can be used for Operating system performance Pipeline performance Multiprocessor performance

Performance Metrics MIPS (Million instructions per second) Includes both integer and floating point performance Number of instructions in a program varies between different computers Number of instructions varies between different programs on the same computer MIPS = Instruction count = Clock rate Execution time x 10 6 CPI x 10 6

Performance Metrics MFLOPS (Million floating point operations per second) Give performance of only floating-point operations Different mixes of integer and floating-point operations may have different execution times: Integer and floating-point units work independently Instruction and data caches provide instruction and data concurrently

Performance Metrics Utilization Speciality ratio 1  general purpose Utilization = Busy time. Total time Speciality ratio = Maximum performance. Minimum performance

Performance Metrics Asymptotic and Half performance r  – asymptotic performance n 1/2 – half performance T = r  (n + n 1/2 ) r  = 1/t n 1/2 = t 0 /t Slope = r  -1 t0t0 -n 1/2 n 1/2 2t 0

Performance Metrics Speedup Express how much faster is system 2 than system 1 Calculated directly from execution time Performance x = 1 = 1 Execution time X T X Speedup 2,1 = Performance 2 = T 1 Performance 1 T 2

Performance Metrics Relative Change It expresses the performance of system 2 relative to system 1 Performance x = 1 = 1 Execution time X T X Relative change 2,1 = Performance 2 - Performance 1 = T 1 - T 2 = Speedup 2,1 - 1 Performance 1 T 2

Performance Metrics Statistical Analysis Used to compare performance Workload consists of many programs Depends on the nature of the data as well as distribution of the test results

Performance Metrics Indices of Central Tendency Used to summarize multiple measurements Mean Median Mode

Performance Metrics Mean (average) Gives equal weight to all measurements Arithmetic mean =  x i, 1 ≤ i ≤ n n MeasurementExecution time X110 X220 X315 X418 X516 Mean15.8

Performance Metrics Median 1. Order all n measurements 2. The middle value is the median. If n is even, median is the mean of the middle 2 values Using Median instead of Mean reduces the skewing effect of the outliers. MeasurementExecution time X110 X220 X315 X418 X516 X6200 Mean46.5 MeasurementExecution time X110 X315 X516 X418 X220 X6200 Median = = 17

Performance Metrics Mode Mode is the value that occurs most frequently If all values occur once, there is no mode If there are several samples that all have the same value, there would be several modes MeasurementExecution time X110 X220 X336 X420 X520 X620 Mode = 20

Mean, Median, Mode Mean Incorporates information from the entire measured values Sensitive to outliers Median and Mode Less sensitive to outliers Do not effectively use all information ex

Performance Metrics Arithmetic mean (average) May be misleading if the data are skewed or scattered Arithmetic mean =  x i, 1 ≤ i ≤ n n MAMA MBMB MCMC Prog Prog Prog Average2000

Performance Metrics Weighted average weight is the frequency of each program in daily processing Results may change with a different set of execution frequencies Weighted average = ∑ w i. x i 1 ≤ i ≤ n weightMAMA MBMB MCMC Prog160% Prog230% Prog310% Average

Performance Metrics Geometric mean Results are stated in relation to the performance of a reference machine Geometric mean = (  x i ) 1/n, 1 ≤ i ≤ n MAMA Normalized to M A M B (reference) Normalized to M B MCMC Normalized to M C Prog Prog Prog Average Results are consistent no matter which system is chosen as reference

Performance Metrics Harmonic mean Used to compare performance results that are expressed as a rate (e.g. operations per second, throughput, etc.) Slowest rates have the greatest influence on the result  It identifies areas where performance can be improved Harmonic mean = n, 1 ≤ i ≤ n ∑ 1/x i

Performance Metrics Characteristics of a good performance metric If the time values are averaged, then the resulting mean value must be directly proportional to the total time. If the rate values are averaged, then the resulting mean value must be inversely proportional to the total time.

Performance Metrics Ex n benchmark programs T i is the execution time of program i F floating-point operations in each program M i = F / T i is the execution rate of program i (MFLOP/s) Arithmetic mean Inappropriate for summarizing rates T A = T A is directly proportional to the total execution time M A = = M A is inversely proportional to the total execution time

Performance Metrics Harmonic mean Inappropriate for summarizing execution times Appropriate for summarizing rates T H = T H is not directly proportional to the total execution time M H = = M H is inversely proportional to the total execution time

Performance Metrics Ex

Performance Metrics Geometric mean Inappropriate for summarizing execution times Inappropriate for summarizing rates T G = T G is not directly proportional to the total execution time M H = = M H is not inversely proportional to the total execution time

Performance Metrics Geometric mean Produces a consistent ordering of the systems but it is the wrong ordering M1M2M3 Prog Prog28370 Prog Prog Prog Geometric mean (Normalized wrt S1) Geometric mean (Normalized wrt S2) Rank321 Total time Arithmetic mean Rank213

Performance Metrics Histogram Used to display the distribution of a set of measured values (variability) First find the minimum and maximum values. Then divide the range into b subranges, called cells.

Histogram Message size (kbytes) Network ANetwork B 0 < xi ≤ < xi ≤ < xi ≤ < xi ≤ < xi ≤ < xi ≤ < xi ≤ 3540

Performance Metrics Index of Dispersion Index of dispersion is used to compare the spread of measurements around the mean value Range is the simplest metric for an index of dispersion Range is sensitive to a few extreme values

Performance Metrics Index of Dispersion Maximum of the absolute values of the difference of each measurement from the mean It is also sensitive to extreme values

Performance Metrics Index of Dispersion Sample variance is the simplest metric for an index of dispersion Requires 2 passes through the data to calculate first x and then s 2 Requires 1 pass

Performance Metrics Index of Dispersion Standard deviation Coefficient of variance (COV): normalizes standard deviation wrt the mean