Lecture 2d: Performance Comparison

Quality of Measurement Characteristics of a measurement tool (timer) Accuracy: Absolute difference of a measured value and the corresponding standard reference value (such as the duration of a second). Precision: Reliability of the measurements made with the tool. Highly precise measurements are tightly clustered around a single value. Resolution: Smallest incremental change that can be detected. Ex: interval between clock ticks

Quality of Measurement accuracy precision mean value true value

Quality of Measurement The uncertainties in the measurements are called errors or noise Sources of errors: Accuracy, precision, resolution of the measurement tool Time required to read and store the current time value Time-sharing among multiple programs Processing of interrupts Cache misses, page faults

Quality of Measurement Types of errors: Systematic errors Are the result of some experimental mistake Usually constant across all measurements Ex: temperature may effect clock period Random errors Unpredictable, nondeterministic Effect the precision of measurement Ex: timer resolution ±T, effects measurements with equal probability

Quality of Measurement Experimental measurements follow Gaussian (normal) distribution Ex: x measured value ±E random error Two sources of errors, each having 50% probability Pg 48 Actual value of x is measured half of the time. Error 1Error 2Measured valueProbability -E x-2E1/4 -E+Ex1/4 +E-Ex1/4 +E x+2E1/4

Confidence Intervals Used to find a range of values that has a given probability of including the actual value. Case 1: number of measurements is large (n≥30) {x 1, x 2, … x n } - Samples Gaussian distribution  – mean  – standard deviation Confidence interval: [ c 1, c 2 ] Confidence level: (1-  )×100 Pr[ c 1 ≤ x ≤ c 2 ] = 1-  Pr[ x c 2 ] =  /2

Confidence Intervals Case 1: number of measurements is large (n≥30) Confidence interval: [ c 1, c 2 ] - Sample mean - Standard deviation is obtained from the precomputed table

Confidence Intervals Case 2: number of measurements is small (n<30) Sample variances s 2 can vary significantly. t distribution: - Sample mean - Standard deviation is obtained from the precomputed table

Confidence Intervals Ex: number of measurements is large (n<30) Pg 51 90% confidence interval means that there is a 90% chance that the actual mean is within that interval.

Confidence Intervals 90%c 1 = 6.5c 2 = %c 1 = 6.1c 2 = %c 1 = 5.3c 2 =10.6 Wider interval  Less precise knowledge about the mean

Confidence Intervals Determining the Number of measurements Needed

Confidence Intervals Determining the Number of measurements Needed Estimating s: 1.Make small number of measurements. 2.Estimate standard deviation s. 3.Calculate n. 4.Make n measurements.

Confidence Intervals Ex: Pg 53

Confidence Intervals Confidence Intervals for Proportions When we are interested in the number of times events occur. Bimonial distribution: If np≥10 it approximates Gaussian distribution with mean p and variance p(1-p)/n - Total events recorded - Number of times desired outcome occurs is the sample proportion

Confidence Intervals Confidence Intervals for Proportions Determining the number of measurements needed:

Confidence Intervals Ex: Pg 55

Comparing Alternatives Three different cases: Before-and-after comparison Comparison of non-corresponding (impaired) measurements Comparisons involving proportions

Comparing Alternatives Before-and-after comparison Used to determine whether some change made to a system has statistically significant impact on its performance. 1.Find a confidence interval for the mean of the differences of the paired observations 2.If this interval includes 0, then measured differences are not statistically significant.

Comparing Alternatives Before-and-after comparison Before measurements: b 1, … b n After measurements: a 1, … a n Differences: d 1 = a 1, - b 1 d 2 = a 2, - b 2 … - Arithmetic mean - Standard deviation n ≥ 30

Comparing Alternatives Before-and-after comparison Ex: pg 65

Comparing Alternatives Non-corresponding Measurements There is no direct corresponding between pairs of measurements. 1.First system: n 1 measurements, find x 1 and s 1 2.Second system: n 2 measurements, find x 2 and s 2 3.Calculate the difference of means and standard deviation of the difference of means 4.If confidence interval includes 0, then no significant difference

Comparing Alternatives Non-corresponding Measurements n 1 ≥ 30 and n 2 ≥ 30

Comparing Alternatives Non-corresponding Measurements n 1 < 30 or n 2 < 30

Comparing Alternatives Non-corresponding Measurements Ex: pg 67

Comparing Alternatives Comparing Proportions m 1 is the number of times the event occurs in system 1 out of a total of n 1 events measured. If m 1 >10 and m 2 >10 the it approximates normal distribution with means and variance and

Comparing Alternatives Comparing Proportions Confidence intervals where Standard deviation

Comparing Alternatives Comparing More than Two Alternatives Analysis of Variance (ANOVA) n - measurements k - alternatives

Comparing Alternatives Comparing More than Two Alternatives Mean of alternative j Overall mean

Comparing Alternatives Comparing More than Two Alternatives Deviation of y ij from mean Deviation of y j from y Therefore

Comparing Alternatives Comparing More than Two Alternatives Total variance observed: 1.Variance due to the actual differences among alternatives (SSA) 2.Variance due to measurement errors (SSE)

Comparing Alternatives Comparing More than Two Alternatives Sum of squares total:

Comparing Alternatives Comparing More than Two Alternatives F-test F distribution Used to test whether two variances are significantly different. If, close to 1, then no significant difference If they are greater than a critical value, then it can not be said that there is no significant difference

Comparing Alternatives Comparing More than Two Alternatives Mean square: Mean square error: If then with confidence level of

Comparing Alternatives Comparing More than Two Alternatives Contrasts Used to compare individual alternatives. c- contrast w- weight

Comparing Alternatives Comparing More than Two Alternatives Variance of c: Confidence interval:

Comparing Alternatives Comparing more than Two Alternatives

Comparing Alternatives Comparing more than Two Alternatives Ex:

Comparing Alternatives Comparing more than Two Alternatives