1. Lecture 2d: Performance Comparison

2. Quality of Measurement Characteristics of a measurement tool (timer) Accuracy: Absolute difference of a measured value and the corresponding standard reference value (such as the duration of a second). Precision: Reliability of the measurements made with the tool. Highly precise measurements are tightly clustered around a single value. Resolution: Smallest incremental change that can be detected. Ex: interval between clock ticks

3. Quality of Measurement accuracy precision mean value true value

4. Quality of Measurement The uncertainties in the measurements are called errors or noise Sources of errors: Accuracy, precision, resolution of the measurement tool Time required to read and store the current time value Time-sharing among multiple programs Processing of interrupts Cache misses, page faults

5. Quality of Measurement Types of errors: Systematic errors Are the result of some experimental mistake Usually constant across all measurements Ex: temperature may effect clock period Random errors Unpredictable, nondeterministic Effect the precision of measurement Ex: timer resolution ±T, effects measurements with equal probability

6. Quality of Measurement Experimental measurements follow Gaussian (normal) distribution Ex: x measured value ±E random error Two sources of errors, each having 50% probability Pg 48 Actual value of x is measured half of the time. Error 1Error 2Measured valueProbability -E x-2E1/4 -E+Ex1/4 +E-Ex1/4 +E x+2E1/4

7. Confidence Intervals Used to find a range of values that has a given probability of including the actual value. Case 1: number of measurements is large (n≥30) {x 1, x 2, … x n } - Samples Gaussian distribution  – mean  – standard deviation Confidence interval: [ c 1, c 2 ] Confidence level: (1-  )×100 Pr[ c 1 ≤ x ≤ c 2 ] = 1-  Pr[ x c 2 ] =  /2

8. Confidence Intervals Case 1: number of measurements is large (n≥30) Confidence interval: [ c 1, c 2 ] - Sample mean - Standard deviation is obtained from the precomputed table

9. Confidence Intervals Case 2: number of measurements is small (n<30) Sample variances s 2 can vary significantly. t distribution: - Sample mean - Standard deviation is obtained from the precomputed table

10. Confidence Intervals Ex: number of measurements is large (n<30) Pg 51 90% confidence interval means that there is a 90% chance that the actual mean is within that interval.

11. Confidence Intervals 90%c 1 = 6.5c 2 = 9.4 95%c 1 = 6.1c 2 = 9.7 99%c 1 = 5.3c 2 =10.6 Wider interval  Less precise knowledge about the mean

12. Confidence Intervals Determining the Number of measurements Needed

13. Confidence Intervals Determining the Number of measurements Needed Estimating s: 1.Make small number of measurements. 2.Estimate standard deviation s. 3.Calculate n. 4.Make n measurements.

14. Confidence Intervals Ex: Pg 53

15. Confidence Intervals Confidence Intervals for Proportions When we are interested in the number of times events occur. Bimonial distribution: If np≥10 it approximates Gaussian distribution with mean p and variance p(1-p)/n - Total events recorded - Number of times desired outcome occurs is the sample proportion

16. Confidence Intervals Confidence Intervals for Proportions Determining the number of measurements needed:

17. Confidence Intervals Ex: Pg 55

18. Comparing Alternatives Three different cases: Before-and-after comparison Comparison of non-corresponding (impaired) measurements Comparisons involving proportions

19. Comparing Alternatives Before-and-after comparison Used to determine whether some change made to a system has statistically significant impact on its performance. 1.Find a confidence interval for the mean of the differences of the paired observations 2.If this interval includes 0, then measured differences are not statistically significant.

20. Comparing Alternatives Before-and-after comparison Before measurements: b 1, … b n After measurements: a 1, … a n Differences: d 1 = a 1, - b 1 d 2 = a 2, - b 2 … - Arithmetic mean - Standard deviation n ≥ 30

21. Comparing Alternatives Before-and-after comparison Ex: pg 65

22. Comparing Alternatives Non-corresponding Measurements There is no direct corresponding between pairs of measurements. 1.First system: n 1 measurements, find x 1 and s 1 2.Second system: n 2 measurements, find x 2 and s 2 3.Calculate the difference of means and standard deviation of the difference of means 4.If confidence interval includes 0, then no significant difference

23. Comparing Alternatives Non-corresponding Measurements n 1 ≥ 30 and n 2 ≥ 30

24. Comparing Alternatives Non-corresponding Measurements n 1 < 30 or n 2 < 30

25. Comparing Alternatives Non-corresponding Measurements Ex: pg 67

26. Comparing Alternatives Comparing Proportions m 1 is the number of times the event occurs in system 1 out of a total of n 1 events measured. If m 1 >10 and m 2 >10 the it approximates normal distribution with means and variance and

27. Comparing Alternatives Comparing Proportions Confidence intervals where Standard deviation

28. Comparing Alternatives Comparing More than Two Alternatives Analysis of Variance (ANOVA) n - measurements k - alternatives

29. Comparing Alternatives Comparing More than Two Alternatives Mean of alternative j Overall mean

30. Comparing Alternatives Comparing More than Two Alternatives Deviation of y ij from mean Deviation of y j from y Therefore

31. Comparing Alternatives Comparing More than Two Alternatives Total variance observed: 1.Variance due to the actual differences among alternatives (SSA) 2.Variance due to measurement errors (SSE)

32. Comparing Alternatives Comparing More than Two Alternatives Sum of squares total:

33. Comparing Alternatives Comparing More than Two Alternatives F-test F distribution Used to test whether two variances are significantly different. If, close to 1, then no significant difference If they are greater than a critical value, then it can not be said that there is no significant difference

34. Comparing Alternatives Comparing More than Two Alternatives Mean square: Mean square error: If then with confidence level of

35. Comparing Alternatives Comparing More than Two Alternatives Contrasts Used to compare individual alternatives. c- contrast w- weight

36. Comparing Alternatives Comparing More than Two Alternatives Variance of c: Confidence interval:

37. Comparing Alternatives Comparing more than Two Alternatives

38. Comparing Alternatives Comparing more than Two Alternatives Ex:

39. Comparing Alternatives Comparing more than Two Alternatives