Download presentation
Presentation is loading. Please wait.
Published byDebra Sophie Thomas Modified over 9 years ago
1
Manijeh Keshtgary Chapter 13
2
How to report the performance as a single number? Is specifying the mean the correct way? How to report the variability of measured quantities? What are the alternatives to variance and when are they appropriate? How to interpret the variability? How much confidence can you put on data with a large variability? How many measurements are required to get a desired level of statistical confidence? How to summarize the results of several different workloads on a single computer system? How to compare two or more computer systems using several different workloads? Is comparing the mean sufficient? 2
3
Sample Versus Population Confidence Interval for The Mean Approximate Visual Test One Sided Confidence Intervals Sample Size for Determining Mean 3
4
Old French word `essample' `sample' and `example' One example theory One sample Definite statement 4
5
5 Generate several million random numbers with mean and standard deviation Draw a sample of n observations: {x 1, x 2, …, x n } ◦ Sample mean (x) population mean ( ) Parameters: population characteristics ◦ Unknown, Use Greek letters ( Statistics: Sample estimates ◦ Random, Use English letters (x, s)
6
it is not possible to get a perfect estimate of the population mean from any finite number of finite size samples. The best we can do is to get probabilistic bounds. Thus, we may be able to get two bounds, for instance, c1 and c2,
7
k samples k Sample means ◦ Can't get a single estimate of ◦ Use bounds c 1 and c 2 : Probability{c 1 c 2 } = 1- ( is very small) Confidence interval: [(c 1, c 2 )] Significance level: Confidence level: 100(1- ) Confidence coefficient: 1- 7 c1c1 c2c2
8
Use 5-percentile and 95-percentile of the sample means to get 90% Confidence interval Need many samples (n > 30) Central limit theorem: Sample mean of independent and identically distributed observations: Where = population mean, = population standard deviation Standard Error: Standard deviation of the sample mean 100(1- )% confidence interval for : z 1- /2 = (1- /2)-quantile of N(0,1) 8 -z 1- /2 z 1- /2
9
For example, for a two–sided confidence interval at 95%, α= 0.05 and p = 1 – α/2 = 0.975. The entry in the row labeled 0.97 and column labeled 0.005 gives z p = 1.960.
10
x = 3.90, s = 0.95 and n = 32 A 90% confidence interval for the mean = We can state with 90% confidence that the population mean is between 3.62 and 4.17. The chance of error in this statement is 10%. 10
11
If we take 100 samples and construct confidence interval for each sample, the interval would include the population mean in 90 cases. 11 c1c1 c2c2 Total yes > 100(1- )
12
The preceding confidence interval applies only for large samples, that is, for samples of size greater than 30. For smaller samples, confidence intervals can be constructed only if the observations come from a normally distributed population 100(1- ) % confidence interval for n < 30
13
◦ Note: can be constructed only if observations come from a normally distributed population t [1- /2; n-1] = (1- /2)-quantile of a t-variate with n-1 degrees of freedom ◦ Listed in Table A.4 in the Appendix 13
14
Table A.4 lists t [p;n]. For example, the t [0.95;13] required for a two–sided 90% confidence interval of the mean of a sample of 14 observation is 1.771
15
Sample ◦ -0.04, -0.19, 0.14, -0.09, -0.14, 0.19, 0.04, and 0.09. Mean = 0, Sample standard deviation = 0.138. For 90% interval: t [0.95;7] = 1.895 Confidence interval for the mean 15
16
16
17
Difference in processor times: {1.5, 2.6, -1.8, 1.3, -0.5, 1.7, 2.4} Question: Can we say with 99% confidence that one is superior to the other? ◦ Sample size = n = 7 ◦ Mean = 7.20/7 = 1.03 ◦ Sample variance = (22.84 - 7.20*7.20/7)/6 = 2.57 ◦ Sample standard deviation = = 1.60 t [0.995; 6] = 3.707 99% confidence interval = (-1.21, 3.27) 17
18
Opposite signs we cannot say with 99% confidence that the mean difference is significantly different from zero Answer: They are same Answer: The difference is zero 18
19
Difference in processor times ◦ {1.5, 2.6, -1.8, 1.3, -0.5, 1.7, 2.4}. Question: Is the difference 1? 99% Confidence interval = (-1.21, 3.27) ◦ The confidence interval includes 1 => Yes: The difference is 1 with 99% of confidence 19
20
Paired: one-to-one correspondence between the i th test of system A and the i th test on system B ◦ Example: Performance on i th workload ◦ Straightforward analysis: the two samples are treated as one sample of n pairs ◦ Use confidence interval of the difference Unpaired: No correspondence ◦ Example: n people on System A, n on System B Need more sophisticated method ◦ t-test procedure 20
21
Performance: {(5.4, 19.1), (16.6, 3.5), (0.6, 3.4), (1.4, 2.5), (0.6, 3.6), (7.3, 1.7)}. Is one system better? Differences: {-13.7, 13.1, -2.8, -1.1, -3.0, 5.6}. Answer: No. They are not different (the confidence interval includes zero) 21
22
1. Compute the sample means 2. Compute the sample standard deviations 22
23
3. Compute the mean difference 4. Compute the standard deviation of the mean difference 5. Compute the effective number of degrees of freedom 6. Compute the confidence interval for the mean difference 7. If the confidence interval includes zero, the difference is not significant 23
24
Times on System A: {5.36, 16.57, 0.62, 1.41, 0.64, 7.26} Times on system B: {19.12, 3.52, 3.38, 2.50, 3.60, 1.74} Question: Are the two systems significantly different? For system A: For System B: 24
25
The confidence interval includes zero the two systems are not different 25
26
26
27
Times on System A: {5.36, 16.57, 0.62, 1.41, 0.64, 7.26} Times on system B: {19.12, 3.52, 3.38, 2.50, 3.60, 1.74} t [0.95, 5] = 2.015 The 90% confidence interval for the mean of A = 5.31 (2.015) = (0.24, 10.38) The 90% confidence interval for the mean of B = 5.64 (2.015) = (0.18, 11.10) Confidence intervals overlap and the mean of one falls in the confidence interval for the other ◦ Two systems are not different at this level of confidence 27
28
Need not always be 90% or 95% or 99% Based on the loss that you would sustain if the parameter is outside the range and the gain you would have if the parameter is inside the range Low loss Low confidence level is fine ◦ E.g., lottery of 5 Million, one dollar ticket cost, with probability of winning 10 -7 (one in 10 million) ◦ 90% confidence buy 9 million tickets (and pay $9M) ◦ 0.01% confidence level is fine 50% confidence level may or may not be too low 99% confidence level may or may not be too high 28
30
Time between crashes was measured for two systems A and B. The mean and standard deviations of the time are listed in Table 13.1. To check if System A is more susceptible to failures than System B,
33
Larger sample Narrower confidence interval resulting in higher confidence Question: How many observations n to get an accuracy of § r% and a confidence level of 100(1- )%? We know that for a sample of size n, the 100(1 - α)% confidence interval of the population mean is r% accuracy implies that confidence interval should be 33
34
Sample mean of the response time = 20 seconds Sample standard deviation = 5 Question: How many repetitions are needed to get the response time accurate within 1 second at 95% confidence? Required accuracy = 1 in 20 = 5% Here, = 20, s= 5, z= 1.960, and r=5, n = A total of 97 observations are needed. 34
35
All statistics based on a sample are random and should be specified with a confidence interval If the confidence interval includes zero, the hypothesis that the population mean is zero cannot be rejected Paired observations Test the difference for zero mean Unpaired observations More sophisticated t-test 35
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.