1. Lecture 2d1: Quality of Measurements

2. Quality of Measurement
Characteristics of a measurement tool (timer)
Accuracy: Absolute difference of a measured value and the corresponding standard reference value (such as the duration of a second).
Precision: Reliability of the measurements made with the tool. Highly precise measurements are tightly clustered around a single value.
Resolution: Smallest incremental change that can be detected. Ex: interval between clock ticks 2

3. Quality of Measurement
accuracy
precision
mean value true value

4. Quality of Measurement
The uncertainties in the measurements are called
errors or noise
Sources of errors:
Accuracy, precision, resolution of the measurement tool
Time required to read and store the current time value
Time-sharing among multiple programs
Processing of interrupts
Cache misses, page faults

5. Quality of Measurement
Types of errors:
Systematic errors
Are the result of some experimental mistake
Usually constant across all measurements
Ex: temperature may effect clock period
Random errors
Unpredictable, nondeterministic
Effect the precision of measurement
Ex: timer resolution ±T , effects measurements with equal probability

6. Quality of Measurement
Experimental measurements follow Gaussian (normal) distribution
Ex:
x measured value
±E random error
Two sources of errors, each having 50% probability
Pg 48
Actual value of x is measured half of the time.
Error 1
Error 2
Measured value
Probability -E
x-2E
1/4 +E x
x+2E

7. Confidence Intervals
Used to find a range of values that has a given probability of including the actual value.
Case 1: number of measurements is large (n≥30)
{x1, x2, … xn} - Samples
Gaussian distribution
m – mean
s – standard deviation
Confidence interval: [ c1, c2 ]
Significance level: 
Confidence coefficient: 1-
Confidence level: (1-)×100
Pr[ c1 ≤ x ≤ c2 ] = 1-
Pr[ x < c1 ] = Pr[ x > c2] = /2

8. Confidence Intervals Case 1: number of measurements is large (n≥30)
Confidence interval: [ c1, c2 ]
Sample variances s2 is a good estimate of 2.
- Sample mean
- Standard deviation
is obtained from the precomputed table

9. Confidence Intervals Ex: number of measurements is large (n ≥ 30)
90% confidence interval means that there is a 90% chance that the actual mean is within that interval.

10. Normal Distribution

11. Confidence Intervals Case 2: number of measurements is small (n<30)
Sample variances s2 can vary significantly.
t distribution:
- Sample mean
- Standard deviation
is obtained from the precomputed table

12. Confidence Intervals Ex: number of measurements is large (n < 30)
90% confidence interval means that there is a 90% chance that the actual mean is within that interval.

13. t Distribution

14. Wider interval  Less precise knowledge about the mean
Confidence Intervals
90% c1= 6.5 c2= 9.4
95% c1= 6.1 c2= 9.7
99% c1= 5.3 c2=10.6
Wider interval  Less precise knowledge about the mean

15. Determining the Number of measurements Needed
Confidence Intervals
Determining the Number of measurements Needed

16. Determining the Number of measurements Needed
Confidence Intervals
Determining the Number of measurements Needed
Estimating s:
Make small number of measurements.
Estimate standard deviation s.
Calculate n.
Make n measurements.

17. Confidence Intervals
Ex: Based on a preliminary test, mean response time is 20 seconds and standard deviation is 5. How many repetitions are needed to get the response time accurate within 1 second at 95% confidence?
e is 1 in 20 

18. Confidence Intervals Confidence Intervals for Proportions
When we are interested in the number of times events occur.
Binomial distribution:
If np≥10 it approximates Gaussian distribution with mean p and variance p(1-p)/n
- Total events recorded
- Number of times desired outcome occurs
is the sample proportion

19. Confidence Intervals for Proportions
Determining the number of measurements needed:

20. Confidence Intervals
Ex: How much time the processor spends executing the operating system compared with how much time it spends executing user programs?
Experiment: At every 10 ms, an interrupt routine increments 2 counters:
n=counts the number of interrupts occurred (incremented every time),
m=counts if operating system is executing
Results in 1 minute are m=658, n=6000
95% confidence level for this ratio is:
With 5% chance of being wrong, we can say that the processor spends % of its time executing operating system.

21. Confidence Intervals
Ex: How long must this experiment be run to know with 95% confidence that the processor spends executing operating system with an error of 0.5%?
 Requires 3.46 hours