Lecture 2d1: Quality of Measurements

Quality of Measurement Characteristics of a measurement tool (timer) Accuracy: Absolute difference of a measured value and the corresponding standard reference value (such as the duration of a second). Precision: Reliability of the measurements made with the tool. Highly precise measurements are tightly clustered around a single value. Resolution: Smallest incremental change that can be detected. Ex: interval between clock ticks 2

Quality of Measurement accuracy precision mean value true value

Quality of Measurement The uncertainties in the measurements are called errors or noise Sources of errors: Accuracy, precision, resolution of the measurement tool Time required to read and store the current time value Time-sharing among multiple programs Processing of interrupts Cache misses, page faults

Quality of Measurement Types of errors: Systematic errors Are the result of some experimental mistake Usually constant across all measurements Ex: temperature may effect clock period Random errors Unpredictable, nondeterministic Effect the precision of measurement Ex: timer resolution ±T , effects measurements with equal probability

Quality of Measurement Experimental measurements follow Gaussian (normal) distribution Ex: x measured value ±E random error Two sources of errors, each having 50% probability Pg 48 Actual value of x is measured half of the time. Error 1 Error 2 Measured value Probability -E x-2E 1/4 +E x x+2E

Confidence Intervals Used to find a range of values that has a given probability of including the actual value. Case 1: number of measurements is large (n≥30) {x1, x2, … xn} - Samples Gaussian distribution m – mean s – standard deviation Confidence interval: [ c1, c2 ] Significance level:  Confidence coefficient: 1- Confidence level: (1-)×100 Pr[ c1 ≤ x ≤ c2 ] = 1- Pr[ x < c1 ] = Pr[ x > c2] = /2

Confidence Intervals Case 1: number of measurements is large (n≥30) Confidence interval: [ c1, c2 ] Sample variances s2 is a good estimate of 2. - Sample mean - Standard deviation is obtained from the precomputed table

Confidence Intervals Ex: number of measurements is large (n ≥ 30) 90% confidence interval means that there is a 90% chance that the actual mean is within that interval.

Normal Distribution

Confidence Intervals Case 2: number of measurements is small (n<30) Sample variances s2 can vary significantly. t distribution: - Sample mean - Standard deviation is obtained from the precomputed table

Confidence Intervals Ex: number of measurements is large (n < 30) 90% confidence interval means that there is a 90% chance that the actual mean is within that interval.

t Distribution

Wider interval  Less precise knowledge about the mean Confidence Intervals 90% c1= 6.5 c2= 9.4 95% c1= 6.1 c2= 9.7 99% c1= 5.3 c2=10.6 Wider interval  Less precise knowledge about the mean

Determining the Number of measurements Needed Confidence Intervals Determining the Number of measurements Needed

Determining the Number of measurements Needed Confidence Intervals Determining the Number of measurements Needed Estimating s: Make small number of measurements. Estimate standard deviation s. Calculate n. Make n measurements.

Confidence Intervals Ex: Based on a preliminary test, mean response time is 20 seconds and standard deviation is 5. How many repetitions are needed to get the response time accurate within 1 second at 95% confidence? e is 1 in 20 

Confidence Intervals Confidence Intervals for Proportions When we are interested in the number of times events occur. Binomial distribution: If np≥10 it approximates Gaussian distribution with mean p and variance p(1-p)/n - Total events recorded - Number of times desired outcome occurs is the sample proportion

Confidence Intervals for Proportions Determining the number of measurements needed:

Confidence Intervals Ex: How much time the processor spends executing the operating system compared with how much time it spends executing user programs? Experiment: At every 10 ms, an interrupt routine increments 2 counters: n=counts the number of interrupts occurred (incremented every time), m=counts if operating system is executing Results in 1 minute are m=658, n=6000 95% confidence level for this ratio is: With 5% chance of being wrong, we can say that the processor spends 10.2-11.8% of its time executing operating system.

Confidence Intervals Ex: How long must this experiment be run to know with 95% confidence that the processor spends executing operating system with an error of 0.5%?  Requires 3.46 hours