An Introduction to the Statistics of Uncertainty Tom LaBone April 17, 2009 SRCHPS Technical Seminar MJW Corporation University of South Carolina

Introduction All physical measurements must be reported with some quantitative measure of the quality of the measurement needed to decide if the measurement is suitable for a particular purpose The concept of “uncertainty” was developed in metrology to partially fill this need The US Guide to the Expression of Uncertainty in Measurement ANSI/NCSL Z540-2-1997 (R2007) provides guidance on calculating and reporting the uncertainty in a measurement US version of the ISO guide referred to as the “GUM”

Overview Illustrate the use of the GUM methodology using a relatively simple physical system Combined Standard Uncertainty Type A uncertainty only Probability Distributions Expanded Uncertainty Monte Carlo Methods Type B Uncertainty Student’s t Distribution

Example The SRS Health Physics Instrument Calibration Laboratory “sells” its radiation fields as a product The uncertainty attached to a radiation field helps the customer decide if the “product” is suitable for their application This is a rather involved case for such a short talk, so let us work with a less complex example

What is the density of the cube? Measure the height, width, length, and mass of the cube Calculate the density r using this formula Single measurements

Measurand The measurands we directly measure (mass and dimensions) are called input quantities The measurand we calculate (the density) is called the output quantity In this discussion the input quantities are assumed to be uncorrelated e.g., the measurement of the height does not influence the measurement of the length

Variability If we repeated the measurements again would we expect to see exactly the same result? Our measurements of dimension and mass will exhibit variability if we measure the “same thing” repeatedly we are likely get a range of answers that vary in a seemingly random fashion

Why Do Measurements Vary? Every measurement is influenced by a multitude of quantities that are not under our control and of which we may not even be aware (influence quantities) Random effects Measurements also vary because the measurand is not and cannot be specified in infinite detail For example, I did not specify how the linear measurements of the cube should be made

Errors Using the input and output quantities we have defined the “true” value of the density The error in a measurement is defined as The true value and hence the error are unknowable, but errors can be classified by how they influence the measurement random and systematic errors error = measured value of density – “true” value of density

Types of Errors Random errors result from random effects in the measurement the magnitude and sign of a random error changes from measurement to measurement measurements cannot be corrected for random errors …but random errors can be quantified and reduced Systematic errors results from systematic effects in the measurement the magnitude and sign of a systematic error is constant from measurement to measurement measurements can be corrected for known systematic errors …but the correction introduces additional random errors

What can we do about random errors? Law of Large Numbers If you repeat measurements many times and take the mean, this sample mean is a good estimator of the true population mean and is taken to be the best estimate of the thing we defined as the measurand Plug the sample means into the equation to obtain the best estimate of r sample means

Repeated Measurements Sample Mean central tendency

Precision of Result Precision is the number of digits with which a value is expressed The calculations here were performed to the internal precision of the computer (~16 digits) The density is arbitrarily presented with 9 digits of precision In which digit do we lose physical significance? ?

Uncertainty “…parameter associated with the result of a measurement that characterizes the dispersion of the values that could reasonably be attributed to the measurand….” an interval that we are reasonably confident contains the true value of the measurand the terms “random” and “systematic” are used with the term “error” but not with the term “uncertainty” associated with the measurement, not the measurement process

Evaluation of Uncertainty Type A evaluation of uncertainty evaluation of uncertainty by the statistical analysis of repeated measurements called Type A uncertainty Type B evaluation of uncertainty evaluation of uncertainty by any other method called Type B uncertainty

Repeated Measurements dispersion central tendency Sample mean sample standard deviation

Standard Uncertainty of Inputs The sample standard deviation s is a term in statistics with a precise meaning In metrology the analogous term is standard uncertainty u For Type A evaluations the standard deviation is the standard uncertainty This may not true for Type B evaluations

Significant Digits Report uncertainty to 2 digits round to even number if the last digit is 5 Round the measurement to agree with the reported uncertainty

Uncertainty in Density We have calculated the standard uncertainty in the input quantities (length, mass, etc) How do we get the standard uncertainty in the output quantity (density)? the combined standard uncertainty Propagation of uncertainty

Combined Standard Uncertainty sensitivity coefficient (often abbreviated as “c”) Given a small change in the length of the cube how much does the density change? Units must match up properly!

Standard Deviation of the Mean describes how repeated estimates of the mean are scattered around their grand mean (mean of the means) describes how individual measurements are scattered around their mean

Which Standard Deviation Should We Use? Sample standard deviation If you want to describe how individual measurements are scattered about their mean Standard deviation of the mean If you want to describe how multiple estimates of the mean are scattered about their grand mean also called the standard error of mean We need to use the standard deviation of the mean in the error propagation

Combined Standard Uncertainty Type A uncertainty only r = 1.46663 g/cm3 with a combined standard uncertainty uc = 8.8 x 10-4 g/cm3

Where We Are We have calculated the density and its combined standard uncertainty (Type A uncertainty only) Next, we want to calculate the expanded uncertainty and address the Type B uncertainty But, we need to discuss probability distributions and other such things first

Probability Distributions Up to this point we have described our data with the mean (central tendency) the standard deviation (dispersion) The mean and standard deviation do not uniquely specify the data Use a mathematical model that defines the probability of observing any given result probability density function (pdf)

Uniform (Rectangular) PDF otherwise a = 1 b = 9 m-s m+s a=1 b=9

Rectangular PDF Notation f(x) is the rectangular probability density function the value of the pdf is not the probability the area under the pdf is probability note that f(x) has units – probability has no units m is the population mean s is the population standard deviation a is the lower bound of the distribution a is a parameter in the pdf the probability of observing a value of x less than a is zero b is the upper bound of the distribution b is a parameter in the pdf the probability of observing a value of x greater than b is zero

Probability P(x < m-1s) =0.2113249 P(x < m+1s) =0.7886751 The area under the curve

Normal PDF m=5 m-s m+s The population parameters are the parameters in the pdf – this is unusual

Probability P(x < m-1s) =0.1586553 P(x < m+1s) =0.8413447 The area under the pdf curve

Normal vs Rectangular P(x < m+1s) =0.7886751 same mean and standard deviation

Sample Statistics and Population Parameters No matter what the probability distribution is, the sample mean and standard deviation are the best estimates (based on the observed data) of the population mean and standard deviation Sample Statistics Population Parameters

Random Numbers 1000 numbers drawn at random from the rectangular distribution 1000 numbers drawn at random from the normal distribution

Uses of PDFs We use the rectangular pdf to describe a random variable that is bounded on both sides and has the equal probability of appearing anywhere between the bounds The normal distribution has a special place in statistics because of the Central Limit Theorem

Central Limit Theorem As the sample size N gets “large”, the mean of a sample will be normally distributed regardless of how the individual values are distributed Theorem provides no guidance on what “large” is The standard deviation of the mean (aka the standard error of the mean) is equal to

So What? No matter what probability distribution you start with, if the sample is large enough the means of data drawn from that distribution are normally distributed What are the practical implications of this? All the input quantities (length, etc) are means The input quantities are normally distributed The output quantity (density) is normally distributed

Normal Probabilities The area under the normal curve m-1s m+1s m-1.96s m+1.96s The area under the normal curve between m-1s and m+1s = 0.6826895 The area under the normal curve between m-1.96s and m+1.96s = 0.95

Expanded Uncertainty It is often desirable to express the uncertainty as an interval around the measurement result that contains a large fraction of results that might reasonably be observed This is accomplished by using multiples of the standard uncertainty the multiplier is called the coverage factor

Intervals Confidence interval Coverage interval interval constructed with standard deviations from known probability distributions the interval has an exact probability of covering the mean value of the measurand Coverage interval interval constructed with uncertainties the interval does not have an exact probability of covering the mean value of the measurand only an approximation uses a coverage factor rather than a standard normal quantile (e.g., the 1.96) coverage factor of 2 (~95%) or 3 (~99%) is typically used

Expanded Uncertainty for Density Type A uncertainty only 68% Confidence Interval 95% Confidence Interval 95% Coverage Interval

Monte Carlo Methods Evaluation of measurement data – Supplement 1 to the “Guide to the expression of uncertainty in measurement” – Propagation of distributions using a Monte Carlo method OIML G 1-101 (2008) Run a statistics experiment using random numbers

Calculate the sample mean and standard deviations of the 106 densities 95% empirical CI

Implementation in R for (i in 1: (10^6)) { d[i] <- rnorm(1,M,s.M) / (rnorm(1,L,s.L) * rnorm(1,W,s.W) * rnorm(1,H,s.H)) } quantile(d,probs=c(0.025,0.975)) mean(d) sd(d) draw a random length, width, and height draw a random mass calculate a density calculate the empirical 95% confidence interval, mean and standard deviation

Advantages of Monte Carlo Intuitive Set up the experiment in the computer just like it occurs in the lab Able to handle very complex problems asymmetric probability distributions No need to mess with the t distribution or effective degrees of freedom you will see what I am talking about shortly

Type B Uncertainty Assessment Calipers Used to measure length, height, and width “Accuracy” of ± 0.02 mm (± 0.002 cm) for measurements <100 mm Scale Used to measure “mass” “Accuracy” of ± 0.0001 gram What do they mean by “accuracy” and how do I use this information?

Calipers The “accuracy” of ± 0.002 cm is taken to mean that if I moved the calipers from 1.500 cm to 1.502 cm the reading could be anywhere from 1.500 cm to 1.504 cm Assume rectangular distribution with an upper limit of X+0.002 cm and a lower limit of X – 0.002 cm The standard uncertainty of this distribution is

Scale The “accuracy” of ± 0.0001 gram is taken to mean that if the weight increased from 5.0000 grams to 5.0001 grams the reading could be anywhere from 5.0000 grams to 5.0002 grams Assume rectangular distribution with an upper limit of X+0.0001 grams and a lower limit of X – 0.0001 grams The standard uncertainty of this distribution is

Standard Uncertainty for Length Combine the Type A and Type B uncertainties in quadrature (i.e., add the variances) Includes Type A and Type B uncertainties The notation uc(L) is used here to indicate the uncertainty includes Type A and B uncertainties

Combined Standard Uncertainty for Density Type A and B uncertainty r = 1.4666 g/cm3 with a combined standard uncertainty uc = 2.1 x 10-3 g/cm3

Type B Uncertainties Remember, once Type B uncertainties are included in the error propagation the equivalence of standard deviations and standard uncertainties is usually lost The assessment of Type B uncertainties usually requires some degree of professional judgment and experience Once you decide what the Type B uncertainty is, it is treated the same as a Type A uncertainty One goal of GUM is to make Type B uncertainties easier to handle by having people report the right information along with the uncertainty itself

Summary of Jargon Standard uncertainty Combined standard uncertainty uncertainty of the result of a single type of measurement (e.g., length) includes Type A and/or Type B uncertainties Combined standard uncertainty standard uncertainties from multiple types of measurements used to calculate an output quantity (e.g., density) Expanded uncertainty the standard uncertainty multiplied by a coverage factor combined standard uncertainty multiplied by a coverage factor

Where We Are For the density, we have calculated combined standard uncertainty expanded uncertainty (including both Type A and Type B uncertainties) The only remaining issue is to account for the impact “small” samples have on the expanded uncertainty

Student’s t Distribution If s is given, i.e., not determined from the data, then k is a standard normal quantile, e.g., 1.96 gives a 95% confidence interval If s is not given and we use s, which is determined from the data, then k is a not standard normal quantile, it is a quantile from a Student’s t distribution

Student’s t PDF n = degrees of freedom (df) t The t distribution looks a lot like the normal distribution but has “fatter” tails normal

So What? A 95% coverage interval for a normal distribution is given by 1.96s A 95% coverage interval for a t distribution might be more like 3s (depending the df) The t distribution converges to the normal distribution as the degrees of freedom (the size of the sample) gets large The t distribution is needed to calculate coverage intervals for small samples taken from normally distributed data for large samples you can use normal quantiles

Expanded Uncertainty for Length (Type A uncertainty only) mean of N = 15 measurements (see slide 21) standard error of the mean length (see slide 21) standard normal quantile for p = 0.95 95% confidence interval t quantile for p = 0.95 (14 df) 95% confidence interval

Degrees of Freedom Data can be used to estimate parameters or estimate variance degrees of freedom is the number of data available to estimate variance after the parameters are estimated one degree of freedom was used to calculate the mean which leaves 15 -1 = 14 left to calculate variance The GUM views degrees of freedom as an indication of the uncertainty in the uncertainty large degrees of freedom = small uncertainty small degrees of freedom = large uncertainty

Expanded Uncertainty for Length (Type A and Type B uncertainty) The Type A uncertainty of the length has 14 degrees of freedom How many degrees of freedom does the Type B uncertainty have? unless “they” tell you the degrees of freedom, you end up doing a bit of hand waving on the df for Type B uncertainties the GUM gives a way to get an approximate df How many degrees of freedom does the combined uncertainty of Length have? There is no exact solution to this problem An approximate solution is given by the Welch-Satterthwaite equation

Degrees of Freedom for Type B Uncertainty Where (Du/u) is the relative uncertainty in the uncertainty 25% relative uncertainty in the uncertainty 10% relative uncertainty in the uncertainty Zero uncertainty in the uncertainty

Welch Satterthwaite Gives the effective degrees of freedom associated with the expanded uncertainty in the length The sensitivity coefficients c are equal to 1 here The Type B uncertainty is assumed to have an infinite degrees of freedom, i.e., the uncertainty has no uncertainty Use this to calculate the t quantile

Expanded Uncertainty for Length (Type A and Type B uncertainty) mean of N = 15 measurements standard uncertainty in length (see slide 49) standard normal quantile for p = 0.95 95% coverage interval t quantile for p = 0.95 (323 df) 95% coverage interval

Expanded Uncertainty for Density (Type A Uncertainty Only)

standard uncertainty uc = 8.8 x 10-4 g/cm3 r = 1.46663 g/cm3 with a combined standard uncertainty uc = 8.8 x 10-4 g/cm3 (see slide 23) (see slide 23) standard normal quantile for p = 0.95 95% confidence interval t quantile for p = 0.95 (41 df) 95% confidence interval

Expanded Uncertainty for Density (Type A and Type B Uncertainty) The Type B uncertainty is assumed to have an infinite degrees of freedom, i.e., the uncertainty has no uncertainty Same as the normal distribution

standard uncertainty uc = 2.1 x 10-3 g/cm3 r = 1.4666 g/cm3 with a combined standard uncertainty uc = 2.1 x 10-3 g/cm3 (see slide 50) (see slide 50) standard normal quantile for p = 0.95 95% coverage interval t quantile for p = 0.95 (1200 df) 95% coverage interval – the final answer!

Why Bother with Student? It is important not to overstate your confidence in a number if you make an error calculating the coverage interval, try to make it too big When you start to include all known sources of uncertainty, some are very likely to have small degrees of freedom the weakest link determines the strength of the chain same goes for uncertainty calculations

Monte Carlo R Code for (i in 1:(10^6)) { r.L <- (L + rt(1,14)*s.L) r.L <- runif(1,r.L-0.002,r.L+0.002) r.W <- (W + rt(1,14)*s.W) r.W <- runif(1,r.W-0.002,r.W+0.002) r.H <- (H + rt(1,14)*s.H) r.H <- runif(1,r.H-0.002,r.H+0.002) r.M <- (M + rt(1,14)*s.M) r.M <- runif(1,r.M-0.0001,r.M+0.0001) d[i] <- r.M / (r.L * r.W * r.H) } quantile(d,probs=c(0.025,0.975)) mean(d) sd(d) Expanded Uncertainty for Density (Type A and Type B Uncertainty) MC 95% coverage interval = (1.4626, 1.4707) GUM 95% coverage interval = (1.4625, 1.4707) (see slide 66)

Summary Combined Standard Uncertainty Probability Distributions Type A uncertainty only Probability Distributions rectangular, normal Expanded Uncertainty Monte Carlo Methods Type B Uncertainty uncertainty in calipers and scale Student’s t Distribution degrees of freedom in Type B uncertainty degrees of freedom in combined uncertainty Welch-Satterthwaite

Recommended Reading The GUM and its Monte Carlo supplement already cited An Introduction to Uncertainty in Measurement by Les Kirkup and Bob Frenkel (Cambridge University Press:2006) An Introduction to Error Analysis by John Taylor (University Science Book: 1982)