UNCERTAINTY ANALYSIS: A BASIC OVERVIEW presented at CAVS by GLENN STEELE www.uncertainty-analysis.com August 31, 2011 Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

EXPERIMENTAL UNCERTAINTY REFERENCES The ISO GUM: The de facto international standard Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

EXPERIMENTAL UNCERTAINTY REFERENCES http://www.oiml.org/publications/?publi=3&publi_langue=en Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

VALIDATION REFERENCES Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

VALIDATION REFERENCES Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

“Degree of Goodness” When we use experimental results (such as property values) in an analytical solution, we should consider “how good” the data are and what influence that degree of goodness has on the interpretation and usefulness of the solution When we compare model predictions with experimental data, as in a validation process, we should consider the degree of goodness of the model results and the degree of goodness of the data. Copyright 2010 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Typical comparison of predictions and data, considering no uncertainties: Result, CD Set point, Re Copyright 2010 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Comparison of predictions and data considering only the likely uncertainty in the experimental result: Result, CD Set point, Re Uncertainties set the resolution at which meaningful comparisons can be made. Copyright 2010 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Validation comparison considering all uncertainties: S  value from the simulation D  data value from experiment E  comparison error E = S - D = S- D where (S= model+ input+ num) Result, CD URe UCD US Set point, Re Copyright 2010 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

“Degree of Goodness” and Uncertainty Analysis When an experimental approach to solving a problem is to be used, the question of “how good must the results be?” should be answered at the very beginning of the effort. This required degree of goodness can then be used as guidance in the planning and design of the experiment. We use the concept of uncertainty to describe the “degree of goodness” of a measurement or an experimental result. Copyright 2010 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

ERRORS & UNCERTAINTIES Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

An error  is a quantity with a sign and magnitude An error  is a quantity with a sign and magnitude. (We assume any error whose sign and magnitude is known has been corrected for, so the errors that remain are of unknown sign and magnitude.) An uncertainty u is an estimate of an interval u that should contain . Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Consider making a measurement of a steady variable X (whose true value is designated as Xtrue) that is influenced by errors i from 5 elemental error sources. Postulate that errors 1 and 2 do not vary as measurements are made, and 3, 4, and 5 do vary during the measurement period: Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

 (varies) β (does not vary) The total error () is the sum of 1 2 The total error () is the sum of  (=  1 +  2) the systematic, or fixed, error  (=  3 +  4 +  5) the random, or repeatability, error  =  +  Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

The kth measurement of X then appears as The total error (k) is the sum of  k the systematic, or fixed, error  k the random, or repeatability, error Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Central Limit Theorem   statistics   ??? Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Histogram of temperatures read from a thermometer by 24 students Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Now consider again making the measurements of X  (varies) β (does not vary) 1 2 Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

We can calculate the standard deviation sX of the distribution of N measurements of X and that will correspond to a standard uncertainty (u) estimate of the range of the i’s. We will call sX the random standard uncertainty. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Copyright 2011 by Coleman & Steele Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

We will estimate systematic standard uncertainties corresponding to the elemental systematic errors i and use the symbol bi to denote such an uncertainty. Thus ±b1 will be an uncertainty interval that should contain 1, ±b2 will be an uncertainty interval that should contain 2, and so on.... The systematic standard uncertainty bi is understood to be an estimate of the standard deviation of the parent population from which the systematic error i is a single realization. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

and, in accordance with the GUM, is given by The standard uncertainty in X -- denoted uX -- is defined such that the interval ± uX contains the (unknown) combination and, in accordance with the GUM, is given by Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Categorizing and Estimating Uncertainties in the Measurement of a Variable GUM categorization by method of evaluation: Type A  “method of evaluation of uncertainty by the statistical analysis of series of observations” Type B  “method of evaluation of uncertainty by means other than the statistical analysis of series of observations” Traditional U.S. categorization by effect on measurement: Random (component of) uncertainty  estimate of the effect of the random errors on the measured value Systematic (component of) uncertainty  estimate of the effect of the systematic errors on the measured value Both are useful, and they are not inconsistent. Use of both will be illustrated in the examples in this course. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

An Additional Uncertainty Categorization In the fields of Risk Analysis, Reliability Engineering, Systems Safety Assessment, and others, uncertainties are often categorized as Aleatory Variability Due to a random process Epistemic Incertitude Due to lack of knowledge Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Uncertainty Categorization 100 % The key is to identify the significant errors and estimate the corresponding uncertainties – whether one divides them into categories for convenience of Random – Systematic Type A – Type B Aleatory – Epistemic Lemons – Chipmunks should make no difference in the overall estimate u if one proceeds properly. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

OVERALL UNCERTAINTY OF A MEASUREMENT At the standard deviation level Systematic Standard Uncertainty = (for 2 elemental systematic errors) Random Standard Uncertainty = sX (or ) Combined Standard Uncertainty = uX Overall or Expanded Uncertainty at C % confidence Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

To obtain a value of the coverage factor k, an assumption about the form of the distribution of the total errors (the ’s) in X is necessary. For large samples, assuming the total errors in the measurements have a roughly Gaussian distribution, and using a 95% confidence level, k95 = 2 and The true value of the variable will then be within the limits about 95 times out of 100. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

RESULT DETERMINED FROM MULTIPLE MEASURED VARIABLES Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

We usually combine several variables using a Data Reduction Equation (DRE) to determine an experimental result. These have the general DRE form There are two approaches used for propagating uncertainties through the DREs: the Taylor Series Method (TSM) the Monte Carlo Method (MCM) Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

TAYLOR SERIES METHOD OF UNCERTAINTY PROPAGATION For the case where the result r is a function of two variables x and y r = f(x,y) the combined standard uncertainty of the result, ur, is given by where sr is calculated from multiple result determinations and the bx and by systematic standard uncertainties are determined from the combination of elemental systematic uncertainties that affect x and y as and Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Uncertainty Propagation Monte Carlo Method of Uncertainty Propagation Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Applying General Uncertainty Analysis – Experimental Planning Phase Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

GENERAL UNCERTAINTY ANALYSIS For a result given by a data reduction equation (DRE) the uncertainty is given by Example DRE Note that (assuming the large sample approximation) the U in the propagation equation can be interpreted as the 95% confidence U95 = 2 u or as the standard uncertainty u as long as each term in the equation is treated consistently. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Example It is proposed that the shear modulus, MS, be determined for an alloy by measuring the angular deformation  produced when a torque T is applied to a cylindrical rod of the alloy with radius R and length L. The expression relating these variables is We wish to examine the sensitivity of the experimental result to the uncertainties in the variables that must be measured before we proceed with a detailed experimental design. The physical situation shown below (where torque T is given by aF) is described by the data reduction equation for the shear modulus Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Copyright 2011 by Coleman & Steele Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

ESTIMATING RANDOM UNCERTAINTIES Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Data sets for determining estimates of standard deviations and random uncertainties should be acquired over a time period that is large relative to the time scales of the factors that have a significant influence on the data and that contribute to the random errors. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Direct Calculation Approach for Random Uncertainty For a result that is determined M times the mean value of the result is and Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

ESTIMATING SYSTEMATIC UNCERTAINTIES Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Propagation of systematic errors into an experimental result: Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

use of previous experience, manufacturer’s specifications, The systematic standard uncertainties for the elemental error sources are estimated in a variety of ways that were discussed in some detail in the course. Among the ways used to obtain estimates are: use of previous experience, manufacturer’s specifications, calibration data, results from specially designed “side” experiments, results from analytical models, and others. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Recall the definition of a systematic standard uncertainty, b Recall the definition of a systematic standard uncertainty, b. It is not the most likely value of  , nor the maximum value. It is the standard deviation of the assumed parent population of possible values of . Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

SYSTEMATIC STANDARD UNCERTAINTY Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Copyright 2011 by Coleman & Steele Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Correlated Systematic Errors Typically occur when different measured variables share one or more elemental error sources multiple variables measured with same transducer probe traversed across flow field multiple pressures ported sequentially to the same transducer (scanivalve) multiple transducers calibrated against same standard electronically scanned pressure (ESP) systems in use in aerospace ground test facilities Examples q = m Cp (To – Ti) u’v’ Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Using the TSM, there is a term in the br2 equation for each pair of variables in the DRE that might share an error source: For q = m Cp (To – Ti) For For u’v’ .... Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Copyright 2011 by Coleman & Steele Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Some Final Practical Points on Estimating Systematic Uncertainties When estimating b, we are not trying to estimate the most probable value nor the maximum possible value of  Always remember to view and use estimates with common sense. For example, a “% of full scale” b should not apply near zero if the instrument is nulled. Resources should not be wasted on obtaining good uncertainty estimates for insignificant sources – a practice we have observed too many times…. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

“V&V” – Verification & Validation: The Process Preparation Specification of validation variables, validation set points, etc. (This specification determines the resource commitment that is necessary.) It is critical for modelers and experimentalists to work together in this phase. The experimental and simulation results to be compared must be conceptually identical. Verification Are the equations solved correctly? (MMS for code verification. Grid convergence studies, etc, for solution verification to estimate unum .) Validation Are the correct equations being solved? (Compare with experimental data and attempt to assess model ) Documentation Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

A Validation Comparison Copyright 2008 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

V&V Overview – Sources of Error Shown in Ovals Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Strategy of the Approach Isolate the modeling error, having a value or uncertainty for everything else E=S-D = model + (input +num - D) model = E - (input +num - D) If ± uval is an interval that includes (input +num - D) then model lies within the interval E ± uval E ± uval Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Uncertainty in experimental result, uD Uncertainty Estimates Necessary to Obtain the Validation Uncertainty uval Uncertainty in simulation result due to numerical solution of the equations, unum (code and solution verification) Uncertainty in experimental result, uD Uncertainty in simulation result due to uncertainties in code inputs, uinput Propagation by Taylor Series Monte Carlo Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Methodology Simulation Uncertainty Modeling error for uncalibrated model used to make calculations between validation points where usp = uncertainty contribution from the uncertainty of input parameters at the simulation calculation point and uE = uncertainty in E at the calculation point from the interpolation process Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Uncertainty of Calibrated Models Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Methodology Instrument Calibration Analogy Uncalibrated instrumentation system where ut = uncertainty of the transducer and um = uncertainty of the meter Calibrated instrumentation system where uc is the calibration uncertainty Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Methodology Instrument Calibration Analogy If a curve-fit is used to develop a relationship between the meter reading and the calibrated output value, then where ucf = the curve-fit uncertainty If the meter used in testing (m2) is different from the meter used in calibration (m1), then Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Methodology Instrument Calibration Analogy The uncertainties, u, in the previous expressions are standard uncertainties, at the standard deviation level. To express the uncertainty at a given confidence level, such as 95%, the standard uncertainty is multiplied by an expansion factor. For most engineering applications, the expansion factor is 2 for 95% confidence. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Methodology Calibrated Model To calibrate a model, the simulation results are compared with a set of data and corrections are applied to the model to make it match the data. The simulation uncertainty is then As in the curve-fit uncertainty in the calibration of a transducer, there will be additional uncertainty in the calibrated model based on the error between the corrected simulation results and the data. Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.

Methodology Calibrated Model would apply for simulation results over the range of the input parameter values used in the calibration of the model with the assumption that the input parameters in the simulation have the same uncertainties that they had in the calibration process. If the input parameter sources or transducers change for a simulation result, then Copyright 2011 by Coleman & Steele. Absolutely no reproduction of any portion without explicit written permission.