1. 1 Speaker/Author: Ricardo A. Nicholas Puget Sound Metrology, Measurement Uncertainty Engineering The Boeing Company P.O. Box 3707, MC 2T-40 Seattle, WA 98124 USA Phone: (206) 544-0569; FAX: (206) 544-5907 E-mail: ricardo.a.nicholas@boeing.com Measurement Uncertainty Evaluation Involving Significant Uncorrected Systematic Errors

2. 2 Overview Introduction GUM Approach Reporting Conformance Summary

3. 3 Introduction The majority of measurements typically contain significant uncorrected systematic errors, e.g., –Micrometer, Caliper, Height gage, Surface plate –Thermometer, Barometer, Hygrometer –Voltmeter, Ammeter, Ohmmeter, Oscilloscope –Attenuator, Amplifier, Signal conditioner –Scale, Balance, Densitometer, Hydrometer –Seismograph, Accelerometer, Electrical filter –Geiger counter, Spectrometer, Gloss meter –Rockwell hardness tester, pH meter, Durometer –Flow meter, Dynamometer, Tension tester

4. 4 Introduction A measurement made with an instrument containing a significant uncorrected systematic error generally has an under-estimated measurement uncertainty specification because a biased distribution has a smaller confidence level for the same standard deviation and specification limits, than an unbiased one. A properly evaluated confidence level for this case would necessarily have larger specification limits to achieve the correct confidence level.

5. 5 Introduction The following are some examples of possibly under- estimated confidence levels. –A mechanical micrometer with an unevenly worn lead screw has a characteristically variable error as a function of lead screw engagement. –A surface plate with a non-flat surface has a characteristically variable error as a function of measurement location. –A voltmeter with different signal conditioning circuitry for each range has a characteristically variable error as a function of the range used.

6. 6 GUM Uncertainty (of measurement) – Parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand. (GUM, B.2.18) –Note 1: By “characterizes the dispersion” is meant that a confidence level and confidence interval can be estimated based on either a statistical analysis of a series of observations, or by other means, I.e., either a Type A or Type B uncertainty evaluation, respectively. –Note 2: By “dispersion” is meant to exclude significant uncorrected systematic errors that cannot reasonably be modeled by the distribution of a random variable.

7. 7 GUM - Based Evaluation (expects)

8. 8 Pseudo GUM - Based Evaluation (typically has)

9. 9 Approach The overall objective is to estimate the area under the distribution within the specification limits. This approach is consistent with the GUM because the area is representative of the confidence level and the the measurand units bounded by the specification limits is representative of the confidence interval.

10. 10 Approach (general case)

11. 11 Approach (reported capability >> typical capability)

12. 12 Approach (reported capability = actual capability with one significant distribution tail)

13. 13 Approach (reported capability = actual capability with two significant distribution tails)

14. 14 Approach (reported capability = actual capability with two significant distribution tails) Excel: NORMSDIST(1.725)-0.5+NORMSDIST(2.725)-0.5 = 0.9545 or Excel: NORMSDIST(1.725)-NORMSDIST(-2.725) = 0.9545 The confidence level can be obtained from a Gaussian distribution table, or by using a software tool such as Excel. Table :

15. 15 Approach Required 1 : –To ensure conformance to customer, contract, legal or safety requirements, or Boeing drawings, documents, and specifications. –To produce measurement data delivered to a customer, published outside the company, or used for internal reference purposes. 1 PRO 1087 “Calibration and Certification of Measurement and Test Equipment”

16. 16 Reporting Result of Measurement –Confidence Interval – f (Calibration X ) –E.g., (±1.8  m) –Confidence Level – f (Calibration X ) –E.g., (95.45%)

17. 17 Conformance Reliability –% within Confidence Interval –f (Calibration 1, Calibration 2,... Calibration n ) of the same population –E.g., (87.5%) –Quality Corridor (85% – 95%)

18. 18 Summary Measurement Uncertainty Evaluation –Document measurement process –Establish functional relationship of variables –Apply law of propagation of uncertainty –Determine combined standard uncertainty –Determine expanded uncertainty

19. 19 Management Methods Adjust performance specification Revise calibration procedure Recommend different standards Consider user environment Specify user restrictions Assign corrections Change calibration interval Employ guardbanding –Survive Competitive Sufficiently profitable Comprehensive Technical Management –Engineering requirements –Drawing tolerances –M&TE performance specifications

20. 20 Guardbanding Reduced Acceptance Limit(s) –Formula-based practice Performance Specification – Expanded Uncertainty –Risk-based practice Consumer risk (false accept) Producer risk (false reject)

21. 21 Risk Consumer Risk (average quality level = 85%) –The unconditional probability that the true value of a measurand exceeds a given performance specification, but is measured to be within the guardband limit(s). 0.80%:Uncertainty ratio of 13.2:1 1.89%:Uncertainty ratio of 5:1 3.65%:Uncertainty ratio of 2:1 5.11%:Uncertainty ratio of 1:1

22. 22 Risk Consumer Risk: A function of –the historical average quality level of the measurand (with data from the same population) –measurand distribution and calibration system distribution type, parameters and net bias –distribution width ratio of measurand and calibration system (on the same confidence level basis) –bilateral or unilateral measurand.

23. 23 Risk Tolerance Ratio: The ratio between the engineering tolerance and the calibration standard tolerance –Least complex, time intensive and accurate Error Ratio: The ratio between the maximum permissible error and the calibration system error –Moderately to most complex, time intensive and accurate Uncertainty Ratio: The ratio between the measurand uncertainty and the calibration system uncertainty –Most complex, time intensive and accurate

24. 24 Risk Producer Risk (average quality level = 85%) –The unconditional probability that the true value of a measurand is within a given performance specification, but is measured to exceed the guardband limit. 0.92%:Uncertainty ratio of 13.2:1 2.69%:Uncertainty ratio of 5:1 8.44%:Uncertainty ratio of 2:1 21.0%:Uncertainty ratio of 1:1

25. 25 Risk Consumer Risk versus Producer Risk –Inversely related with respect to guardband limit, i.e., as the guardband limits are lowered, the consumer risk decreases and producer risk increases. –As the guardband limits are lowered, the risk to the consumer decreases and the cost to the producer increases.

26. 26 Risk

27. 27 Cost Cost generally increases with: –Formula-based guardband practice Performance Specification – Expanded Uncertainty Cost generally decreases with: –Risk-based guardband practice Consumer risk-based Producer risk-based Combination risk-based

28. 28 Cost Consumer Risk-Based Guardbanding –Accept the resulting false reject producer cost. Producer Risk-Based Guardbanding –Accept the resulting false accept consumer risk. Combination Risk-Based Guardbanding –Achieve desired balance between risk and cost.

29. 29 Summary Accomplishments –Computer-based guardbanding tools –Training courses (fundamental to advanced) –Local measurement uncertainty mentors Direct savings per year from using risk-based guardbanding in place of formula-based practice for Puget Sound Metrology (PSM)/Boeing –$169K/year realized (40% PSM utilization) –$422K/year expected (100% PSM utilization) –$1.27M/year expected (100% Boeing utilization)

30. 30 Thank you. Questions ?