Putting Confidence Into Your Lab’s Results Alan Steele, Barry Wood & Rob Douglas National Research Council Ottawa, CANADA National ResearchConseil national Council Canadade recherches

Slide 2 Steele Wood and Douglas: Confidence NCSL International, July 2001 Outline Comparison Measurements –interpretation of results –proficiency testing for accreditation Probability Calculus –confidence intervals –confidence levels A Toolkit for Excel –some Visual Basic Code A Worked Example –with real comparison data Conclusions

Slide 3 Steele Wood and Douglas: Confidence NCSL International, July 2001 Proficiency Testing Accreditation bodies routinely specify that “proficiency testing” on a regularly scheduled basis is a requirement for maintaining accreditation Usually the Pilot Laboratory for the comparison is the National Metrology Institute Usually the Pilot Laboratory result is taken as the comparison reference value, and the participants’ are evaluated against this “truth” This is a time-consuming and expensive exercise!

Slide 4 Steele Wood and Douglas: Confidence NCSL International, July 2001 Comparisons Measurement comparisons provide the main experimental evidence for “equivalence” In general, all participants measure a common artifact and their various results are analyzed from a single common perspective The participants may be different laboratories, or different measurement stations on your shop floor

Slide 5 Steele Wood and Douglas: Confidence NCSL International, July 2001 Key Comparisons and NMIs National Metrology Institutes have recently signed a “Mutual Recognition Arrangement” in which the validity of their Calibration and Measurement Capabilities is expressed The scientific underpinning for this arrangement is a series of “Key Comparisons” which are conducted at the very highest levels of metrology In practice, they are not much different from the proficiency tests already in general use among accredited laboratories around the world

Slide 6 Steele Wood and Douglas: Confidence NCSL International, July 2001 Reporting Results A metrologist reports a result in two parts –the mean value: m L –the uncertainty: u L The results are plotted as data points with error bars

Slide 7 Steele Wood and Douglas: Confidence NCSL International, July 2001 Uncertainty Budgets The ISO Guide to the Expression of Uncertainty of Measurement is widely used as the basis for formulating and publishing laboratory uncertainty statements regarding measurement capabilities “Error bars” are an intrinsically probabilistic description of our belief in “what will happen next time” based on what we have done in the past Flip x and y axes

Slide 8 Steele Wood and Douglas: Confidence NCSL International, July 2001 Probability Distributions An ISO Guide-compliant uncertainty statement means that the error bars represent the most expert opinion about the underlying normal (Gaussian) probability distribution The fancy name for working with these distributions is Probability Calculus In general, we are interested in integrals of the probability distribution Integration is only “fancy addition”

Slide 9 Steele Wood and Douglas: Confidence NCSL International, July 2001 Confidence Levels A confidence level is what we get upon integrating a probability distribution over a given range [a,b] The fractional probability of observing a value between a & b is the normalized integration of the probability distribution function in the range [a, b] This is just addition of all the ‘bits’ of the function between a & b 1   68%2   95%

Slide 10 Steele Wood and Douglas: Confidence NCSL International, July 2001 Confidence Intervals Remember: a confidence level is what we get by integrating the distribution over a given range [a,b] The confidence interval is the fancy name for the range associated with the confidence level The range [-1 ,+1  ] is the 68% confidence interval The range [-2 ,+2  ] is the 95% confidence interval 1   68%2   95%

Slide 11 Steele Wood and Douglas: Confidence NCSL International, July 2001 Why would you want to do this? Lots of time and energy (and expense!) is invested in creating a laboratory result in a comparison Getting the maximum amount of information from a measurement comparison is desirable You’d like to show off your “confidence” to colleagues (and auditors!) Quantifying things is what we do as metrologists Your clients may want specific quantified answers to questions of Demonstrated Equivalence based on your Proficiency Testing results

Slide 12 Steele Wood and Douglas: Confidence NCSL International, July 2001 How hard is it to do this? With normal distributions, the arithmetic is pretty easy You can try this for yourself and really see how it works… …or you can let us do it for you! We have generated simple expressions to help evaluate normal confidence levels and normal confidence intervals, using well known statistical methods developed over the last hundred years or so We have put these expressions into a Toolkit for Excel

Slide 13 Steele Wood and Douglas: Confidence NCSL International, July 2001 A Toolkit for Excel At NRC, we have written a Quantified Demonstrated Equivalence Toolkit for Microsoft Excel ® The Toolkit is freely available by contacting us at We’ll add you to our mailing list and send you a copy of the sample spreadsheet with the Toolkit, plus a “User’s Guide” in.pdf format

Slide 14 Steele Wood and Douglas: Confidence NCSL International, July 2001 Toolkit Functions and Macros The Toolkit contains Functions to: –calculate pair uncertainties (including correlations) –calculate weighted averages –calculate confidence levels –calculate confidence intervals The Toolkit contains Macros to: –generate bilateral “tables of equivalence” –generate bilateral “tables of confidence intervals” –generate bilateral “tables of confidence levels”

Slide 15 Steele Wood and Douglas: Confidence NCSL International, July 2001 Toolkit Philosophy and Operation Functions and Macros are built right in to the Spreadsheet, and work just like “regular” Excel components

Slide 16 Steele Wood and Douglas: Confidence NCSL International, July 2001 Toolkit Philosophy and Operation The code is written in Visual Basic You can examine the code to see how it works Long variableNames help to “self document” the programs You don’t have to look at the code or write your own functions to use the QDE Toolkit from NRC

Slide 17 Steele Wood and Douglas: Confidence NCSL International, July 2001 A Worked Example 13 Laboratories participated in a Proficiency Test at 10 k 

Slide 18 Steele Wood and Douglas: Confidence NCSL International, July 2001 Comparison to the NMI: E n One common measure of success in Proficiency Tests is the “Normalized Error” This is the ratio of the laboratory deviation to the expanded uncertainty: E n (k=2) = abs(m Lab - m Ref )/sqrt(U Lab 2 + U Ref 2 ) Generally, the Laboratory “passes” when E n < 1 E n is a dimensionless quantity

Slide 19 Steele Wood and Douglas: Confidence NCSL International, July 2001 Comparison to the NMI: QDC A quantified approach to Proficiency Tests is to ask the following question: What is the probability that a repeat comparison would yield results such that Lab 1’s 95% uncertainty interval encompasses the Pilot Lab value? We call this “Quantified Demonstrated Confidence” QDC is a dimensionless quantity expressed in %

Slide 20 Steele Wood and Douglas: Confidence NCSL International, July 2001 Comparison to the NMI: E n vs QDC and are both dimensionless quantities E n and its interpretation as an acceptance criterion are difficult to explain to non-metrologists QDC and its numerical value are easily explained to non-metrologists Note that when E n = 1 (and U Ref << U Lab ) QDC = 50% Normalized ErrorQuantified Demonstrated Confidence

Slide 21 Steele Wood and Douglas: Confidence NCSL International, July 2001 Comparison to the NMI: QDE 0.95 A different quantified approach to Proficiency Tests is to ask the following question: Within what confidence interval can I expect the Lab 1 value and the Pilot Lab value to agree, with a 95% confidence level? We call this “Quantified Demonstrated Equivalence” QDE 0.95 is a dimensioned quantity, same units as V

Slide 22 Steele Wood and Douglas: Confidence NCSL International, July 2001 Comparison between Labs: Agreement We can ask similar questions about agreement between any two participants in the experiment: Within what confidence interval (in ppm) can I expect the Lab 1 value and the Lab 2 value to agree, with a 95% confidence level?

Slide 23 Steele Wood and Douglas: Confidence NCSL International, July 2001 Comparison between Labs: Confidence What if we ask: What is the probability that a repeat comparison would yield results such that Lab 1’s 95% uncertainty interval encompasses Lab 2’s value? Or how about: What is the probability that a repeat comparison would yield results such that Lab 2’s 95% uncertainty interval encompasses Lab 1’s value?

Slide 24 Steele Wood and Douglas: Confidence NCSL International, July 2001 Comparison between Labs: Confidence The answers to these questions of Quantified Demonstrated Confidence are shown here

Slide 25 Steele Wood and Douglas: Confidence NCSL International, July 2001 Quantifying Equivalence What is the probability that a repeat comparison would have a Lab 2 value within Lab 1’s 95% uncertainty interval? Probability Calculus tells us the answer: QDC = 47% This is exactly the type of “awkward question” that a Client might ask! 95% interval

Slide 26 Steele Wood and Douglas: Confidence NCSL International, July 2001 Quantifying Equivalence What is the probability that a repeat comparison would have a Lab 1 value within Lab 2’s 95% uncertainty interval? Probability Calculus tells us the answer: QDC = 22% These subtly different “awkward” questions have very different “straightforward” answers! 95% interval

Slide 27 Steele Wood and Douglas: Confidence NCSL International, July 2001 Tricky things about Equivalence Equivalence is not transitive –Lab 1 and Lab 2 may both be “equivalent” to the Pilot, but not to each other! Equivalence is not commutative –we are asking two very different questions here! 95% interval QDC = 47% 95% interval QDC = 22%

Slide 28 Steele Wood and Douglas: Confidence NCSL International, July 2001 Conclusions You are already doing quite a bit of Probability Calculus when you present your results The arithmetic for quantified calculations is very straightforward when we have Normal Distributions Adding Statistical Confidence explicitly into your Lab’s results helps you to explain them to non-metrologists, and to present precisely what Proficiency Testing has demonstrated for: –equivalence from different National Laboratories –accreditation assessment –your clients –your factory floor