1. UNCERTAINTY OF MEASUREMENT Andrew Pascall Technical Director Integral Laboratories (Pty) Ltd andrew@integrallabs.co.za www.integrallabs.co.za

2. I used to be uncertain - now I'm not so sure www.integrallabs.co.za

3. Items covered in this presentation Why is measurement uncertainty so important? What is the difference between error and uncertainty? Where does measurement uncertainty come from? The basic statistical analysis used to calculate uncertainty – how many readings are needed? How to get the best estimate? How can we assess whether our estimate is reasonable? What can lab personnel do to reduce the measurement uncertainty in their labs? What is coverage factor / confidence level? How to express the results? www.integrallabs.co.za

4. Why do we need uncertainty? The aim of measurement is to obtain the true value Every effort is made to optimize the measurement procedure so that the measured value is as close as possible to the true value. measurements are just an estimate of the true value. The actual true value of an analyte will (almost) always remain unknown to us. Therefore, we cannot know exactly how near our measured value is to the true value – our estimate always has some uncertainty associated with it. www.integrallabs.co.za

5. What is the difference between error and uncertainty? The difference between the measured (or estimated) value and the true value is called error – composed of two parts random error and systematic error – Can be positive or negative The quality of the measurement result, its accuracy, is characterized by measurement uncertainty – which defines an interval around the measured value C MEASURED, where the true value C TRUE lies with some probability. The measurement uncertainty “u” itself is the half-width of that interval and is always non-negative (“U” = expanded uncertainty = u x k where k = coverage factor) www.integrallabs.co.za

6. What is the difference between error and uncertainty? Acceptance range for a CRM e.g Titrivin AA5 Batch A031212175 Reducing Sugars 2.73 g/L U = 0.32g/L, u = 0.16g/L 2.57 – 2.89 g/L, k=2, 95% Confidence level 2.73 2.57 2.89 www.integrallabs.co.za

7. What is the difference between error and uncertainty? Uncertainty Error 2.73 www.integrallabs.co.za

8. What is the difference between error and uncertainty? What is our aim as analytical scientists? -To put measures in place to reduce the uncertainty www.integrallabs.co.za Uncertainty

9. Some sources of Uncertainty www.integrallabs.co.za

10. Why is measurement uncertainty so important? When test results are used to assess compliance i.e. to decide whether specifications or regulations are met, the measurement uncertainty of the test results has to be taken into account. www.integrallabs.co.za

11. Compliance or non-compliance? Compliance limit <5.0 g/L √ √ X X www.integrallabs.co.za 6.0 5.2 4.8 4.0

12. Compliance or non-compliance? √ √ √ √ X X X Compliance limit X www.integrallabs.co.za 4.0 4.8 5.2 6.0

13. Compliance or non-compliance? Does the regulator use a decision rule? √ √ X X √ Compliance limit Upper limit for a decision rule www.integrallabs.co.za 4.0 4.8 5.2 6.0 5.3

14. Normal distribution probabilities www.integrallabs.co.za It is good to know the standard deviation, because we can say that any value is: likely to be within 1 s.d. (68 out of 100 should be) very likely to be within 2 s.d. (95 out of 100 should be) almost certainly within 3 s.d. (997 out of 1000 should be)

15. The basic statistical analysis used to calculate uncertainty Depends on the approach used … Type A evaluation of uncertainty is by a statistical calculation from a series of repeated observations of measurement process that is UNDER CONTROL. The statistically estimated standard deviation of those observations is called a Type A standard uncertainty Type B evaluation of uncertainty is by means other than that used for Type A. For example, information about the sources of uncertainty may come from data in calibration certificates, from previous measurement data, from experience with the behaviour of the instruments, from manufacturers’ specifications and from all other relevant information. www.integrallabs.co.za

16. How many readings are needed? For Type A, there is no minimum since your coverage factor is determined by the DOF and confidence level. The smaller the “n” the larger the coverage factor and vice versa …WHY? For a normal distribution, the coverage factor is typically equal to 1, 1.96, 2, or 3, with a level of confidence of, respectively, 68.27%, 95%, 95.45%, or 99.73% These values are only strictly valid when the number N of replications of measurements is high. Type A determination – your coverage factor can be looked up – dependant on number of measurements (n and D.O.F.) NB …. SEE NEXT PICTURE www.integrallabs.co.za

17. How many readings are needed? - Student t Distribution "Student" is W. S. Gossett, who was an employee of Guinness Breweries and who first published these values under the pseudonym "A. Student" in 1908. www.integrallabs.co.za

18. How do we know if our estimation of uncertainty is correct? www.integrallabs.co.za

19. Does measurement level impact on uncertainty? www.integrallabs.co.za

20. Examples -Actual analysis duplicate data approach – Type A Calcium as Ca mg/L Date AnalystLab NoDuplicate 1Duplicate 2 14.05.20141JvSPW-14-0613232.7032.90 19.05.20142JvSPW-14-0622915.4015.70 22.05.20143JvSmulti 109.949.93 10.06.20144JvSPW-14-069711.401.29 17.06.20145JvSPW-14-070811.19 03.07.20146JvSKW-14-020028.638.33 09.07.20147JvSPW-14-080068.888.90 8JvSPW-14-080132.482.45 9JvSPW-14-0806514.4014.50 23.08.201410JvSPW-14-0926820.5020.70 11JvSPW-14-0927224.7024.80 25.08.201412JvSPW-14-092124.774.70 13JvSPW-14-0934321.8020.90 27.08.201414JvSPW-14-0960311.90 15JvSPW-14-096471.841.76 16JvSPW-14-096817.747.57 01.09.201417JvSPW-14-0979514.6014.50 18JvSPW-14-099001.151.16 05.09.201419JvSPW-14-101123.443.07 20JvSPW-14-102975.735.55 Calcium as Ca mg/L 2n mean C SD (duplicates) = [sqrt(VAR)] RSD (SD/mean) RSD% 2.02 x RSD% Uncertainty (upper) at conc ave Uncertainty (lower) at conc ave Uncertainty for Calcium as Ca mg/L www.integrallabs.co.za

21. Examples of calculation methods Budget approach – Type B www.integrallabs.co.za

22. What units should be used to express uncertainty? Reducing Sugar 2.73 g/L, U = 0.32g/L (Expanded Uncertainty with k = 2 at 95% confidence level) Or Reducing Sugar 2.73 g/L, U = 11.7% (Expanded Uncertainty with k = 2 at 95% confidence level) In other words, we have a high degree of confidence (95%CI) that our answer falls somewhere in the range 2.57 -2.89g/L www.integrallabs.co.za

23. Last slide - Reasonability In the end, every uncertainty estimate should be subjected to a reasonability check. The analyst should ask questions such as “Is this estimate in line with what I know about the nature of the measurement and of the material?” eg is U of 0.001% for EtOH reasonable? “Can this estimate be supported with proficiency testing data, or data accumulated as part of a measurement assurance program?” Uncertainty estimates that look strange – either too big or too small -- should be re-evaluated – look first for mathematical blunders, – second for uncertainty contributors whose magnitudes may have been poorly estimated or neglected. – Finally, it may be necessary to revise the mathematical model. Human judgment based on sound technical experience and professional integrity is of paramount importance in evaluating uncertainty. Look at your QC data from a standard Shewart chart www.integrallabs.co.za

24. Thanks for Listening Any Questions? Andrew Pascall Integral Laboratories (Pty) Ltd andrew@integrallabs.co.za www.integrallabs.co.za