1. Simplifying Measurement Uncertainties Bill Hirt, Ph.D / February 2016

2. Start the Process Light the fuse (candle) Shed light on the way 2

3. Back to the beginning Accredited CALIBRATION labs report measurements on tools and devices needing regular service … typically including uncertainties for those measurements. Accredited TESTING labs, when requested or when needed to interpret Statements of Compliance, report uncertainties alongside their test measurements. 3

4. Framework for uncertainties Unless calculating MDL’s or LOD’s or LOQ’s … It is presumed that ISO 17025 accredited labs demonstrate the competence to calculate and report GUM uncertainties … estimated at roughly a 95 % confidence. 4

5. When do accredited labs report MU’s ? Every accredited calibration certificate, unless requested otherwise Test reports : When a customer requests it When the uncertainty affects compliance with a specification limit When it is relevant to the validity or application of the result 5

6. What do we mean by MU? Measurement value ± uncertainty (MU) MU usually at 95 % confidence – Why? MU usually reported at k = 2 6

7. 7

8. Where do we find most MUs? On calibration certificates On test reports In footnotes 8

9. Did you know that … A high percentage of ISO 17025 accredited calibration laboratories issue incorrect uncertainties on their certificates ? 9

10. ISO 17025 includes – Calibration and Testing A good percentage of calibration labs do not report MU’s on their certificates Most testing labs do not report MU’s on their test reports And WHY ??? 10

11. First Group Discussion Scenario used in 3-day MU course outlined first Have groups discuss other real-life scenarios where uncertainty can be critical or life-saving Put examples on index cards on the tables 11

12. Uncertainty and Traceability Metrological traceability (not sample traceability) Two platforms of confidence Comparison with hi-quality stds thru chain to SI through an NMI or DI Confidence that GUM unc’s used … and actual measurement error clearly known 12

13. Uncertainty and Proficiency Testing (PT) Calibration PT programs typically request that one or more devices have measurements taken … and Both the measurements and their uncertainties be reported. The Cal PT study issues a report with standardized results including an En 13

14. E n value Determination (Calibration Labs)

15. Interpretation of E n values

16. Accredited Testing Labs and PT Very few ISO 17025 accredited PT programs for testing include a requirement for MU’s Some now request them Many in the future will require them In addition to z-scores, many future reports will include En values too 16

17. 17

18. “The Big Three” in 17025 Measurement Uncertainty Metrological Traceability Proficiency Testing 18

19. Uncertainty and Statements of Compliance High percentage of measurements are made to ensure manufactured or natural materials meet a narrow range of specifications Manufacturers and regulators define specs No measurement is perfect MU may or may not be critical to have CONFIDENCE that specs are being met 19

20. Group discussion - 2 At your tables, discuss specific areas of compliance specs that you are aware of Make a list of at least 6 different groups or types of specs at your table to report later The group will share after your discussion 20

21. Uncertainty and Guard Banding Management of manufacturing or other monitoring to assure material is safely within specifications, including consideration of uncertainty Organization may adjust their spec to capture either more potentially defective samples or allow more acceptable product out the door 21

22. 22

23. 23 Definitions Uncertainty – a property of a measurement result that defines the range of probable values of the measurand Uncertainty budget – the systematic description of uncertainty determinations relevant to specific measurements including ranges plus all factors, assumptions and calculations included (must include both type A and type B factors)

24. 24 Basic Steps in Uncertainty Budgets 1.List ALL potential factors affecting variability in measurements - make table 2.Determine the standard uncertainty for each factor (includes distribution) 3.Perform RSS for all factors to create the combined (standard) uncertainty 4.Multiply by distribution factor (k=2 … or ?)

25. 25 Root Sum Squaring

26. 26 Sample Standard Deviation(s)

27. Calibration MU example Budget for 6 inch CALIPERS iComponent of Uncertainty Uncertainty, U(xi) DistributionDivisorStd Unc, u(xi) 1Standard uncertainty6Normal, 2s2.003uin 2Resolution500Rectangular1.73289uin 3Repeatability297.98Normal, 1s1.00298uin 4Uncompensated error25Rectangular1.7314.4uin 5Temperature difference between instrument and gage blocks 20.1U-Shaped1.4114.2uin 6Temperature variance from 68º F16.08U-Shaped1.4111.4uin 7 combined standard uncertainty, u c 416uin coverage factor, k2 expanded uncertainty, U c 832uin Expanded uncertainty rounded UP to 2 significant figures840uin 27

28. Second Cal MU example Uncertainty Budget for Bench Scale - 150 lb capacity iComponent of Uncertainty Uncertainty, U(xi) DistributionDivisorStd Unc, u(xi) 1Standard uncertainty0.000106Normal, 2s2.000.000lb 2Resolution0.05Rectangular1.730.029lb 3Uncompensated error (Three 50 # weights) 0.0105Rectangular1.730.006lb 4Repeatability0.0217Normal, 1s1.000.02lb combined standard uncertainty, u c 0.0366lb coverage factor, k2 expanded uncertainty, U c 0.0732lb Expanded uncertainty rounded UP to 2 significant figures0.074lb 28

29. Testing MU Very often much more complicated than calibration -- why ?? Many stages in test processes Many error types with different units of measure Many errors not defined and require guess 29

30. 30

31. 31 Table A6.2: Summary of results from collaborative trial of the method and in-house repeatability check

32. 32 Table A6.4: Combined standard uncertainties

33. 33

34. Key current and future factor SAMPLING and sample error factors 34

35. Nitrogen in Forage Budget 35

36. Nitrogen in Dry Feed Budget 36

37. New proposed TABLE Handout shows old / traditional version of the Student’s T Table Back side shows our proposed new version Let’s review its features 37

38. Open quiz for the room Using the new Student’s T table, what is the k- factor … for an MU … at 95% confidence … when the number of repeatability measurements is : 3 ? 5 ? 10 ? 30 ? 38

39. Open quiz for the room - 2 … and the test measurement mean is 100 mg and the combined uncertainty is 5 mg, what is each 95% MU : With 3 repeats With 5 repeats With 10 repeats With 30 repeats 39 For an MU … at 95% confidence … when the number of repeatability measurements is : 3 5 10 30

40. Open quiz for the room - 3 # of Repeat Measurements k-factor Multiplier Uexp (95%) To report 3 5 10 30 ( test measurement mean is 100 mg and the combined uncertainty is 5 mg ) 40

41. Open quiz for the room – 3+ # of Repeat Measurements k-factor Multiplier Uexp (95%) To report 34.3 21.5 5 2.78 13.9 10 2.26 11.3 30 2.0 10 41

42. Example of questionable MU on commercial calibration certificate 42

43. Standard Deviation of the Mean The equation on the left for repeatability / SD becomes  43 Caution – it does NOT replace repeatability SD

44. 44 Control Charts Plots of long-term measurements of a single parameter to note trends or variability Often the main basis for testing uncertainties Main testing lab equivalent of repeatability and/or reproducibility

45. 45

46. 46

47. 47

48. AAFCO example of Control Chart of Matrix Spike 48

49. 49 Basic Steps in Uncertainty Budgets 1.List ALL potential factors affecting variability in measurements - make table 2.Determine the standard uncertainty for each factor (includes distribution) 3.Perform RSS for all factors to create the combined (standard) uncertainty 4.Multiply by distribution factor (k=2 … or ?)

50. 50

51. Group discussion - 3 We want to simplify the process for MU determination We want the fundamentals of uncertainties to be understood We know that may tests may be confusing even as to whether an MU is needed Discuss at your tables and list confusing examples … for our larger discussion 51

52. 52 Fundamental MU Nuggets Key definitions to be able to distinguish: Type A and Type B factors Repeatability vs reproducibility Combined uncertainty vs expanded uncertainty (crucial k-factor to use) In unc budgets, any factor contributing less than 10% to the total can be eliminated

53. 53 Fundamental MU Nuggets - 2 Use at least 7 data points for any standard deviation Try to use at least 30 data points (equivalent to infinite number) in Type A factors Metrological traceability and the traceability chain needs GUM uncertainties Calibration PT studies involve uncertainties and testing PT studies may do so soon (En and z-scores)

54. 54 Key Nuggets - 3 Repeatability/reproducibility studies may not adequately cover the range of test tolerances. You may need a high and low range repeatability study. Be careful not to replace repeatability standard deviation factor with standard error of the mean in a MU budget. Add not replace.

55. Key Nuggets - 4 Control charts can often capture full testing errors, but use caution to be sure ALL significant errors are included Repeatability studies or reproducibility studies can often represent the basis for MU determinations For any test, consider all potential errors but budgets and calculations may NOT be needed 55

56. When may MU budgets and calculations not be needed? If alternative statistics used, eg MDLs, LODs If tests qualitative If test protocols already define testing variability, precision, repeatability This includes 95% confidence determinations for microbial MPN studies 56

57. Remember … No measurement is perfect Measurement errors historically combined as Uncertainties … BUT …. Uncertainties trigger too often – confusion 95% confidence is the basis for uncertainties 57

58. Final Mantra … (not) UNCERTAINTY … (but) … CONFIDENCE RANGE !! 58

59. Thank you 59