1. Measurement of Uncertainty – One Lab’s Experience Patricia Hanson Biological Administrator I Florida Department of Agriculture and Consumer Services, Food Safety, Microbiology Laboratory

2. What is Uncertainty Quality of a Measurement “Give or Take” Quantification of Doubt

3. Sources of Uncertainty Analyst Experience and Skill Level Equipment Stability Environmental Factors Integrity and Composition of the Sample Process of Taking Measurements and Interpreting Data Random Error

4. Error vs. Uncertainty Error is the difference between the measured value and the true value. An example of error is a correction factor that is applied to a thermometer. Errors that are corrected for are not included in Measurement of Uncertainty.

5. Error vs. Uncertainty Uncertainty is a measure of errors that can not be corrected. These may be unknown or known. An example of uncertainty would be two competent analysts independently analyze the same sample by the same method but come up with different numbers.

6. Mistakes and Failures Mistake: method is not followed as written Failure: a component of the method such as a reagent or a piece of equipment does not perform as expected Data associated with known mistakes or failures should be omitted from the Measurement of Uncertainty

7. Case Study #1 An Unforeseen Source of Variation

8. An Unforeseen Source of Variation Two different technicians ran the water membrane filtration method for coliform enumeration. They switched off every few weeks. They used the same process control. There was a notable shift in the process control chart when analysts switched off. The technicians were observed by a senior analyst to both have acceptable pipetting techniques. Both technicians consistently passed their Proficiency Tests.

9. Why the Variation? The obvious source of variation was the technicians. While there were not identifiable errors or mistakes, there was a level of uncertainly in how each technician prepared the serial dilution of the process control. It was amplified because of the number of serial dilutions that needed to be made to dilute the overnight culture to get it down to the countable range.

10. Resolution Since the dilution steps were not performed on the samples, the variation introduced from the serial dilution of the control should not be part of the Measurement of Uncertainty for the method. A different means of achieving the correct count in the process control had to be found. We chose to use pellets with a known count.

11. Case Study #2 A Missing Source of Variation

12. Missing Sources of Variation The lab had just validated a new method for Staphylococci enumeration. There were not enough data points from the validation to establish the control chart. Each of the four trained analysts ran several pellets to obtain 15 additional data points over two days. The control chart based on these data points had unusually tight limits.

13. Where was the Variation? In this case, not all the sources of uncertainty were taken into account. The data collection process was repeated with each of the same four analysts running process controls over a six week period. The Standard Deviation of the 15 points set up within two days was ½ the Standard Deviation of the 15 points that were set up over 6 weeks. Sources of Uncertainty that are introduced by running a procedure over a period of time were being missed.

14. Lessons Learned from Case Studies Do not introduce extra sources of variation Do include all sources of variation Samples used for MU calculation should, as closely as possible, follow the same procedure as your samples  Include Matrix  Different Analysts  Different Equipment  Different Days  Appropriate Analyte Level  Follow Method as Closely as Possible

15. Where to get data for MU Process Control Samples* Proficiency Test Samples Duplicate Samples

16. Microbiology Enumeration Method Challenges The sample matrix needs to be free of target Solution: Use a sterile surrogate matrix Limitation: Sterile samples/surrogate matrices do not have background microflora that are present in samples

17. Proficiency Test Samples for MU Proficiency Test (PT) samples can be used to determine Measurement of Uncertainty Use of PT samples typically results in a small data set, even if PTs are run monthly, there are only 12 points per year PT samples often come pre-weighed or in pellet form and do not go through the same steps as the samples

18. Process Control Samples for MU The process control is run through the entire method by each analyst on different days so most of the variability from the method is captured Lots of data points are collected This data typically is already being collected in the laboratory so it is readily available for Measurement of Uncertainty Calculations

19. Challenges of using Process Control Pellets The “claimed” count per pellet must be verified in house This normally differs from the “claimed” count on the vial The test method can not be used to determine the count because of the uncertainly of the test method Use of non-selective agar for counts Count several pellets and use an average, discard outlying counts

20. Percent Recovery Data Fairly straight forward once you have an established count (spike value) for your inoculated organism (log 10 T / log 10 S) x 100 where T = test value (result from your PC) S = spike value (count from your pellet)

21. Determine MU from Percent Recovery Data Take the Standard Deviation (SD) of all Process Control Percent Recovery Data Points. Multiply by the coverage factor, k  for samples sets with 30 or more data points k=2 for a 95% confidence level  for sample sets with fewer than 30 data points there is a table: “t-statistic for 95% confidence” that will give you the value for k for your data set  The fewer numbers in your data set, the higher the value for k because fewer data points may not capture all of the uncertainty

22. Recovery Replicates Data Advantage: no need to determine the “absolute” count to determine percent recovery Disadvantage: cost and time of additional replicates Caution: should be two separate samples taken through the procedure, not one sample that is just analyzed in duplicate

23. Microbiology Qualitative Method Challenges Far fewer challenges and limitations than enumeration methods when it comes to picking a matrix The sample matrix does not need to be free of target Either a sterile surrogate matrix or a spiked sample can be used The challenges come when determining a meaningful Measurement of Uncertainty

24. Microbiology Semi-Quantitative Methods What we do  Use false positive and false negative rates  False positive rate is based on number of un- spiked samples that screen positive but are non-culturable vs number of samples that screen positive and are confirmed.  False negatives are based on number of PT and Process Control failures

25. Microbiology Semi-Quantitative Methods Why we do it  Provides the most useful information about our method  False positive rate takes into consideration sample composition – matrix and background microflora challenges to confirmation

26. Microbiology Semi-Quantitative Methods What we don’t do and why  We do not use kit controls as these don’t take into account sample composition and enrichment procedure.  We do not use OD or Ct values. These are useful for tracking trending in a method. For our purpose, as long as a sample is beyond the established cut off it is considered a positive screen. How far from the cutoff the sample is does not have an impact on if the lab pursues confirmation.

27. Food Chemistry Methods Food Chemistry is more straight forward Can use spikes or known positive samples

28. When to use a Combined Uncertainty When your control sample does not go through all the steps of your method When you can independently measure the uncertainty of each step Combined Uncertainty is calculated by root sum of squares of each component

29. What Methods we do not have a MU for Cultural qualitative methods Rapid qualitative methods where there is only a detection/non-detection result with no confirmation step Cultural methods that use rapid or semi- quantitative methods as a step in the method but that step does not factor into the final result

30. Practical Uses for Method MU’s Method Evaluation PT Evaluation Analyst Evaluation Regulatory Determination

31. References Guidelines for Estimating Uncertainty for Microbiological Counting Methods. American Association for Laboratory Accreditation, September 3, 2014. Hammack, Stacie. Measurement of Uncertainty. Bureau of Food Laboratories Laboratory Quality Management System FL QA 125, Version 4.3. Bureau of Food Laboratories, Division of Food Safety, Florida Department of Agriculture and Consumer Services, February 6, 2015. Bell, Stephanie. A Beginner’s Guide to Uncertainty of Measurement. Measurement Good Practice Guide No 11, Issue 2. National Physical Laboratory, August 1999.

32. Questions

33. Group Activity Determine how you would compute the Measurement of Uncertainty for the following method:  Standard Total Coliform Fermentation Technique. Standard Methods For the Examination of Water and Wastewater, 22 nd Edition, 9221 B.

34. Procedure Sample: 100ml vended drinking water Dispense 10ml sample into each of ten 2x Lauryl Tryptose Broth LBT tubes, incubate and check for growth and gas Transfer one loopful from each positive LTB tube to one Brilliant Green Lactose Bile (BGLB) broth tube and one loopful to one EC Broth tube, incubate and check for growth and gas

35. Interpretation of Results Gas and Growth in LST and BGLG = confirmed coliform tube Gas and Growth in LST, BGLB, and EC = confirmed fecal coliform tube Look up number of positive tubes on 10 tube MPN table to determine result

36. 10 Tube MPN Table # Positive TubesMPN/100ml 0<1.1 11.1 22.2 33.6 45.1 56.9 69.2 712 816 923 10>23

37. Available Process Control Organisms Overnight Culture of Escherichia coli: approximately 2 x 10 9 cfu/ml (positive in all three broths) Overnight Culture of Enterobacter aerogenes: approximately 2 x 10 9 cfu/ml (positive in LTB and BGLB broths) Microbiologics ® Pellet of Escherichia coli: approximately 2 x 10 3 cfu/pellet (positive in all three broths) Microbiologics ® Pellet of Enterobacter aerogenes: approximately 2 x 10 3 cfu/pellet (positive in LTB and BGLB broths)

38. Points to Consider What steps in the method have the potential for uncertainty? Can you take a process control through all of these steps or do you need to do a combined uncertainty? Can you minimize the potential for adding uncertainty to the process control that is not found in the samples? Can you construct a process control that is in the same countable range for the method (1.1 to 23 cfu)?