1. Is it Method Validation, Verification or or Just Semantics?
Michael Brodsky
Brodsky Consultants

2. Objectives Distinguish method validation from verification
Consider the requirements for method validation and in-house verification from an ISO Standard perspective
Review AOACI OMA and RI SLV and Multi-Lab Collaborative Study validation protocols as applicable to “in-house” validation
Outline the requirements for a verification protocol that defines a method’s fitness-for purpose

3. What is Test Method Validation?
Validation is the establishment of one or more performance characteristics for a test method
By single laboratory validation (SLV)
By multi-laboratory collaborative study
Always by comparative analysis to a “Reference” Method

4. Verification Demonstration of Analytical competency.
Performance characteristics of the method, when performed, as prescribed, by laboratory analysts, conforms to the performance characteristics of the method as established during the validation study.
Ensures that the method is fit for its intended purpose
Includes determination of uncertainty of measurement (quantitative methods).

5. Performance Characteristics of Microbiological Methods
Relative Accuracy (%Recovery)
Precision (Repeatability and Reproducibility)
Specificity (Selectivity for target analyte)
Sensitivity (Distinguishing target from non target)
Inclusivity (Range of target analytes detected by method)
Exclusivity (Range of non-target analytes excluded)
False Positive and False Negative Rates
Limits of Detection (LOD)
Limit of Quantitation (LOQ)
Scope of application

6. Categories of Microbiological Test Methods
Performance Characteristics Included in a Validation Study
Performance Characteristic
Identification
Quantitative
Qualitative
(P/A)
Relative Accuracy No
Yes
Matrix Effects (Scope)
Precision
Sensitivity
Specificity
Inclusivity
Exclusivity
False Positive Rate
False Negative Rate
LOD
LOQ
Ruggedness
Linearity/Range

7. Performance Characteristic
Categories of Microbiological Test Methods
Performance Characteristics Included in the Verification of a Validated Method
Performance Characteristic
Identification
Quantitative
Qualitative
(P/A)
Verification
(Where Applicable)
Relative Accuracy No
Yes
Matrix Effects (Scope)
Precision
Sensitivity
Specificity
Inclusivity
Exclusivity
False Positive Rate
False Negative Rate
LOD
LOQ
Ruggedness
Linearity/Range

8. 17025 Requirements Methods requiring validation are:
Modified official methods
In-house developed methods
Previously validated methods extended to a component, analyte or matrix not previously tested or included in the validation study

9. Food Categories and Types

11. Food Categories and Types (Based on Physiochemical and Innate Microbial Characteristics)

12. Food Categories/Matrices
Raw milk and dairy products
Heat processed milk and dairy products
Ready-to-eat, ready-to-reheat meat products
Raw poultry and ready-to-cook poultry products
Ready-to-eat, ready-to-reheat meat poultry products
Eggs and derivatives
Raw and ready-to-cook fish and seafoods (unprocessed)
Ready-to-eat, ready-to-reheat fishery products
Fresh produce and fruit
Processed fruits and vegetables
Infant formula and infant cereals
Dried cereals, fruits, nuts, seeds and vegetables
Chocolate, bakery products and confectionary
Multi-component foods or meal components
Pet food and animal feed
Environmental samples (food or feed production)

14. Food Categories and Types
9. Fresh produce and fruit
Types:
Cut ready-to-eat fruit
Cut ready-to-eat vegetables
Produce grown in or in contact with the ground
Sprouts
Raw fruit/vegetable juices (unpasteurized)
Leafy greens
Vegetables and fruits (unprocessed) not described above

15. Food Categories and Types
15. Pet food and animal feed
Types:
Animal origin ingredients
Plant origin ingredients
Other ingredients
Dry food (aw ≤ 0.7)
Wet food (aw > 0.7)
Canned
Animal feeds (bovine, ovine, pig)
Animal feeds (poultry)
Animal feeds (fish)

16. AOACI Pre-Collaborative/PTM Study Design (SLV)
For Each Food Category to be claimed:
≥2 food types/category
If claiming if applicable to a broad range of foods:
Test ~20 samples from 9 different Food categories

17. AOACI Pre-Collaborative/PTM Study Design (SLV) (Quantitative Method)*
20 analyses for each food type:
5 portions of each
Low, medium and high levels of contamination and uncontaminated
Comparison of recovery to reference method using ANOVA and comparison of means
*Use as guide for “in-house’ validation

18. AOACI Pre-Collaborative/PTM Study Design (SLV) (Qualitative Method)*
30 analyses for each food type:
5 replicate test portions per level the high inoculation level,
20 for the fractional positive level
5 for the uncontaminated level
Comparison to reference method using Chi-square analysis &/or Probability of Detection (POD models)
*Use as guide for “in-house’ validation

19. AOACI Collaborative Study – Quantitative
laboratories
Minimum of 8 labs with acceptable data
Test samples*
1 Matrix
Low, medium and high contamination +uncontaminated
Five samples tested at each level
Both naturally and artificially contaminated food
Compare against a reference method
Repeatability, Reproducibility and differences between means
*Use as guide for “in-house’ validation

20. AOACI Collaborative Study (Qualitative Method)
laboratories
Minimum of 10 labs with acceptable data
Test samples*
1 matrix
12 test portions per high analyte level
12 test portions per fractional Positive samples
12 uncontaminated test portions
*Use as an alternate guide for “in-house’ validation
Comparison to reference method using Chi-square analysis &/or Probability of Detection (POD models)

21. "I think you should be more explicit 
here in step two."

22. 17025 Requirements
Official reference methods already published and intended for a specific matrix will be incorporated into the current method document format.
Do not need to be fully validated.
The ability to perform the analysis must be verified using spiked samples, proficiency samples or CRMs

25. Verification of Quantitative Microbiological Methods
Precision (Repeatability and Reproducibility)
Uncertainty of Measurement
Fitness-for-Purpose

26. Validation Data from Collaboratively Studied Methods
Provides reference values for RSDR and RSDr
e.g. Pour Plate counting (SMEDP)
e.g. RSDr ≤ 7.7% (0.077) (within analysts)
e.g. RSDR ≤ 18.2% (0.182) (between analysts)

27. Validation Data from Collaboratively Studied Methods
Can also be used to estimate uncertainty of measurement
e.g. Pour Plate counting (SMEDP)
e.g. RSDr ≤ 7.7% (0.077) (within analysts)
e.g. RSDR ≤ 18.2% (0.182) (between analysts)

28. Calculation of Combined Uncertainty
Calculate the combined uncertainty (Uc) using standard propagation of error rules (the square root of the sums of squares of SDs known as the “root sum of squares” - RSS).

29. Validation Data from Collaboratively Studied Methods
Calculation of Combined Uncertainty (Uc):
Root Sum of Squares: √(RSDr)2 + (RSDR)2
For Pour Plate (HPC)
Sum of Squares: (0.077)2 + (0.182)2 =0.0371
Uc = √(0.0371) = 0.193=19.3%
Expanded uncertainty (Ue): (Use coverage factor k=2* for 95% confidence) = 2 x 19.3% = 38.6%
*≥30 Observations

30. Verification of Qualitative Microbiological Methods
Specificity
Sensitivity
Fitness-for-Purpose

31. How Many Samples Are Needed for Verification?

32. How Many Samples Are Needed for Verification?
No fixed number
But, a minimum of 15 positive samples run in duplicate (30 observations) per matrix is not unreasonable
10 is the minimum requirement

33. Who’s Doing What
ISO: Microbiology of food and animal feed – Method Verification– Part 4: Protocol for the verification of reference and alternative methods implemented in a single laboratory
HC/CFIA: Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing

34. ISO Protocol for the verification of reference and alternative methods implemented in a single laboratory (under review )
Minimum of 10 artificially inoculated samples
Use CRM, if available
For Qualitative methods inoculate with 1-5 CFU per test portion.  
For Quantitative methods, the levels of contamination shall cover the range of the method
Analyse samples on 10 different occasions or days
be performed by at least 2 technicians working independently and with separate samples and reagents.

35. How to Interpret Verification?
For Quantitative methods, ISO proposes to accept:
participation in interlaboratory comparisons such as proficiency testing and plotting z-scores to show any trends;
use of microbiological RM or CRM;
recovery experiments with spiked samples using a non-selective method.

36. For Qualitative Methods,
Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing by Health Canada (April 2015)
For Qualitative Methods,
Analyze 3-5 samples spiked at 3-5 times the reported or determined LOD (1-3 CFU/analytical unit)
For Quantitative Methods
using selective medium, measure repeatability by analyzing 10 or more replicate samples of a representative food matrix naturally or artificially contaminated

37. How to Interpret Verification?
For Qualitative methods, ISO and HC agree: 100% (comparative) sensitivity All samples must be correctly identified
For Quantitative methods, for acceptable precision as suggested by HC, the repeatability (r) must be less than half of the reproducibility (R) data. 2r < R. If R value if published.
If there is no published performance characteristic data, ???
Calculate Ue

38. 17025 Requirements Fitness-for-Purpose
The data obtained from the method approval process must show it is fit for the intended use and relevant to customers’ needs.
Does the method have performance characteristics that meets the expectation of the laboratory and the needs of the client?

39. References
AOAC International Methods Committee Guidelines For Validation Of Qualitative And Quantitative Food Microbiological Official Methods Of Analysis, Appendix J.
The Fitness for Purpose of Analytical Methods, A Laboratory Guide to Method Validation and Related Topics, EURACHEM Working Group
How to Meet ISO Requirements for Method Verification, 2015, AOAC Technical Division for Laboratory Management (TDLM), the Analytical Laboratory Accreditation Criteria Committee (ALACC)
ISO/FDIS Microbiology of the food chain — Method validation — Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method (ISO/TC 34/SC 9) (Draft)

40. Thank you for your attention
Any Questions?