Is it Method Validation, Verification or or Just Semantics? Michael Brodsky Brodsky Consultants mhbrodsky@rogers.com

Objectives Distinguish method validation from verification Consider the requirements for method validation and in-house verification from an ISO Standard 17025 perspective Review AOACI OMA and RI SLV and Multi-Lab Collaborative Study validation protocols as applicable to “in-house” validation Outline the requirements for a verification protocol that defines a method’s fitness-for purpose

What is Test Method Validation? Validation is the establishment of one or more performance characteristics for a test method By single laboratory validation (SLV) By multi-laboratory collaborative study Always by comparative analysis to a “Reference” Method

Verification Demonstration of Analytical competency. Performance characteristics of the method, when performed, as prescribed, by laboratory analysts, conforms to the performance characteristics of the method as established during the validation study. Ensures that the method is fit for its intended purpose Includes determination of uncertainty of measurement (quantitative methods).

Performance Characteristics of Microbiological Methods Relative Accuracy (%Recovery) Precision (Repeatability and Reproducibility) Specificity (Selectivity for target analyte) Sensitivity (Distinguishing target from non target) Inclusivity (Range of target analytes detected by method) Exclusivity (Range of non-target analytes excluded) False Positive and False Negative Rates Limits of Detection (LOD) Limit of Quantitation (LOQ) Scope of application

Categories of Microbiological Test Methods Performance Characteristics Included in a Validation Study Performance Characteristic Identification Quantitative Qualitative (P/A) Relative Accuracy No Yes Matrix Effects (Scope) Precision Sensitivity Specificity Inclusivity Exclusivity False Positive Rate False Negative Rate LOD LOQ Ruggedness Linearity/Range

Performance Characteristic Categories of Microbiological Test Methods Performance Characteristics Included in the Verification of a Validated Method Performance Characteristic Identification Quantitative Qualitative (P/A) Verification (Where Applicable) Relative Accuracy No Yes Matrix Effects (Scope) Precision Sensitivity Specificity Inclusivity Exclusivity False Positive Rate False Negative Rate LOD LOQ Ruggedness Linearity/Range

17025 Requirements Methods requiring validation are: Modified official methods In-house developed methods Previously validated methods extended to a component, analyte or matrix not previously tested or included in the validation study

Food Categories and Types

Food Categories and Types (Based on Physiochemical and Innate Microbial Characteristics)

Food Categories/Matrices Raw milk and dairy products Heat processed milk and dairy products Ready-to-eat, ready-to-reheat meat products Raw poultry and ready-to-cook poultry products Ready-to-eat, ready-to-reheat meat poultry products Eggs and derivatives Raw and ready-to-cook fish and seafoods (unprocessed) Ready-to-eat, ready-to-reheat fishery products Fresh produce and fruit Processed fruits and vegetables Infant formula and infant cereals Dried cereals, fruits, nuts, seeds and vegetables Chocolate, bakery products and confectionary Multi-component foods or meal components Pet food and animal feed Environmental samples (food or feed production)

Food Categories and Types 9. Fresh produce and fruit Types: Cut ready-to-eat fruit Cut ready-to-eat vegetables Produce grown in or in contact with the ground Sprouts Raw fruit/vegetable juices (unpasteurized) Leafy greens Vegetables and fruits (unprocessed) not described above

Food Categories and Types 15. Pet food and animal feed Types: Animal origin ingredients Plant origin ingredients Other ingredients Dry food (aw ≤ 0.7) Wet food (aw > 0.7) Canned Animal feeds (bovine, ovine, pig) Animal feeds (poultry) Animal feeds (fish)

AOACI Pre-Collaborative/PTM Study Design (SLV) For Each Food Category to be claimed: ≥2 food types/category If claiming if applicable to a broad range of foods: Test ~20 samples from 9 different Food categories

AOACI Pre-Collaborative/PTM Study Design (SLV) (Quantitative Method)* 20 analyses for each food type: 5 portions of each matrix @ Low, medium and high levels of contamination and uncontaminated Comparison of recovery to reference method using ANOVA and comparison of means *Use as guide for “in-house’ validation

AOACI Pre-Collaborative/PTM Study Design (SLV) (Qualitative Method)* 30 analyses for each food type: 5 replicate test portions per level the high inoculation level, 20 for the fractional positive level 5 for the uncontaminated level Comparison to reference method using Chi-square analysis &/or Probability of Detection (POD models) *Use as guide for “in-house’ validation

AOACI Collaborative Study – Quantitative 10 - 12 laboratories Minimum of 8 labs with acceptable data Test samples* 1 Matrix Low, medium and high contamination +uncontaminated Five samples tested at each level Both naturally and artificially contaminated food Compare against a reference method Repeatability, Reproducibility and differences between means *Use as guide for “in-house’ validation

AOACI Collaborative Study (Qualitative Method) 12 - 15 laboratories Minimum of 10 labs with acceptable data Test samples* 1 matrix 12 test portions per high analyte level 12 test portions per fractional Positive samples 12 uncontaminated test portions *Use as an alternate guide for “in-house’ validation Comparison to reference method using Chi-square analysis &/or Probability of Detection (POD models)

"I think you should be more explicit  here in step two."

17025 Requirements Official reference methods already published and intended for a specific matrix will be incorporated into the current method document format. Do not need to be fully validated. The ability to perform the analysis must be verified using spiked samples, proficiency samples or CRMs

Verification of Quantitative Microbiological Methods Precision (Repeatability and Reproducibility) Uncertainty of Measurement Fitness-for-Purpose

Validation Data from Collaboratively Studied Methods Provides reference values for RSDR and RSDr e.g. Pour Plate counting (SMEDP) e.g. RSDr ≤ 7.7% (0.077) (within analysts) e.g. RSDR ≤ 18.2% (0.182) (between analysts)

Validation Data from Collaboratively Studied Methods Can also be used to estimate uncertainty of measurement e.g. Pour Plate counting (SMEDP) e.g. RSDr ≤ 7.7% (0.077) (within analysts) e.g. RSDR ≤ 18.2% (0.182) (between analysts)

Calculation of Combined Uncertainty Calculate the combined uncertainty (Uc) using standard propagation of error rules (the square root of the sums of squares of SDs known as the “root sum of squares” - RSS).

Validation Data from Collaboratively Studied Methods Calculation of Combined Uncertainty (Uc): Root Sum of Squares: √(RSDr)2 + (RSDR)2 For Pour Plate (HPC) Sum of Squares: (0.077)2 + (0.182)2 =0.0371 Uc = √(0.0371) = 0.193=19.3% Expanded uncertainty (Ue): (Use coverage factor k=2* for 95% confidence) = 2 x 19.3% = 38.6% *≥30 Observations

Verification of Qualitative Microbiological Methods Specificity Sensitivity Fitness-for-Purpose

How Many Samples Are Needed for Verification?

How Many Samples Are Needed for Verification? No fixed number But, a minimum of 15 positive samples run in duplicate (30 observations) per matrix is not unreasonable 10 is the minimum requirement

Who’s Doing What ISO: Microbiology of food and animal feed – Method Verification– Part 4: Protocol for the verification of reference and alternative methods implemented in a single laboratory HC/CFIA: Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing

ISO Protocol for the verification of reference and alternative methods implemented in a single laboratory (under review ) Minimum of 10 artificially inoculated samples Use CRM, if available For Qualitative methods inoculate with 1-5 CFU per test portion.   For Quantitative methods, the levels of contamination shall cover the range of the method Analyse samples on 10 different occasions or days be performed by at least 2 technicians working independently and with separate samples and reagents.

How to Interpret Verification? For Quantitative methods, ISO proposes to accept: participation in interlaboratory comparisons such as proficiency testing and plotting z-scores to show any trends; use of microbiological RM or CRM; recovery experiments with spiked samples using a non-selective method.

For Qualitative Methods, Part 5: Guidelines to Verify Standard Food Microbiological Methods for Implementation in Routine Testing by Health Canada (April 2015) For Qualitative Methods, Analyze 3-5 samples spiked at 3-5 times the reported or determined LOD (1-3 CFU/analytical unit) For Quantitative Methods using selective medium, measure repeatability by analyzing 10 or more replicate samples of a representative food matrix naturally or artificially contaminated

How to Interpret Verification? For Qualitative methods, ISO and HC agree: 100% (comparative) sensitivity All samples must be correctly identified For Quantitative methods, for acceptable precision as suggested by HC, the repeatability (r) must be less than half of the reproducibility (R) data. 2r < R. If R value if published. If there is no published performance characteristic data, ??? Calculate Ue

17025 Requirements Fitness-for-Purpose The data obtained from the method approval process must show it is fit for the intended use and relevant to customers’ needs. Does the method have performance characteristics that meets the expectation of the laboratory and the needs of the client?

References AOAC International Methods Committee Guidelines For Validation Of Qualitative And Quantitative Food Microbiological Official Methods Of Analysis, Appendix J. The Fitness for Purpose of Analytical Methods, A Laboratory Guide to Method Validation and Related Topics, EURACHEM Working Group How to Meet ISO 17025 Requirements for Method Verification, 2015, AOAC Technical Division for Laboratory Management (TDLM), the Analytical Laboratory Accreditation Criteria Committee (ALACC) ISO/FDIS 16140-2 Microbiology of the food chain — Method validation — Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method (ISO/TC 34/SC 9) 2014-09-05 (Draft)

Thank you for your attention Any Questions? mhbrodsky@rogers.com