1 Validation of Qualitative Microbiological Test Methods NCS Conference Brugge, October 2014 Pieta IJzerman-Boon (MSD) Edwin van den Heuvel (TUe, UMCG/RUG)

2 Contents Introduction Statistical Detection Mechanisms Validation Issues Likelihood-Based Inference Conclusions

3 Introduction Validation parameters Qualitative tests Microbiological guidelines Analytical guideline Accuracy and precisionEP RepeatabilityUSP SpecificityEP/USPICH Detection LimitEP/USPICH RuggednessUSP RobustnessEP/USP Guidelines on validation do not agree

4 Introduction In this presentation we will show an optimal validation strategy: –Compare methods –Two dilutions –Optimal densities for the two dilutions –Required number of samples Optimal validation strategy differs substantially from the guidelines

5 Statistical Detection Mechanisms Suppose a test sample is tested with a qualitative test The sample contains X organisms –X=0: sample is sterile –X>0: sample is contaminated The outcome of the test is Z –Z=1: positive test result –Z=0: negative test result

6 Statistical Detection Mechanisms Classification of test result So we need to look at the conditional probabilities The function describing this detection probability is referred to as the detection mechanism Okay False Positive False Negative Okay Number of Organisms X=0X>0 Test Result Z=0 Z=1

7 Statistical Detection Mechanisms Zero-deflated binomial mechanism: –  is the false positive rate:  (0)=  –p is the detection proportion: if  =0 then it is the probability to detect just one organism:  (1)=p –If  =0 and p=1 the test method is perfect –  and p are related to specificity and accuracy –The binomial mechanism (  =0) was introduced in Van den Heuvel and IJzerman-Boon (2013)

8 Statistical Detection Mechanisms  =0, p=0.85  =0.10, p=0.70

9 Validation Issues Estimate detection mechanism via experiments –Exact low spikes of X cannot be generated –Hence the detection probability  (x) cannot be estimated, only the average proportion over samples Expected proportion of positive test results: –Assume that the number of organisms X ~ Poi(

10 Validation Issues The detection proportion p cannot be estimated –Without knowledge on the average number of organisms in the test samples –With serial dilution experiments The false positive rate  can always be estimated using samples from a blank dilution ( =0)  Compare alternate with compendial method –Using the same for both methods –Likelihood ratio test (LRT)

11 Likelihood-Based Inference Experimental Design Suppose we test samples from the same dilution with two methods –Alternate method: i=1 –Compendial method: i=2 –Dilution has on average organisms per sample –Number of samples tested per method: n Expected proportion of positive results now depends on method i (i=1,2):

12 Likelihood-Based Inference Experimental Design Asymptotic distribution of LRT for comparing these proportions converges to -distribution with Hence, power can be optimized by maximizing –Bacterial density can be optimized independently from sample size n –There is a single optimal density

13 Likelihood-Based Inference Experimental Design Compendial:  2 =0.01 p 2 =0.95

14 Likelihood-Based Inference Simulations Simulation Results: Single dilution Average density Detection proportions p AL =0.7 and p CM =1 Power (%) of likelihood ratio test LRT for differences in detection probabilities for various false positive rates  AL =  CM =0  AL =0.05,  CM =0  AL =  CM =0.05 n=150n=200n=250n=150n=200n=250n=150n=200n=

15 Conclusions  Optimal strategy when parameters are unknown –Compare alternate with compendial method –Two dilutions are needed 1. Blank dilution 2. Dilution with on average ~2 organisms –Sample size should be at least n=200 False positive rates can be tested with LRT Accuracy p AL /p CM can be tested with appropriate CIs as an alternative for the LRT for the ratio p AL /p CM (IJzerman-Boon and Van den Heuvel, 2014)

16 Conclusions Differences with guidelines –Only specificity and accuracy need to be considered –Two dilutions are needed, using five 10-fold dilutions is a loss of power –The optimal density is ~2 CFU/unit, ~5 CFU/unit is much too high –Use 200 instead of 5 samples per method and dilution to detect a 30% drop in accuracy with 80% power

17 References IJzerman-Boon PC, Van den Heuvel ER, Validation of Qualitative Microbiological Test Methods, Submitted, Van den Heuvel ER, IJzerman-Boon PC, A Comparison of Test Statistics for the Recovery of Rapid Growth-Based Enumeration Tests, Pharmaceutical Statistics, 2013; 12(5): EP Alternative Methods for Control of Microbiological Quality USP Validation of Alternative Microbiological Methods ICH Q2 (R1) Validation of Analytical Procedures