1. Validation of Screening Methods
Aisling Treacy, MSc.
Prof. Tom Buckley, MSc, FIBMS, FAMLS

2. Background Established in 1984 – Equine diagnostic laboratory
Screening laboratory during the “Angel Dust Era”
Antibiotic residue testing for the porcine industry where there was approx. 25% non-compliance → currently < 1% non-compliance
Screening laboratory for the NRCP → obtained ISO Accreditation for NRCP tests in 2005 → Currently accredited for 94 analyte / matrix / species combinations
Rapid screening laboratory for Phenylbutazone (PBZ) during the horse meat scandal – 2013
Self Monitoring Plan
Equine anti-doping testing → testing other species

3. Pre-validation criteria to be considered
Analyte(s) – regulatory limit – MRL, MRPL, RPA
Selection of the method – LOD, cross-reactivity, single / multi-analyte method, reliability, matrix
Client requirements
Review of relevant regulatory or guidance documents
Commission Decision EC/2002/657
CRL Guidance Paper of 7 December 2007
Community Reference Laboratories Residues (CRLs) 20/1/2010
ISO/IEC 17025:2005 General requirements for the competence of testing and calibration laboratories

4. Pre-validation work
Test performance – recovery of spikes (various levels), reproducible results
Matrix / species effects
Sample treatment / optimum extraction

5. Study design / Validation plan
Analyte, matrix and species to be validated
CCβ, Screening Target Concentration → validation spiking level
Number of samples to be analysed
Factors to be included in ruggedness
Specificity testing with related analytes
Applicability
Stability testing
Repeatability / method verification
Initial validation
Further validation

6. Performance characteristics to be determined according to EC/2002/657
Semi-quantitative screening:
CCβ
Precision
Selectivity / specificity
Applicability
Ruggedness
Stability

7. CCβ Detection capability
Detection capability CCβ – is defined as the smallest content of the analyte that may be detected, identified and/or quantified in a sample with an error probability of β
The β error is the probability that the tested sample is truly non-compliant even though a compliant measurement has been obtained
For screening tests the β error (i.e. false compliant rate) should be <5%
For analytes with a regulatory limit, CCβ must be ≤ the regulatory limit
For analytes with no regulatory limit, CCβ must be as low as possible
CCβ must be established for analytes across individual matrices
LOD:
A consideration when selecting a method
CCβ and threshold values to describe the detection capability of the method

8. Determination of number of samples required for validation and screening target concentration
The STC in relation to the regulatory limit will determine the number of samples required for validation
Spiking Level CCβ / Screening Target Concentration
Number of Samples required for Validation
No. of acceptable False Compliant Results
≤ 50 % of the MRL 20
≤ 1
≥ 50 % and ≤ 90 % of the MRL 40
≤ 2
≥ 90 % and ≤ 100 % of the MRL, MRPL, RPA, analytes included in CRL Guidance Paper of 7 Dec 2007 and analytes for which there is no guideline 60
≤ 3

9. Implementation of validation
Determination of STC and number of samples required for validation
Analyses carried out on 3 different days by different analysts

10. Inclusion of other species and matrices / applicability
Separate validation is carried out for each individual matrix type, e.g. serum and muscle
If multiple species are to be validated within one matrix type, then the 60 samples could comprise different species, e.g. 20 x bovine, 20 x porcine and 20 x ovine
If an additional species is to be included after validation is complete, 20 spiked and 20 blank samples made up of the additional species may be analysed and added to the initial validation for the calculation of threshold value and cut-off value

11. Test specificity
Testing of blank samples to determine threshold will demonstrate whether there is matrix interference – calculation of threshold values
Testing of similar analytes – recovery should fall below cut-off value established during validation

12. Calculation of cut-off value
Two approaches to establishing cut-off levels outlined in CRL Guidelines for the Validation of Screening Methods for Residues of Veterinary Medicines 2010:
Approach I – Lowest response
The lowest response in the spiked samples analysed during validation is taken as the cut-off giving a response greater than this level is deemed to be ‘screen positive’ and exceeds the CCβ of the screening method
Approach II – Statistical approach
Cut-off factor (FM) = Mean Response – 1.64 x SD

13. Calculation of cut-off value
The multiplication factor approach to establish a cut-off value
The IEC uses a multiplication factor applied to the mean of two spiked ‘cut-off’ samples analysed alongside routine samples to establish a cut-off value for each test run
A one-tailed 95% confidence interval of the validation data generated from spiked samples is calculated and then divided by the mean (M)
To apply this fraction to the results of routine spiked cut-off samples, this figure is subtracted from 1 to give a multiplication factor to be applied to the mean of the spiked cut-off samples
Multiplication factor = 1 - (1.64 x Std. Dev.) M
In routine testing, any sample with a reading at or above the calculated ‘cut-off applied’ will be considered ‘screen positive’ or non-compliant and forwarded for confirmatory analysis
This approach to cut-off calculation has been approved by regulatory authorities and INAB auditors

14. Validation data generated from analysis of three related analytes
Hex serum
Des serum
Den serum
Name
STI22AU1
STI22AU2
STI22AU3
STI18JU5
STI18JU3
STI18JU4
VAL1 1
2.15
1.84
1.9
1.31
1.58
1.89
DEN 1
0.99
0.69
1.08
VAL1 2
2.21
2.02
1.48
2.18
2.55
DEN 2
0.78
0.87
1.15
VAL1 3
1.95
1.67
1.63
0.89
DEN 3
0.34
0.66
1.04
VAL1 4
1.62
2.17
2.03
1.07
1.24
1.17
DEN 4
0.57
0.51
1.06
VAL1 5
2.13
1.87
2.14
1.53
1.19
0.81
DEN 5
1.05
0.82
VAL1 6
2.09
1.57
2.35
0.97
1.38
1.5
DEN 6
0.54
0.83
VAL1 7
2.37
2.07
1.2
1.39
DEN 7
0.5
0.84
0.86
VAL1 8
2.08
1.83
1.97
0.96
0.95
DEN 8
0.44
0.88
VAL1 9
2.1
2.29
2.06
DEN 9 1
VAL1 10
2.6
2.11
0.92
0.85
DEN 10
VAL1 11
2.01
2.3
1.52
DEN 11
0.49
VAL1 12
1.59
1.22
DEN 12
0.38
0.41
0.7
VAL1 13
2.49
1.68
1.88
DEN 13
1.03
VAL1 14
1.75
1.32
DEN 14
1.01
VAL1 15
1.76
1.61
1.12
DEN 15
0.9
1.1
VAL1 16
1.79
0.98
DEN 16
0.42
1.02
VAL1 17
1.74
1.92
DEN 17
VAL1 18
2.24
DEN 18
0.75
VAL1 19
2.27
1.29
1.34
1.25
DEN 19
0.52
VAL1 20
2.59
1.7
1.09
0.94
DEN 20
0.37
0.4
Mean
2.1415
1.9225
1.968
1.1915
1.263
1.28
0.664
0.689
0.934
Std. Dev.
0.1899
Overall mean
Overall std dev
%RSD
Validation Statistic
Approach II
Approach I
Hex Serum
Approach I: 1.38
Approach II:
Mul.Fact.:
Des Serum
Approach I: 0.81
Approach II:
Mul.Fact.:
Den Serum
Approach I: 0.34
Approach II:
Mul.Fact.:

15. Ruggedness
Introduction of minor reasonable variations and observations of consequences
The Youden approach to ruggedness is employed at the IEC – facilitates the introduction of several variations simultaneously
Factors that may influence measurement result are selected:
Pre-treatment
Clean-up, SPE
Analysis
Factors that influence results are subjected to further testing – these factors are described in the validation report and may be included in the SOP

16. Ruggedness parameters investigated
 Factor
 Factor value F
Combinations of determinations number 1 2 3 4 5 6 7 8
 Centrifugation
A/a
Centrifuged vs. Not A a
 Vortex
B/b
2 min vs. 1 min B b
C/c
4000 rpm vs rpm C c
 Evaporation temp.
D/d
25°c vs. 35°c D d
Re-suspension buffer
E/e
IAC WB vs. ddH₂O E e
F/f F f
 Elution buffer
G/g
70% EtOH vs. 80% EtOH G g
 Observed result R S
0.24 T U
0.06 V
0.1 W
0.16 X
0.18 Y
0.12 Z
0.15

17. Ruggedness results Mean results for each parameter
Differences between normal and varied parameters
Formula
Result
AA = Σ(Ai)/4
0.16
AB = Σ(Bi)/4
0.205
AC = Σ(Ci)/4
0.145
AD = Σ(Di)/4
0.1875
AE = Σ(Ei)/4
0.1575
AF = Σ(Fi)/4
0.1625
AG = Σ(Gi)/4
Aa = Σ(ai)/4
0.1525
Ab = Σ(bi)/4
0.1075
Ac = Σ(ci)/4
0.1675
Ad = Σ(di)/4
0.125
Ae = Σ(ei)/4
0.155
Af = Σ(fi)/4
0.15
Ag = Σ(gi)/4
Differences (Di)
Squares of differences (Di2)
Z-Scores (Di2)
Da = A - a =
Da2 = value a
0.00
Db = B - b =
Db2 = value b
0.0095
Dc = C - c =
Dc2 = value c
0.0005
Dd = D - d =
Dd2 = value d
De = E - e =
De2 = value e
Df = F - f =
Df2 = value f
Dg = G - g =
Dg2 = value g
0.0001
Standard Deviation Of The Differences Di (SDi)
SD1 = √2*Σ(D12/7)
Standard deviation of method
Ruggedness Study Conclusion - The only parameter that showed significant difference was elution buffer of the immunoaffinity column procedure. This is controlled in the procedure. This test has proven to be robust against all other changes to the parameters included in the ruggedness testing

18. Stability
The degradation of the analyte under different conditions relevant to the test / laboratory
Observed through ongoing monitoring of QC materials (negatives and spikes) – plotted on Shewhart charts
Stability experiments carried out if any degradation is observed over time
Stability experiments – assessment of stability of analytes:
Stability of analytes in solution
Stability of analytes in matrix
Assessment carried out in line with EC/2002/657 and SOP P

19. Repeatability Assessment of spiked samples over three days
Calculation of Relative Standard Deviation measure of within lab reproducibility  must be < 35%

20. Validation acceptability criteria – Was validation successful?
RSD < 35%
Approach I – No overlap between threshold value and cut-off value
Approach II – The rate of false positive is acceptable (i.e. 5%) when FM > T if B < FM < T the false positive rate is higher than 5%
According to Commission Decision 2002/657/EC, CCβ is validated when FM > B
If the above criteria are met the method is considered robust, specific and fit for purpose

21. Validation data generated from analysis of three related analytes
Hex serum
Des serum
Den serum
Name
STI22AU1
STI22AU2
STI22AU3
STI18JU5
STI18JU3
STI18JU4
VAL1 1
2.15
1.84
1.9
1.31
1.58
1.89
DEN 1
0.99
0.69
1.08
VAL1 2
2.21
2.02
1.48
2.18
2.55
DEN 2
0.78
0.87
1.15
VAL1 3
1.95
1.67
1.63
0.89
DEN 3
0.34
0.66
1.04
VAL1 4
1.62
2.17
2.03
1.07
1.24
1.17
DEN 4
0.57
0.51
1.06
VAL1 5
2.13
1.87
2.14
1.53
1.19
0.81
DEN 5
1.05
0.82
VAL1 6
2.09
1.57
2.35
0.97
1.38
1.5
DEN 6
0.54
0.83
VAL1 7
2.37
2.07
1.2
1.39
DEN 7
0.5
0.84
0.86
VAL1 8
2.08
1.83
1.97
0.96
0.95
DEN 8
0.44
0.88
VAL1 9
2.1
2.29
2.06
DEN 9 1
VAL1 10
2.6
2.11
0.92
0.85
DEN 10
VAL1 11
2.01
2.3
1.52
DEN 11
0.49
VAL1 12
1.59
1.22
DEN 12
0.38
0.41
0.7
VAL1 13
2.49
1.68
1.88
DEN 13
1.03
VAL1 14
1.75
1.32
DEN 14
1.01
VAL1 15
1.76
1.61
1.12
DEN 15
0.9
1.1
VAL1 16
1.79
0.98
DEN 16
0.42
1.02
VAL1 17
1.74
1.92
DEN 17
VAL1 18
2.24
DEN 18
0.75
VAL1 19
2.27
1.29
1.34
1.25
DEN 19
0.52
VAL1 20
2.59
1.7
1.09
0.94
DEN 20
0.37
0.4
Mean
2.1415
1.9225
1.968
1.1915
1.263
1.28
0.664
0.689
0.934
Std. Dev.
0.1899
Overall mean
Overall std dev
%RSD
Validation Statistic
Approach II
Approach I
Hex Serum
%RSD
Des Serum
%RSD
Den Serum
%RSD

22. When is additional or re-validation required?
Additional validation carried out when extra species are introduced or minor changes to an existing method
Re-validation will occur if:
Significant change to the method
Method fails on a continual basis
Change in client requirements, e.g. lower STC

23. Continuous verification
QC samples
Negative control (TQCN)
Positive controls (Cut-off 1 and 2)
Check sample (blind to analyst)
Spiking analyte – the analyte with the poorest recovery (worst case / lowest specificity) is used to spike QC or most relevant analyte according to test
Last 20 sets of QC analyses plotted on Shewhart charts - ≤ 1 outlier per 20 analyses, i.e. ≤ 5%
Verifies overall method performance, analyst accuracy / performance, spike stability, kit performance / stability
PT’s – CRL / State Lab submit samples for PT analysis for each analyte at least once yearly
External providers – Progetto Trieste, FAPAS
In-house
Out of specification results subjected to an in-house investigation procedure

24. Thank You