Method Validation-where do I start? Dr Geraldine O’Donnell Director of DNA Your logo(s) here

Overview-Where do I start? Why, when, when not and how to validate Instrumental methods (qualitative)-performance parameters Instrumental methods (quantitative)-performance parameters Template for validation plan/report Touch on human based method validation

Why Validate? During the process of the introduction or implementation of a new method a specific step must be taken to prove in an objective way that the method is suitable for its intended use. This step is called validation Demand for validated methods has been driven by customers, accreditation bodies (i.e. as part of ISO accreditation requirement), and forensic community. an integral part in the development and implementation process of a new method

When to Validate non-standard methods, ISO/IEC 17025 5.4.5.2 non-standard methods, laboratory-designed/developed methods standard methods used outside their intended scope amplifications and modifications of standard methods

When not to validate for standardised methods such as ISO, ASTM a full validation is not necessary need to verify the in-house performance of the method as detailed in ISO/IEC 17025 5.4.2 the laboratory shall confirm that it can properly operate standard methods before introducing the tests or calibrations called verification according to VIM

How to validate Make decision that initial method development is finished Often not possible to determine exactly where method development finishes and validation begins Many of the method performance parameters that are associated with method validation are in fact usually evaluated, at least somewhat, as part of method development. Document the measurement procedure (SOP)

How to validate Part A- Initiate Project A designated person is appointed to draw up the validation plan Plan should contain the following four elements: 1. The laboratory and customer requirements 2. The performance parameters, that will need to be used to ensure that the outputs meet the laboratory and customer requirements - parameters will be dependent on the technique or process under consideration, but should in general address, as appropriate:

Sampling Precision Repeatability Within-lab reproducibility Bias Matrix/substrate effects Specificity Working range Limit of detection/sensitivity Linearity Robustness Environmental susceptibility Competency of personnel Eurachem guide: Terminology in analytical measurement – Introduction to VIM 3 (2011) available from www.eurachem.org.

How to validate Part A- Initiate Project 3.The Acceptance Criteria to be used to assess whether the performance parameters have been met Note: It is critical to the success of the validation that the acceptance criteria are set as specific as possible prior to the commencement of the validation work. 4. The design of the validation tests should also be considered at this stage to ensure that they are as objective as possible. validation plan will be reviewed by the operational manager/quality manager

Start the validation work How to validate Part B Start the validation work tests performed should be those specified in the validation plan. New or additional tests should not be introduced, or planned tests not performed, unless authorised by the operational manager for the designated area. Prepare implementation plan designated person has to consider what needs to be in place before the new technique or process can be implemented and how the implementation will be carried out. These considerations should be included as part of the final validation report

Where appropriate the following should be addressed: How to validate part B Where appropriate the following should be addressed: The staff training plan and the arrangements for competence assessment and proficiency testing The protocols for calibration, monitoring and maintenance of any equipment The supply and traceability of any standards/reference materials The supply and quality control of key materials and reagents

Review progress on validation work How to validate part B The SOP documents for the process and the interpretation/reporting of results Anti-contamination protocols Any special requirements associated with health and safety  Review progress on validation work

Complete and prepare validation report How to validate Part C Complete and prepare validation report important that this includes all the information needed to facilitate independent assessment of the fitness for purpose of the technique or process A summary of the raw experimental data will normally suffice. But the raw data must be available. A statement of fitness for purpose regarding the method is added in the report.

Review validation report and implementation plan How to validate Part C Review validation report and implementation plan final validation report and implementation plan will be reviewed and approved at least by the operational manager for the designated area or the quality manager. must be signed of formally as deemed fit for use. consideration given to executive summary aim of this is to provide those making decisions on the use of the results with a summary of the validation steps performed, and key issues surrounding the validation

Instrumental Methods (Qualitative) Performance parameters Qualitative analysis can be defined as “Classification according to specified criteria In analytical chemistry and related disciplines, the ‘criteria’ are understood to relate, in general, to information about the presence, composition and/or structure of materials A qualitative method would normally give three possible results, positive, negative or inconclusive For single laboratory validation of qualitative methods recommend the Metrology of Qualitative Chemical Analysis (MEQUALAN) , report EUR 20605 EN, ISBN 92-894-5194-7, European Commission

Instrumental Methods (Qualitative) Performance parameters Precision most useful estimate is the within-lab reproducibility which is the precision measured with different analysts, over extended timescales and, within a single laboratory Precision for qualitative measurement can be stated in terms of a percentage of similar results obtained for test samples precision is generally dependent on analyte concentration, and so should be determined at different levels

Instrumental Methods (Qualitative) Performance parameters Trueness – Bias is an expression of how close the mean of a set of results is to the reference value can measure the false positive and false negative rates when we have prior information about the presence or absence of an analyte in a test sample For a given test method, the basic properties that need to be measured are the numbers of true positive (TP) and true negative (TN) results and the numbers of false positive (FP) and false negative (FN) results obtained on a range of samples. From these numbers, the fundamental measures of reliability viz. the false positive and false negative rates can be calculated.  

Instrumental Methods (Qualitative) Performance parameters The false rates vary with the level of any analyte. For high levels of analyte, the likelihood of false negatives will be very low but at levels slightly above the threshold it will be relatively high.

Instrumental Methods (Qualitative) Performance parameters Measurement range For a qualitative test there is a LOD or a threshold. For further guidance see The Fitness for Purpose of Analytical Methods; a Laboratory Guide to Method Validation and Related Topics, 2nd Edition, Eurachem (2014)

Instrumental Methods (Qualitative) Performance parameters Ruggedness Control of uncertainties in test parameters, such as times, temperatures, lengths etc, are vital for reliable qualitative testing. expected to control factors affecting the test result to within specified tolerances or in a validation demonstrate that the possible variation in individual test parameters have no significant influence on the outcome of the test. Measurement uncertainty Not directly applicable for a qualitative test.

Instrumental Methods (Quantitative) Performance parameters Precision the most useful estimate is the within-laboratory reproducibility is usually stated in terms of standard deviation or relative standard deviation (RSD) is generally dependent on analyte concentration, and so should be determined at a number of concentrations

Instrumental Methods (Quantitative) Performance parameters Trueness – Bias trueness is normally expressed in terms of measurement bias bias is estimated from the difference between the mean value of several measurement results preferable obtained under within-laboratory reproducibility conditions and a reference value trueness can also be investigated as selectivity e.g. by measuring bias at different levels of known interferences and in different matrices.

Instrumental Methods (Quantitative) Performance parameters Measurement range measuring range is an interval of the concentration, which can be measured with a specified uncertainty using the method the lower limit of the measuring range is often considered to be the limit of quantification, LOQ Eurachem guide: Terminology in analytical measurement -Introduction to VIM 3 (2011)

Instrumental Methods (Quantitative) Performance parameters Ruggedness/Robustness In any method there will be certain steps which, if not carried out sufficiently carefully, will have a severe effect on method performance steps should be identified and their influence on method performance can be evaluated involves making deliberate variations to the method, and investigating the subsequent effect on performance

Instrumental Methods (Quantitative) Performance parameters Measurement uncertainty expanded measurement uncertainty is normally what the laboratory reports to the customer provides an interval within which the value of the measurand is believed to lie with a high level of confidence (normally 95%) in proficiency testing the claimed uncertainty can be verified by comparing the difference between the result of the laboratory and the assigned value

Template of the validation plan and report IN-HOUSE VALIDATION OF MEASUREMENT PROCEDURES VALIDATION PLAN

Template of the validation plan and report IN-HOUSE VALIDATION OF MEASUREMENT PROCEDURES VALIDATION PLAN

Template of the validation plan and report

Template of the validation plan and report

Template of the validation plan and report

Template of the validation plan and report

The key to understanding how to validate human based methods is understanding what changes arise from substituting the INSTRUMENT with the HUMAN

Human Based Aspects not as self determining as for instrumental methods (i.e. selection of critical features) No standards available Hence: As method development is more demanding two requirements arise: documentation of the feature set to demonstrate that the method is fit for purpose documentation of the decision / choices made for the selection of the critical features Need to check the basis of these decisions ahead of performance testing (method development check)

Performance testing involves validating the analysis, comparison Instrumental Methods parameters to be validated e.g. precision, trueness, measurement range etc. Human Based Methods Specimen Analysis Comparison Evaluation Verification Performance testing involves validating the analysis, comparison and Verification steps Must use samples where the ground truth is known for the features in performance testing

THANK YOU!! geraldineaodonnell@fsl.gov.ie