May F. Mo FDA/Industry Statistics Workshop

Slides:



Advertisements
Similar presentations
Member consultation 2007 Draft ISPM: Sampling of Consignments Steward: David Porritt.
Advertisements

User verification of performance Zahra Khatami/Robert Hill BHR University Hospitals/Sheffield Teaching Hospitals Focus 2013.
Forecasting Using the Simple Linear Regression Model and Correlation
Hypothesis Testing Steps in Hypothesis Testing:
Sampling: Final and Initial Sample Size Determination
Chapter 11- Confidence Intervals for Univariate Data Math 22 Introductory Statistics.
World Health Organization
Multivariate Data Analysis Chapter 4 – Multiple Regression.
8 Statistical Intervals for a Single Sample CHAPTER OUTLINE
© 2000 Prentice-Hall, Inc. Chap Forecasting Using the Simple Linear Regression Model and Correlation.
8-1 Introduction In the previous chapter we illustrated how a parameter can be estimated from sample data. However, it is important to understand how.
Chemometrics Method comparison
Method Comparison A method comparison is done when: A lab is considering performing an assay they have not performed previously or Performing an assay.
Quality Assessment 2 Quality Control.
Statistics for Managers Using Microsoft Excel, 4e © 2004 Prentice-Hall, Inc. Chap 12-1 Chapter 12 Simple Linear Regression Statistics for Managers Using.
Copyright © 2005 Brooks/Cole, a division of Thomson Learning, Inc Chapter 24 Statistical Inference: Conclusion.
Chapter 7 Estimation: Single Population
QUALITY ASSURANCE Reference Intervals Lecture 4. Normal range or Reference interval The term ‘normal range’ is commonly used when referring to the range.
Standardization and Test Development Nisrin Alqatarneh MSc. Occupational therapy.
Stats for Engineers Lecture 9. Summary From Last Time Confidence Intervals for the mean t-tables Q Student t-distribution.
Many times in statistical analysis, we do not know the TRUE mean of a population of interest. This is why we use sampling to be able to generalize the.
All you wanted to know about laboratory tests, but were afraid to ask By Dietmar Stöckl STT Consulting The informed patient meeting the laboratory.
Chapter Nine Copyright © 2006 McGraw-Hill/Irwin Sampling: Theory, Designs and Issues in Marketing Research.
Chapter 8: Confidence Intervals
Analysis and Visualization Approaches to Assess UDU Capability Presented at MBSW May 2015 Jeff Hofer, Adam Rauk 1.
Method Validation and Verification: An Overview Patricia Hanson, Biological Administrator I Florida Department of Agriculture and Consumer Services, Food.
Estimation in Sampling!? Chapter 7 – Statistical Problem Solving in Geography.
Lecture 12 Statistical Inference (Estimation) Point and Interval estimation By Aziza Munir.
Quantitative Skills 1: Graphing
Biostatistics: Measures of Central Tendency and Variance in Medical Laboratory Settings Module 5 1.
Quality Control Lecture 5
New Draft Guidance for Multiplex Tests Elizabeth Mansfield and Michele Schoonmaker Office of In Vitro Diagnostic Device Evaluation and Safety (OIVD) CDRH/FDA.
Estimating a Population Proportion
Session Objectives To revisit the Audit Risk Model and Materiality concepts; To explain the Theory of Sampling as applied to audit To Explain the link.
Lecture 4: Statistics Review II Date: 9/5/02  Hypothesis tests: power  Estimation: likelihood, moment estimation, least square  Statistical properties.
Sampling distributions rule of thumb…. Some important points about sample distributions… If we obtain a sample that meets the rules of thumb, then…
Comparability of methods and analysers Nora Nikolac
Validation Defination Establishing documentary evidence which provides a high degree of assurance that specification process will consistently produce.
1 Module One: Measurements and Uncertainties No measurement can perfectly determine the value of the quantity being measured. The uncertainty of a measurement.
+ The Practice of Statistics, 4 th edition – For AP* STARNES, YATES, MOORE Chapter 8: Estimating with Confidence Section 8.2 Estimating a Population Proportion.
 Remember Chemistry panel Quality Control:-  In a medical laboratory, it is a statistical process used to monitor and evaluate the analytical process.
Confidence Intervals Dr. Amjad El-Shanti MD, PMH,Dr PH University of Palestine 2016.
SEMINAR ON PRESENTED BY BRAHMABHATT BANSARI K. M. PHARM PART DEPARTMENT OF PHARMACEUTICS AND PHARMACEUTICAL TECHNOLGY L. M. COLLEGE OF PHARMACY.
Diagnostic clinical chemistry
Chapter 8: Estimating with Confidence
Inference: Conclusion with Confidence
Chapter 8: Estimating with Confidence
CHAPTER 8 Estimating with Confidence
Chapter 9 One and Two Sample Estimation
Multivariate Analysis Lec 4
Practical clinical chemistry
This teaching material has been made freely available by the KEMRI-Wellcome Trust (Kilifi, Kenya). You can freely download,
Quality Assurance Reference Intervals.
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Quality Control Lecture 3
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
2/3/ Estimating a Population Proportion.
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
DESIGN OF EXPERIMENTS by R. C. Baker
Quality Assessment The goal of laboratory analysis is to provide the accurate, reliable and timeliness result Quality assurance The overall program that.
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Chapter 8: Estimating with Confidence
Presentation transcript:

Total Analytical Error Evaluation in Quantitative and Qualitative Assays May F. Mo FDA/Industry Statistics Workshop Washington, DC - September 29, 2006

Background Total Analytical Error (TAE) NOT a new concept Assay errors from all sources of the data collection experiment – including systematic and random error CLSI Proposed Guideline January 2002 CLSI Approved Guideline April 2003 FDA/Industry collaboration in developing the guideline TAE concepts have been seen in publications since 1960s. TAE does not include pre-analytical and post-analytical errors.

Cause and Effect Diagram for Total Analytical Error (from EP21-A)

Importance of TAE For Clinicians For Laboratories For Manufacturers TAE provides a metric of assay quality that can directly equate to medical errors For Laboratories TAE provides a simple, cost effective method for comparing assay products For Manufacturers TAE guides manufacturers to set appropriate design goals and evaluate the product directly to user needs From a clinician’s standpoint, TAE is more important metric, because an incorrect laboratory result, regardless of which error source causes it, is harmful. It is difficult to otherwise two assays that one has a higher precision and the other has a higher bias. For regulator, this can be a quick, one-step acceptance of substantial equivalence of two assays.

TAE Evaluation Methods TAE is evaluated by assessing if the combination of errors from all sources is within some acceptance limit. The common evaluation methods are (Krouwer, 2002): Direct Estimation Indirect Estimation Simulated Estimation Note: All methods are based on the quantitative assays.

Direct Estimation Direct Estimation (Distribution of Differences Method) EP21 Recommendation Calculate differences of study assay vs. reference assay TAE limit estimated as the CLI of the differences - No need to estimate and combine individual source errors Visual tools - Bland & Altman and “Mountain Plot” Mountain Plot – a folded empirical cumulative distribution plot CLI – Confidence Limits for Individual Observations

Direct Estimation (Continued) Is distribution of differences normal? Parametric? Nonparametric? or Data Transformation? Tolerance Interval: A range that contains a specified proportion of a sampled population with a specified confidence. (EP21) Parametric or nonparametric methods to report tolerance intervals are covered in Statistical Intervals by Hahn and Meeker (1991). TAE limit can be considered as the 2.5th/97.5th percentile (nonparametric) or CLI, which have uncertainty as well. The interval is an one time measure of the range that contains a specified proportion of sampled population. Tolerance Interval: Statistical inference – a range with confidence. A two-sided nonparametric 100(1-)% tolerance interval to contain at least 100p% of the sampled differences from a sample of size n is [x(l), x(u)].

Tolerance Interval (TI) Calculation A two-sided 100(1-)% TI to contain at least 100p% of the sampled differences from a sample of size n: Parametric: where k depends on , p and n, and can be obtained from published tables, or derived using noncentral t-distribution (Odeh and Owen, 1980) Nonparametric TI based on rank statistics is [x(l), x(u)]. The value of l and u depend on , p and n, and can be derived using published tables, or using equation provided in Hahn and Meeker (1991) Both calculations are now available in SAS: PROC UNIVARIACIPCTDF for nonparamatric (distribution-free) CIPCTNORMAL for parametric The only problem is SAS currently only provides TL for percentiles 1, 5, 10, 25. 50. 75, 90, 95, 99. The most needed 2.5th and 97.5th are not currently provided. This concern has been raised to SAS.

Indirect Estimation Indirect Estimation (Simple Combination Model) TAE = Bias + k * Imprecision Simple to apply Fits well with the Six-Sigma quality concepts Not recommended by EP21 - Underestimate TAE Simulation Method (Research Use) Cost-efficient Model the impact of an individual error component to TAE Can underestimate TAE K is the number of SD. It is mostly assigned as 2 or 1.65. This model underestimate TAE for the following reasons: Not using real patient samples Neglecting random interference bias Not including all sources of potential systematic error Can simulation method be used to support TAE claim?

Industry Practices Quantitative Assays Set Product Requirement Set allowable TAE (aTAE) limit based on medical opinion and marketing needs Break aTAE down to precision and bias design requirements using simple combination model. Example: TAE = Bias + 2*SD (for 2 sigma performance) Design Verification and Validation Verify design requirements individually A direct assessment of total error should be conducted to ensure aTAE is met. This is also called ‘error budgeting’ Each error components meeting the design requirement can not guarantee aTAE is met.

Industry Practices Quantitative Assays TAE Claims in Package Inserts Not Yet Common Concerns? Indirect method – Is it sufficient to establish a TAE claim? Direct method - When the reference assay is not a “gold standard”, TAE has measurement error from both assays … How to interpret? User environment – Are our customers ready to use TAE claims in the package inserts?

Generalize TAE to Qualitative Assays Motivation TAE provides rationale to establish product requirements TAE provides a final metric to evaluate if a product meets medical needs Challenge TAE is a quantitative concept TAE does not apply to low negative and high positive samples

TAE in Qualitative Assays The performance of a qualitative assay relies on its precision and bias near the medical decision point (cutoff) … TAE evaluation applicable and desirable for the cutoff zone Direct estimation not realistic due to limited patient samples around cutoff Indirect modeled approach may be more practical

aTAE in Qualitative Assays How to define aTAE for the near cutoff zone? EP12-A Guidance: The 95% interval - the values above and below the cutoff point at which repeated results are 95% positive or 95% negative, respectively Grayzone - the zone of uncertainty between the 95% interval limits It seems that the focus of EP12-A in measurement error around cutoff is random error (precision) rather than systematic error (bias) …

aTAE in Qualitative Assays Grayzone 5% 5% Upper 95% Interval Lower 95% Interval Cutoff

aTAE in Qualitative Assays Set aTAE for the near cutoff zone Incorporating measurement errors from both precision and bias, aTAE can be set as ½ of the grayzone. That is, the test result at 1.2 S/CO should read as positive (>1.0 S/CO) while test results at 0.8 S/CO should read as negative (<1.0 S/CO). Assuming that grayzone is 0.8-1.2 S/CO for an assay, aTAE will be 0.2 S/CO or 20% (assuming cutoff=1.0 S/CO) When the above aTAE is met, there should be <  (i.e., 0.05) probability that a test results will deviate >20% from the true value, and cause a change in the final interpretation.

aTAE in Qualitative Assays Set aTAE for the near cutoff zone For an assay that reports an indeterminate result, aTAE can be extended as the size the grayzone. The test result at 1.2 S/CO should not read as negative (<0.8 S/CO) while test results at 0.8 S/CO should not read as positive (>1.2 S/CO). In this case aTAE can be set at 0.4 S/CO or 50% for results around 0.8 S/CO and 33% for results around 1.2 S/CO.

TAE Evaluation for Qualitative assay Direct Evaluation – Not realistic Indirect Evaluation TAE = Bias +  * Precision Challenge: How to define and measure Bias? How to include ALL bias sources? Alternative: TAE =  * Reproducibility Reproducibility includes variations contributed from instrument, lot, calibration, operator, etc … Underestimate TAE is still then main concern The instrument to instrument, lot to lot, calibration to calibration variation are typically considered as bias, rather than imprecision. If TAE estimate<aTAE, a claim cannot be established; however, if TAE>aTAE, then the conclusion can be made that the product failed the aTAE goal.

TAE in Qualitative Assays How do different types of bias impact TAE? Bias impacts both S and CO As a proportional bias The will be no bias in S/CO As a fixed bias The bias in S/CO is unknown and depends on the value of S relative to CO (=1.0) and size of the bias CO=1.0 Bias S -0.2 -0.1 0.1 0.2 0.8 0.75 0.78 0.82 0.83 0.9 0.88 0.89 0.91 0.92 1.0 1.00 1.1 1.13 1.11 1.09 1.08 1.2 1.25 1.22 1.18 1.17

TAE in Qualitative Assays How does different type of biases impact TAE? Fixed bias for both S and CO: The impact on S/CO is relatively small for the near cutoff range (0.8-1.2 S/CO) Biases that impact S and CO differently: Interferences Tube Type Storage Sample Carry Over Crossreactivity …

Conclusion TAE is an useful evaluation method for both quantitative and qualitative assays. For qualitative assays, TAE should be evaluated for the near cutoff zone using indirect estimation. Since indirect method underestimates TAE, it may not be sufficient to support a claim.

References NCCLS. Estimation of Total Analytical Error for Clinical Laboratory Methods; Approved Guideline. NCCLS Document EP21-A [ISBN 1-56238-502-X]. NCCLS, 940 West Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 USA, 2003. NCCLS. User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline. NCCLS Document EP12-A [ISBN 1-56238-468-6]. NCCLS, 940 West Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 USA, 2002. Hahn GJ, Meeker WQ. Statistical Intervals: A Guide for Practitioners. New York: John Wiley and Sons, 1991. Krouwer JS, Monti KL. A simple graphical method to evaluate laboratory assays. Eur J Clin Chem Biochem 1995; 33:525–7. Odeh, RE, Owen, DB. Tables for Normal Tolerance Limits, Sampling Plans, and Screening. New York Marcel Dekker, 1980.