CHOICE OF METHODS AND INSTRUMENTS Part 3

3- Method Evaluation Comparison Of Methods The paired t-Test The paired t-Test uses the means of the two sets of numbers produced by the test method and the comparative method and determines if a significant difference exists between them However, it should be emphasized that the t-test is not adequate and appropriate for that purpose.

3- Method Evaluation Comparison Of Methods The F-Test This test compares the variance of the test method with that of the comparative method Variance is the square of the Sd and estimates precision or random error A method with little random error (good precision) will have a small value for the variance To perform the test, the larger variance is divided by the smaller If there is no difference between the precision of the two methods, then the two numbers should be the same

Qualitative Methods Qualitative diagnostic tests have been used since the early days of laboratory medicine for the screening, diagnosis, and management of a variety of diseases. Qualitative methods are test methods that provide only two categorical responses They include semi quantitative testing that use cut offs such as hepatitis testing and some molecular testing. No values/concentrations are included in the patient report.

Qualitative Methods Test results are reported as: Positive/negative, Normal/ borderline/abnormal, Reactive/nonreactive, Detected /not detected, etc

Qualitative Methods The following are the required components of validation: Accuracy Precision Sensitivity Specificity

Qualitative Methods Method validation: Accuracy Most sources recommend comparing at least 40 patient specimens. CLIA current guidance suggests a minimum of 20 samples. A larger number has a better chance to detect interferences. The actual number is less important than the quality of the samples. Test material can include: controls, proficiency testing material with known results, samples tested by another lab using the same or similar method. Test material matrix should match or be as close to the sample matrix as possible.

Qualitative Methods Method validation: Accuracy A method comparison experiment for accuracy is recommended to be done over a minimum of 5 days. Continue for another 5 days if discrepancies are observed. Document the results of the new method comparing the known results from the reference sources, or with results from the current method. Calculate the percent of positive, negative and total accuracy by dividing observed results over known results multiplied by 100.

Qualitative Methods Method validation: Accuracy Percent positive accuracy 19/20 X 100 = 95% Percent negative accuracy 20/20 X 100 = 100% Total accuracy 39/40 X 100 = 98% New method Current method Positive 19 20 Negative

Qualitative Methods Method validation: Precision Also known as Reproducibility. Can the new method duplicate the same results? Use samples that have a matrix as close as possible to the real specimen. For clinical tests patient samples are the first choice followed by control material and reference solutions. Most sources agree that a minimum of 2 negative samples and 2 positive samples run in triplicate for 5 days will provide data for within-run and between-run components to estimate precision.

Qualitative Methods Method validation: Precision Calculate the percent within-run, between-run and total precision by dividing observed results over known results multiplied by 100. Total Precision: 58/60 X 100 = 96.7%. (acceptable agreement 90%) ID Day 1 Day 2 Day 3 Day 4 Day 5 Between run % Pos sample Pos 15/15x100 = 100% Neg 14/15x100 = 93% Neg sample Within run % 12/12/x 100 = 100% 11/12 x 100 = 92%  

Qualitative Methods Method validation: Sensitivity Due to the lack of quantitative data, Qualitative sensitivity validation is not addressed by: Analytical Sensitivity- (Detection limit) defined as “the lowest concentration of the analyte which the test can reliably detect as positive in the given matrix”. But rather by: Diagnostic Sensitivity – The percent of subjects with the target condition whose test values are positive. (Ability to correctly identify individuals with disease) Calculate by dividing the number of true positives by the sum of the number of true positives plus the number of false negatives and multiplying by 100.

Qualitative Methods Method validation: Sensitivity Estimated Diagnostic Sensitivity Use the data from the comparison study to calculate Diagnostic Sensitivity.

All with disease (TP + FN) True Positive False Positive False Negative True Negative New method Old method Sensitivity = 100 X TP All with disease (TP + FN) Sensitivity = 100 X 11 = 92% 11 + 1

Qualitative Methods Method validation: Specificity Specificity - Due to the lack of quantitative data, Qualitative specificty validation is not addressed by: Analytical Specificity – the ability of a method to detect only the analyte that it was designed to detect. But rather by: Diagnostic Specificity – the percent of subjects without the target condition whose test values are negative. (Ability to correctly identify individuals without disease). Estimated Diagnostic Specificity

All without disease (FP + TN) True Positive False Positive False Negative True Negative New method Old method Specificity = 100 X TN All without disease (FP + TN) Specificity = 100 X 5 = 62.5% 3 + 5

Kit Evaluation A test " kit " is a simplified test method packaged with all of the reagents, supplies, and materials needed to perform a test Most kits are designed so that they are easy to use and understand, relatively inexpensive, produce results rapidly, and can be performed by relatively inexperienced persons Because of their low cost, ease of use, and simplicity, one may suppose that these " tests in a box " are reliable and do everything as advertised

Kit Evaluation This is not always the case. Test kits should be as rigorously evaluated Lot variation in reagents, failure to perform as advertised, and loss of sensitivity with age are some of the problems that cause problems in test kits. Follow the same procedures as above in selecting and evaluating the kit.

Kit Evaluation Factors to Consider Method Complexity Some kits are easier to use than others. Tests with fewer steps are better suited to the office laboratory Methods that require critical timing steps are prone to error and disrupt work flow of the test operator Some methods require more operator time and attention than others. Those methods that rate poorly for these factors should be avoided *****

Kit Evaluation Manufacturer Support We have found major differences in the quality of user's manuals Some manuals are complex for even an experienced laboratory technologist to follow In some kits, a well illustrated manual, showing each step in cartoon form, may quickly guide the operator through the procedure. The user's manual should contain a problem-solving section and provide information on obtaining technical support through a toll-free phone number. Local availability of supplies can be a great convenience in reducing the need for storage of the kits.

Kit Evaluation Be sure that your supplier has a sufficient turnover of stock to avoid delivery of kits with an early expiration date. Although quality control testing is required, not all manufacturers package quality control materials with their kits, and in some cases do not provide them at all. Additionally, a logically arranged kit arrangement, with ready-to-use reagent containers in a rack in the order of use and color coded bottles and caps to prevent the wrong cap being replaced on a bottle, may prove to be more user- friendly

Kit Evaluation Reliability The ultimate factor in evaluation of a test method or kit is the ability of the test to accurately provide clinically useful information. In test performance terms, this is expressed as sensitivity and specificity of the test. This information should be provided by the test manufacturer.

Kit Evaluation Economy and cost The actual cost per patient result of a kit depends on many things and is often very different from what the vender calculated by dividing the selling price by the number of test units in the package Expiration of the kit should be considered Possibility that test strips, slides or plates may deteriorate once sealed package has been opened

Evaluation of the Vendor Sometimes the choice between comparable methods comes down to which vendor gives the best service in terms of cost, reagent delivery and service to the instrument Choose a reliable vendor, one who has a history of prompt service and who  fulfill  his promises

Implementation of a new method, instrument or kit Once the method has been chosen, it must be properly prepared and introduced for use A final working procedure should be prepared following the guidelines already mentioned and should be done before any training of personnel A quality control program to monitor the method’s precision and a maintenance program should be designed and already started prior to the reporting of results

Reference Interval

Reference Interval (Normal Values) Provided by the manufacturer and verified by running known normal patients. If the lab has a similar patient population then the manufacturer’s ranges or even published reference intervals from textbooks or scientific articles may be used after verification. The Reference interval can be verified by testing 20 known normal samples; if no more than 2 results fall outside the manufacturer/ published range then that reference interval can be considered to be verified. If the laboratory cannot verify the reference intervals, then the reference interval will need to be established. This involves a selection of at least 120 reference samples for each group or subgroup that needs to be characterized.

Reference Interval The upper and lower reference limits are set to define a specified percentage (usually 95%) of the values for a population; this means that a percentage (usually 5%) of patients will fall outside of the reference interval in the absence of any condition or disease. Reference intervals are sometimes erroneously referred to as “normal ranges.” While all normal ranges are in fact reference intervals, not all reference intervals are normal ranges. This is represented by the reference interval for therapeutic drug levels.

Reference Interval In this case, a “normal” individual would not have any drug in their system, whereas a patient on therapy has a defined target range. Reference intervals are sometimes referred to as reference ranges; the preferred term is reference interval because range implies the absolute maximum and minimum values. The two main types of reference interval studies that are reviewed in this section are: Establishing a reference interval and Verifying a reference interval.

Reference Interval The clinical laboratory is required by good laboratory practice and accreditation agencies (i.e., College of American Pathologists [CAP] checklist) to either verify or establish reference intervals for any new tests or significant changes in methodology. Reference interval use can be grouped into three main categories: Diagnosis of a disease or condition, Therapeutic management, and Monitoring of a physiologic condition.

Reference Interval Diagnosis of a disease or condition Therapeutic management

Monitoring of a physiologic condition Reference Interval Monitoring of a physiologic condition

Reference Interval Selection of Reference Interval Study Individuals The selection of individuals who can be included in a reference interval study requires defining detailed: Inclusion criteria what factors (e.g., age, gender) are required to be used for the study Exclusion criteria. list factors that render individuals inappropriate for the study Determination of the necessary inclusion and exclusion criteria for donor selection may require extensive literature searches and review with laboratory directors and clinicians. Initially, it must be exactly defined what is a “healthy”/“normal” donor for associated reference values. For example, for a -human chorionic gonadotropin ( hCG) reference interval study, one would exclude pregnant women or those who may be pregnant, as well as individuals with hCG-producing tumors.

Reference Interval Selection of Reference Interval Study Individuals The use of donors who may not represent the population of interest has the potential to skew the evaluation data used to establish the reference interval. Capturing the appropriate information for the inclusion and exclusion criteria, such as donor health status, often requires a well-written confidential questionnaire and consent form.

Preanalytical and Analytical Considerations Preanalytical and analytical variables must be controlled and standardized in order to generate a valid reference interval.

Determining Whether to Establish or Verify Reference Intervals Whether to verify a reference interval or establish an entirely new reference interval for a new method/analyte depends on several factors, such as: The presence of an existing reference interval for assay and on The results of a statistical analysis comparing the test method with the reference method. The most basic method comparison involves plotting a reference method against a test method and fitting a linear regression. If the correlation coefficient is 1.0, slope is 1.000 and intercept is 0.000, the two methods agree and may not require new reference ranges. In this case, a simple reference interval verification study is all that may be required. Conversely, if the two methods differ considerably, then a new reference interval needs to be established.

Data Analysis to Establish a Reference Interval To establish a reference interval, it is recommended that the study include at least 120 individuals. This can be challenging and costly, but it may be necessary Once the raw data have been generated, the next step is to actually define the reference interval. The reference interval is calculated statistically using methods that depend on the distribution of the data. In the most basic sense, data may be either normally distributed (gaussian) or skewed (nongaussian).

Data Analysis to Establish a Reference Interval Gaussian Nongaussian

Data Analysis to Establish a Reference Interval If reference data are normally distributed, the reference interval can be determined using a parametric method. A parametric method defines the interval by the mean ± 2 SDs; by centering on the mean, this formula will include the central 95% of values as given in the example in Figure 2

Data Analysis to Establish a Reference Interval In reality, most analytes do not display a normal (gaussian) distribution. For example, the distribution of hCG in pregnant individuals is skewed. Data that are not normally distributed (i.e., nongaussian) must be analyzed using nonparametric analyses. Nonparametric determination of the reference interval is analyzed using percentages, which do not depend on the distribution. The reference interval is determined by using the central 95% of values; the reference range is therefore defined by the 2.5th to the 97.5th percentiles, as demonstrated in Figure

Determining Reference Intervals on Gaussian-distributed Data First, calculate the mean, median and standard deviation of the data set If the mean, median and mode are essentially the same number, then the distribution is Gaussian If these values are significantly different, then this is not Gaussian distribution and other method must be used If distribution is Gaussian, define a range of values ± 3SD from the mean and delete any values beyond these limits (to eliminate outliers) Recalculate the mean and SD, where the reference interval will be defined by ± 2SD

Data Analysis to Establish a Reference Interval Most reference interval analyses are determined using nonparametric analysis. This is because nonparametric analysis can be used on gaussian distributed data and it is the CLSI-recommended method

Determining Reference Intervals on Gaussian-distributed Data(Prothrombin Times (sec.) Mean = 11.307 Median = 11.3 Mode = 11.3 SD = 0.78 3SD range = 9.0 to 13.6 2SD range = 9.7 to 12.9

Calculation of Reference Intervals Using Percentile Ranking This method involves: Rearranging values in ascending order and selecting the middle 95% as the reference interval Find 2.5 percentile rank, multiply the value n + 1 (n equals the number of values in the data set) by 2.5% [(n+1) X 2.5 % = ??] Use this number and count from the 1st percentile (lowest value) by that amount. This will be the lower limit Do the same for the upper limit. Count down from the highest percentile to the 97.5 percentile If we narrow the range to 90%, large number of false positive tests may occur

Calculation of Reference Intervals Using Percentile Ranking No. of values = 100 2.5% → (100 + 1) x 2.5% = 2.5 ~ 3 Third value from the top is 10 s Third value from the bottom is 12.7 s The values between these numbers are the reference range which are 10.0 to 12.7 s in this case 97.5% percentile 2.5% percentile

Presence of outliers can affect the calculation of reference interval Calculation of Reference Intervals Using Percentile Ranking Problem of Outliers Presence of outliers can affect the calculation of reference interval Outlier is an observation point that is distant from other observations They can be due to measurement errors Errors in data recording or data transcription

The problem of outliers may be reduced by using the following Calculation of Reference Intervals Using Percentile Ranking Problem of Outliers The problem of outliers increase with small population and decrease with large sample population The problem of outliers may be reduced by using the following If the difference between the highest (or lowest) value and the next in the order is more than one third of the reference range, discard the value and then recalculate the percentile and the range X1: the upper value of the range, X2: the second value from the top Xn – X1 is the difference between the highest and lowest in the reference range X2 – X1 > 1/3 Xn – X1

Therefore, 10 should be discarded Calculation of Reference Intervals Using Percentile Ranking Problem of Outliers Example: Is a value of 10 in a group of 120 values with a reference range of 10 to 40 an outlier if the next value is 22? Therefore, 10 should be discarded X2 – X1 = 22 – 10 12 = 0.4 > 1/3 Xn – X1 40 – 10 30