1. CHOICE OF METHODS AND INSTRUMENTS
Part 3

2. 3- Method Evaluation Comparison Of Methods
The paired t-Test
The paired t-Test uses the means of the two sets of numbers produced by the test method and the comparative method and determines if a significant difference exists between them
However, it should be emphasized that the t-test is not adequate and appropriate for that purpose.

3. 3- Method Evaluation Comparison Of Methods
The F-Test
This test compares the variance of the test method with that of the comparative method
Variance is the square of the Sd and estimates precision or random error
A method with little random error (good precision) will have a small value for the variance
To perform the test, the larger variance is divided by the smaller
If there is no difference between the precision of the two methods, then the two numbers should be the same

4. Qualitative Methods
Qualitative diagnostic tests have been used since the early days of laboratory medicine for the screening, diagnosis, and management of a variety of diseases.
Qualitative methods are test methods that provide only two categorical responses
They include semi quantitative testing that use cut offs such as hepatitis testing and some molecular testing.
No values/concentrations are included in the patient report.

5. Qualitative Methods Test results are reported as: Positive/negative,
Normal/ borderline/abnormal,
Reactive/nonreactive,
Detected /not detected, etc

6. Qualitative Methods
The following are the required components of validation:
Accuracy
Precision
Sensitivity
Specificity

7. Qualitative Methods Method validation: Accuracy
Most sources recommend comparing at least 40 patient specimens.
CLIA current guidance suggests a minimum of 20 samples.
A larger number has a better chance to detect interferences.
The actual number is less important than the quality of the samples.
Test material can include: controls, proficiency testing material with known results, samples tested by another lab using the same or similar method.
Test material matrix should match or be as close to the sample matrix as possible.

8. Qualitative Methods Method validation: Accuracy
A method comparison experiment for accuracy is recommended to be done over a minimum of 5 days.
Continue for another 5 days if discrepancies are observed.
Document the results of the new method comparing the known results from the reference sources, or with results from the current method.
Calculate the percent of positive, negative and total accuracy by dividing observed results over known results multiplied by 100.

9. Qualitative Methods Method validation: Accuracy
Percent positive accuracy 19/20 X 100 = 95%
Percent negative accuracy 20/20 X 100 = 100%
Total accuracy 39/40 X 100 = 98%
New method
Current method
Positive 19 20
Negative

10. Qualitative Methods Method validation: Precision
Also known as Reproducibility.
Can the new method duplicate the same results? Use samples that have a matrix as close as possible to the real specimen.
For clinical tests patient samples are the first choice followed by control material and reference solutions.
Most sources agree that a minimum of 2 negative samples and 2 positive samples run in triplicate for 5 days will provide data for within-run and between-run components to estimate precision.

11. Qualitative Methods Method validation: Precision
Calculate the percent within-run, between-run and total precision by dividing observed results over known results multiplied by Total Precision: 58/60 X 100 = 96.7%. (acceptable agreement 90%) ID
Day 1
Day 2
Day 3
Day 4
Day 5
Between run %
Pos sample
Pos
15/15x100 = 100%
Neg
14/15x100 = 93%
Neg sample
Within run %
12/12/x 100 = 100%
11/12 x 100 = 92%

12. Qualitative Methods Method validation: Sensitivity
Due to the lack of quantitative data, Qualitative sensitivity validation is not addressed by:
Analytical Sensitivity- (Detection limit) defined as “the lowest concentration of the analyte which the test can reliably detect as positive in the given matrix”.
But rather by:
Diagnostic Sensitivity – The percent of subjects with the target condition whose test values are positive. (Ability to correctly identify individuals with disease)
Calculate by dividing the number of true positives by the sum of the number of true positives plus the number of false negatives and multiplying by 100.

13. Qualitative Methods Method validation: Sensitivity
Estimated Diagnostic Sensitivity
Use the data from the comparison study to calculate Diagnostic Sensitivity.

14. All with disease (TP + FN)
True Positive
False Positive
False Negative
True Negative
New method
Old method
Sensitivity =
100 X TP
All with disease (TP + FN)
Sensitivity =
100 X 11
= 92%
11 + 1

15. Qualitative Methods Method validation: Specificity
Specificity - Due to the lack of quantitative data, Qualitative specificty validation is not addressed by:
Analytical Specificity – the ability of a method to detect only the analyte that it was designed to detect.
But rather by:
Diagnostic Specificity – the percent of subjects without the target condition whose test values are negative. (Ability to correctly identify individuals without disease).
Estimated Diagnostic Specificity

16. All without disease (FP + TN)
True Positive
False Positive
False Negative
True Negative
New method
Old method
Specificity =
100 X TN
All without disease (FP + TN)
Specificity =
100 X 5
= 62.5%
3 + 5

17. Kit Evaluation
A test " kit " is a simplified test method packaged with all of the reagents, supplies, and materials needed to perform a test
Most kits are designed so that they are easy to use and understand, relatively inexpensive, produce results rapidly, and can be performed by relatively inexperienced persons
Because of their low cost, ease of use, and simplicity, one may suppose that these " tests in a box " are reliable and do everything as advertised

18. Kit Evaluation
This is not always the case. Test kits should be as rigorously evaluated
Lot variation in reagents, failure to perform as advertised, and loss of sensitivity with age are some of the problems that cause problems in test kits.
Follow the same procedures as above in selecting and evaluating the kit.

19. Kit Evaluation Factors to Consider Method Complexity
Some kits are easier to use than others.
Tests with fewer steps are better suited to the office laboratory
Methods that require critical timing steps are prone to error and disrupt work flow of the test operator
Some methods require more operator time and attention than others. Those methods that rate poorly for these factors should be avoided
*****

20. Kit Evaluation Manufacturer Support
We have found major differences in the quality of user's manuals
Some manuals are complex for even an experienced laboratory technologist to follow
In some kits, a well illustrated manual, showing each step in cartoon form, may quickly guide the operator through the procedure.
The user's manual should contain a problem-solving section and provide information on obtaining technical support through a toll-free phone number.
Local availability of supplies can be a great convenience in reducing the need for storage of the kits.

21. Kit Evaluation
Be sure that your supplier has a sufficient turnover of stock to avoid delivery of kits with an early expiration date.
Although quality control testing is required, not all manufacturers package quality control materials with their kits, and in some cases do not provide them at all.
Additionally, a logically arranged kit arrangement, with ready-to-use reagent containers in a rack in the order of use and color coded bottles and caps to prevent the wrong cap being replaced on a bottle, may prove to be more user- friendly

22. Kit Evaluation Reliability
The ultimate factor in evaluation of a test method or kit is the ability of the test to accurately provide clinically useful information.
In test performance terms, this is expressed as sensitivity and specificity of the test.
This information should be provided by the test manufacturer.

23. Kit Evaluation Economy and cost
The actual cost per patient result of a kit depends on many things and is often very different from what the vender calculated by dividing the selling price by the number of test units in the package
Expiration of the kit should be considered
Possibility that test strips, slides or plates may deteriorate once sealed package has been opened

24. Evaluation of the Vendor
Sometimes the choice between comparable methods comes down to which vendor gives the best service in terms of cost, reagent delivery and service to the instrument
Choose a reliable vendor, one who has a history of prompt service and who  fulfill  his promises

25. Implementation of a new method, instrument or kit
Once the method has been chosen, it must be properly prepared and introduced for use
A final working procedure should be prepared following the guidelines already mentioned and should be done before any training of personnel
A quality control program to monitor the method’s precision and a maintenance program should be designed and already started prior to the reporting of results

26. Reference Interval

27. Reference Interval (Normal Values)
Provided by the manufacturer and verified by running known normal patients.
If the lab has a similar patient population then the manufacturer’s ranges or even published reference intervals from textbooks or scientific articles may be used after verification.
The Reference interval can be verified by testing 20 known normal samples; if no more than 2 results fall outside the manufacturer/ published range then that reference interval can be considered to be verified.
If the laboratory cannot verify the reference intervals, then the reference interval will need to be established.
This involves a selection of at least 120 reference samples for each group or subgroup that needs to be characterized.

28. Reference Interval
The upper and lower reference limits are set to define a specified percentage (usually 95%) of the values for a population; this means that a percentage (usually 5%) of patients will fall outside of the reference interval in the absence of any condition or disease.
Reference intervals are sometimes erroneously referred to as “normal ranges.”
While all normal ranges are in fact reference intervals, not all reference intervals are normal ranges.
This is represented by the reference interval for therapeutic drug levels.

29. Reference Interval
In this case, a “normal” individual would not have any drug in their system, whereas a patient on therapy has a defined target range.
Reference intervals are sometimes referred to as reference ranges; the preferred term is reference interval because range implies the absolute maximum and minimum values.
The two main types of reference interval studies that are reviewed in this section are:
Establishing a reference interval and
Verifying a reference interval.

30. Reference Interval
The clinical laboratory is required by good laboratory practice and accreditation agencies (i.e., College of American Pathologists [CAP] checklist) to either verify or establish reference intervals for any new tests or significant changes in methodology.
Reference interval use can be grouped into three main categories:
Diagnosis of a disease or condition,
Therapeutic management, and
Monitoring of a physiologic condition.

31. Reference Interval Diagnosis of a disease or condition
Therapeutic management

32. Monitoring of a physiologic condition
Reference Interval
Monitoring of a physiologic condition

33. Reference Interval Selection of Reference Interval Study Individuals
The selection of individuals who can be included in a reference interval study requires defining detailed:
Inclusion criteria
what factors (e.g., age, gender) are required to be used for the study
Exclusion criteria.
list factors that render individuals inappropriate for the study
Determination of the necessary inclusion and exclusion criteria for donor selection may require extensive literature searches and review with laboratory directors and clinicians.
Initially, it must be exactly defined what is a “healthy”/“normal” donor for associated reference values.
For example, for a -human chorionic gonadotropin ( hCG) reference interval study, one would exclude pregnant women or those who may be pregnant, as well as individuals with hCG-producing tumors.

34. Reference Interval Selection of Reference Interval Study Individuals
The use of donors who may not represent the population of interest has the potential to skew the evaluation data used to establish the reference interval.
Capturing the appropriate information for the inclusion and exclusion criteria, such as donor health status, often requires a well-written confidential questionnaire and consent form.

35. Preanalytical and Analytical Considerations
Preanalytical and analytical variables must be controlled and standardized in order to generate a valid reference interval.

36. Determining Whether to Establish or Verify Reference Intervals
Whether to verify a reference interval or establish an entirely new reference interval for a new method/analyte depends on several factors, such as:
The presence of an existing reference interval for assay and on
The results of a statistical analysis comparing the test method with the reference method.
The most basic method comparison involves plotting a reference method against a test method and fitting a linear regression.
If the correlation coefficient is 1.0, slope is and intercept is 0.000, the two methods agree and may not require new reference ranges.
In this case, a simple reference interval verification study is all that may be required.
Conversely, if the two methods differ considerably, then a new reference interval needs to be established.

38. Data Analysis to Establish a Reference Interval
To establish a reference interval, it is recommended that the study include at least 120 individuals.
This can be challenging and costly, but it may be necessary
Once the raw data have been generated, the next step is to actually define the reference interval.
The reference interval is calculated statistically using methods that depend on the distribution of the data.
In the most basic sense, data may be either normally distributed (gaussian) or skewed (nongaussian).

39. Data Analysis to Establish a Reference Interval
Gaussian
Nongaussian

40. Data Analysis to Establish a Reference Interval
If reference data are normally distributed, the reference interval can be determined using a parametric method.
A parametric method defines the interval by the mean ± 2 SDs; by centering on the mean, this formula will include the central 95% of values as given in the example in Figure 2

41. Data Analysis to Establish a Reference Interval
In reality, most analytes do not display a normal (gaussian) distribution.
For example, the distribution of hCG in pregnant individuals is skewed.
Data that are not normally distributed (i.e., nongaussian) must be analyzed using nonparametric analyses.
Nonparametric determination of the reference interval is analyzed using percentages, which do not depend on the distribution.
The reference interval is determined by using the central 95% of values; the reference range is therefore defined by the 2.5th to the 97.5th percentiles, as demonstrated in Figure

42. Determining Reference Intervals on Gaussian-distributed Data
First, calculate the mean, median and standard deviation of the data set
If the mean, median and mode are essentially the same number, then the distribution is Gaussian
If these values are significantly different, then this is not Gaussian distribution and other method must be used
If distribution is Gaussian, define a range of values ± 3SD from the mean and delete any values beyond these limits (to eliminate outliers)
Recalculate the mean and SD, where the reference interval will be defined by ± 2SD

43. Data Analysis to Establish a Reference Interval
Most reference interval analyses are determined using nonparametric analysis.
This is because nonparametric analysis can be used on gaussian distributed data and it is the CLSI-recommended method

44. Determining Reference Intervals on Gaussian-distributed Data(Prothrombin Times (sec.)
Mean =
Median = 11.3
Mode = 11.3
SD = 0.78
3SD range = 9.0 to 13.6
2SD range = 9.7 to 12.9

45. Calculation of Reference Intervals Using Percentile Ranking
This method involves:
Rearranging values in ascending order and selecting the middle 95% as the reference interval
Find 2.5 percentile rank,
multiply the value n + 1 (n equals the number of values in the data set) by 2.5% [(n+1) X 2.5 % = ??]
Use this number and count from the 1st percentile (lowest value) by that amount. This will be the lower limit
Do the same for the upper limit. Count down from the highest percentile to the 97.5 percentile
If we narrow the range to 90%, large number of false positive tests may occur

46. Calculation of Reference Intervals Using Percentile Ranking
No. of values = 100
2.5% → ( ) x 2.5% = 2.5 ~ 3
Third value from the top is 10 s
Third value from the bottom is 12.7 s
The values between these numbers are the reference range which are to 12.7 s in this case
97.5% percentile
2.5% percentile

47. Presence of outliers can affect the calculation of reference interval
Calculation of Reference Intervals Using Percentile Ranking Problem of Outliers
Presence of outliers can affect the calculation of reference interval
Outlier is an observation point that is distant from other observations
They can be due to measurement errors
Errors in data recording or data transcription

48. The problem of outliers may be reduced by using the following
Calculation of Reference Intervals Using Percentile Ranking Problem of Outliers
The problem of outliers increase with small population and decrease with large sample population
The problem of outliers may be reduced by using the following
If the difference between the highest (or lowest) value and the next in the order is more than one third of the reference range, discard the value and then recalculate the percentile and the range
X1: the upper value of the range, X2: the second value from the top
Xn – X1 is the difference between the highest and lowest in the reference range
X2 – X1
> 1/3
Xn – X1

49. Therefore, 10 should be discarded
Calculation of Reference Intervals Using Percentile Ranking Problem of Outliers
Example:
Is a value of 10 in a group of 120 values with a reference range of 10 to 40 an outlier if the next value is 22?
Therefore, 10 should be discarded
X2 – X1 =
22 – 10 12
= 0.4 > 1/3
Xn – X1
40 – 10 30