1 Module PANEL TESTING High level

2 Content Overview What is panel testing? What is panel testing used for? PT advantages and disadvantages Preparation of test smears Validation of panel batches Panels composition Organization of a panel testing round Analysis of results; scoring system Interpretation and feedback Forms

3 What is Panel Testing? One of EQA methods One of EQA methods System of sending stained and/or unstained smears from NRL to peripheral laboratories to check proficiency in performing AFB smear microscopy and reporting AFB results System of sending stained and/or unstained smears from NRL to peripheral laboratories to check proficiency in performing AFB smear microscopy and reporting AFB results Tests individual performance of a laboratory worker, not the laboratory overall Tests individual performance of a laboratory worker, not the laboratory overall

4 Rapid assessment of performance of a laboratory staff to prioritize training and supervisory activities Rapid assessment of performance of a laboratory staff to prioritize training and supervisory activities Quick detection of problems associated with very poor performance Quick detection of problems associated with very poor performance Evaluation of competency of laboratory technicians prior to and following training Evaluation of competency of laboratory technicians prior to and following training What is Panel Testing Used For?

5 A minimal step for EQA with limited resources A minimal step for EQA with limited resources Monitoring performance of individuals in absence of a rechecking program Monitoring performance of individuals in absence of a rechecking program Supplements rechecking programs Supplements rechecking programs Investigation of excessive errors found in recheckingInvestigation of excessive errors found in rechecking What is Panel Testing Used For? (2)

6 Low workload for a peripheral center Improves laboratory credibility Improves laboratory credibility Rapid response countrywide possible Rapid response countrywide possible Possible identification of gross deficiencies Possible identification of gross deficiencies Use of stained and unstained smears can help identify source of problem Use of stained and unstained smears can help identify source of problem Advantages of Panel Testing

7 Technicians know they are being evaluated Technicians know they are being evaluated Does not measure routine performance Does not measure routine performance High workload for NRL High workload for NRL need for additional resources - appropriate equipment, highly qualified staff to produce panels need for additional resources - appropriate equipment, highly qualified staff to produce panels a system for panel sets distribution, data collection, analysis and feedback a system for panel sets distribution, data collection, analysis and feedback May not be motivating to improve daily performance May not be motivating to improve daily performance Disadvantages of Panel Testing

8 Preparation of Test Smears Routine patient smears: Problems with consistency Only stained smears available Smears of the required AFB quantification may not be easily available in the needed quantity Specially manufactured smears: Can be stained or unstained Provide uniformity of the test for technicians Provide known quantification of AFB required

9 Manufacturing Smears for PT Should be done at NRL Should be done at NRL requires a safety hood, centrifuge, vortex, water bath, lab supplies (pipettes, tubes, slides, boxes etc.)requires a safety hood, centrifuge, vortex, water bath, lab supplies (pipettes, tubes, slides, boxes etc.) BIOSAFETY MEASURES!BIOSAFETY MEASURES! Smears are prepared from known positive and negative sputa Smears are prepared from known positive and negative sputa Reference for the manufacturing procedures: Reference for the manufacturing procedures: AFB Smear Microscopy EQA Guidelines Requires time practice and expertise

10 Validation of Panel Batches Mandatory requirement! Pre-validation: Validation of consistency of panel batches prior to sending test panels out to periphery Post-validation: Validation of panel slides / batches after receiving aggregate results from all laboratories Keep accurate records of batches prepared and detailed results of the validation process Keep accurate records of batches prepared and detailed results of the validation process

11 Pre-validation Stain at least 6 slides from each batch to be examined independently by 3 or more technicians Stain at least 6 slides from each batch to be examined independently by 3 or more technicians Calculate the average results and standard deviation (SD) Calculate the average results and standard deviation (SD) The average minus 2 SD should be > 0 to accept the batch The average minus 2 SD should be > 0 to accept the batch Use the Validation Log to record results Use the Validation Log to record results

12 Normal Distribution All values are symmetrically distributed around the mean All values are symmetrically distributed around the mean Characteristic bell- shaped curve Characteristic bell- shaped curve Assumed for all quality control statistics Assumed for all quality control statistics Frequency x Variable

13 What is Standard Deviation? The principle calculation used in the laboratory to measure dispersion of a group of values around a mean Standard Deviation – Statistical Formula

14 Standard Deviation and Probability For a set of data with a normal distribution, a value will fall within a range of: For a set of data with a normal distribution, a value will fall within a range of: +/- 1 SD 68.2% of the time+/- 1 SD 68.2% of the time +/- 2 SD 95.5% of the time+/- 2 SD 95.5% of the time +/- 3 SD 99.7% of the time+/- 3 SD 99.7% of the time Laboratories use the +/- 2 SD criteria for the limits of the acceptable range for a control value Laboratories use the +/- 2 SD criteria for the limits of the acceptable range for a control value When the QC measurement falls within that range, there is 95.5% confidence that the measurement is correct When the QC measurement falls within that range, there is 95.5% confidence that the measurement is correct 68.2% 95.5% 99.7% Frequency -3s- 2s -1s Mean +1s +2s+3s

15 Sample Form: Validation Log for AFB Panel Testing Slide Batches (pre-validation) Intended positives should never be negative Intended negative smear should never be positive Quantification differences should not reach 2 steps on scale

16 The same smear error reported by a majority of technicians may represent a problem with the panel slide / batch: Technical difficulties in preparing panel slides Error in the pre-validation Incorrect recording of the expected result Fading of smears during transportation to peripheral sites Post-validation

17 Logbook of Panel Slides Sets / Post-validation

18 Coding of Panel Smears Ensure that result can not be guessed by an examinee – to avoid reading bias Ensure that result can not be guessed by an examinee – to avoid reading bias Make identification of a panel smear clear to a supervisor in charge of a panel testing exercise Make identification of a panel smear clear to a supervisor in charge of a panel testing exercise Example of a smears code : a panel set number a smear serial number in a panel a batch number

19 Panels Composition The composition of a panel set is determined by NRL The composition of a panel set is determined by NRL Number and types of slides to reassure that correct or incorrect results are not accidental Number and types of slides to reassure that correct or incorrect results are not accidental At least 10 slides provides a valid and fair testAt least 10 slides provides a valid and fair test Batch of stained and unstained smearsBatch of stained and unstained smears Unstained smears: Unstained smears: Evaluate staining technique; provide information about stain preparation and qualityEvaluate staining technique; provide information about stain preparation and quality

20 Examples of Panel Sets Compositions 1 slide graded 3+ 1 Slide graded 2+ 1 slide graded 1+ 2 slides graded 1-9 / 100 fields 1-9 / 100 fields 5 negative slides 1 slide graded 3+ 1 slide graded 2+ 2 slides graded 1+ 3 slides graded 1-9 / 100 fields 3 negative slides 1 slide graded slides graded 1+ 3 slides graded 1-9 / 100 fields 4 negative slides A panel test should represent a challenge in terms of difficulty: - some scanty and low-positive smears

21 Getting Started: Issues to Consider System for sending slides Frequency of testing Forms to record and report results Time allowed for technicians to complete PT Availability of microscopes Performance criteria Feedback and corrective action if needed Mechanism to resolve discrepant results

22 Implementation of Panel Testing Responsibility of the NRL- from preparation of slides to analysis of results and feedback Responsibility of the NRL- from preparation of slides to analysis of results and feedback Determine the number of AFB technicians who will participate in PT (ensure preparation of the needed number of panels) Determine the number of AFB technicians who will participate in PT (ensure preparation of the needed number of panels) Communicate with Public Health Directors regarding EQA activities Communicate with Public Health Directors regarding EQA activities Prepare the schedule for panel testing in each location Prepare the schedule for panel testing in each location Collaborate with intermediate laboratories Collaborate with intermediate laboratories

23 Sending Slides Delivery system based on services, regulations, resources available: mail/post courier supervisory visit Turnaround time Safe package to prevent breakage of slides: strong plastic slide holders

24 National reference laboratory Intermediate laboratory Peripheral laboratories A POSSIBLE SCHEME OF A PANEL TESTING ROUND Peripheral laboratories

25 Performing a Panel Test Round Frequency: at least one to two times a year Frequency: at least one to two times a year A standardized PT reporting form / an accompanying letter to provide instructions A standardized PT reporting form / an accompanying letter to provide instructions Individual, not group work Individual, not group work No incentives or punitive actions as a result of the PT exercise No incentives or punitive actions as a result of the PT exercise Time allowed to complete the PT exercise, Time allowed to complete the PT exercise,maximum: 2 hours for a stained slide set2 hours for a stained slide set 3 hours for an unstained slide set3 hours for an unstained slide set

26 Individual Results of Panel Testing / Feedback Form

27 Management of PT During a Supervisory Visit Administration of PT during on-site visits: Administration of PT during on-site visits: can be effective in some circumstancescan be effective in some circumstances provides direct observation of work under PT exerciseprovides direct observation of work under PT exercise corrective action may be easily facilitatedcorrective action may be easily facilitated BUT: may be impractical in routine conditionsBUT: may be impractical in routine conditions can be done in a special surveycan be done in a special survey Important: PT must not disrupt routine patients examinations, therefore consider: Important: PT must not disrupt routine patients examinations, therefore consider: Careful planning of a supervisory visitCareful planning of a supervisory visit Allocating sufficient time for a visitAllocating sufficient time for a visit

28 Analysis of PT Results A scoring system is to be developed prior to test A scoring system is to be developed prior to test Distinguish major and minor errors Distinguish major and minor errors false positive/negative related to 1+, 2+ or 3+ errors are major errorsfalse positive/negative related to 1+, 2+ or 3+ errors are major errors quantification errors (at least a 2 grade difference) and false positive / negative errors in the scanty group (1-9 AFB) are considered minorquantification errors (at least a 2 grade difference) and false positive / negative errors in the scanty group (1-9 AFB) are considered minor Determine successful score Determine successful score Determine plan of action for poor performances Determine plan of action for poor performances

29 Types and Classification of Errors Correct:No errors QEQuantification errorMinor error LFNLow False NegativeMinor error LFPLow False PositiveMinor error HFNHigh False NegativeMajor error HFPHigh False Positive Major error

30 Example of PT Scoring Set of 10 slides, each slide is worth 10 points, total possible score = 100 Set of 10 slides, each slide is worth 10 points, total possible score = 100 HFP and HFN scores 0HFP and HFN scores 0 LFP, LFN and QE scores 5LFP, LFN and QE scores 5 (QE = 2 grades difference) Passing score = 80 – 90Passing score = 80 – 90

31 Analysis of PT Results Study the aggregate results from all laboratories Study the aggregate results from all laboratories Post-validate panel slides/batches Post-validate panel slides/batches Assure that poor performance is not due to panel slide problemsAssure that poor performance is not due to panel slide problems If a majority of technicians fail to report correct results for the same slide/batch it may represent a problem with panel slide preparation: exclude this slide from scoring exclude this slide from scoring check returned discrepant slides check returned discrepant slides detect problems in preparation of panel smears detect problems in preparation of panel smears undertake measures to improve the quality of panel smears preparation undertake measures to improve the quality of panel smears preparation

32 PT– Interpretation of Results False positive and negative errors should be considered separately False positive and negative errors should be considered separately False positives - lack of proficiency / faulty microscope False positives - lack of proficiency / faulty microscope False negatives - poor stain / inadequate examination time / poor microscope False negatives - poor stain / inadequate examination time / poor microscope

33 Feedback to Laboratories on PT results Timely and confidential Timely and confidential Individual and aggregate test results Individual and aggregate test results Criteria for acceptable performance Criteria for acceptable performance Reports to TB program coordinator should provide appropriate background information and recommendations and not simply scores Reports to TB program coordinator should provide appropriate background information and recommendations and not simply scores Poor performance often requires a visit to laboratory Poor performance often requires a visit to laboratory

34 PT Aggregate Results of Multiple Laboratories

35 PT Aggregate Results Report: Example

36 Conclusion: Laboratories submitting unacceptable PT results with documented consistency and quality of PT slides experience serious problems with AFB microscopy. Additional resources should be obtained for supervisory visits, correction of problems identified, including replacement of faulty microscopes (and/or stains), retraining if needed, and follow-up panel testing.

37 Key Messages (I): PT is an effective method when it is necessary to quickly obtain information about capabilities of individual laboratory technicians to read smears and report results according to standards approved by NTP. PT is considered to be less effective than rechecking because it does not monitor routine performance; BUT PT can be more effective than rechecking in the areas where prevalence of positives is low.

38 Key Messages (II): The main prerequisite to start a panel testing program in a country is availability of a laboratory with a highly qualified staff capable to safely produce panel sets of appropriate quality and required composition. The main prerequisite to start a panel testing program in a country is availability of a laboratory with a highly qualified staff capable to safely produce panel sets of appropriate quality and required composition. Validation of panel batches (pre- and post-validation) is the mandatory requirement for PT. Validation of panel batches (pre- and post-validation) is the mandatory requirement for PT. A well functioning system should be established to distribute panels, collect and analyze data; provide timely feedback to peripheral laboratories. A well functioning system should be established to distribute panels, collect and analyze data; provide timely feedback to peripheral laboratories.