Module 6: QC Basic Rules and Charts Analysis, Charts and Interpretation of Rules Quality Control: Samples, Charts and Rules

Learning Objectives Upon completion of this module you will be able to: Define: Control materials, Calibration materials, Levey-Jennings Chart, Westgard Single Rule and Multi-Rule Criteria, Systemic Error, & Random Error Describe a quality control sample in terms of assayed versus unassayed and their uses

Learning Objectives Upon completion of this module you will be able to: Discuss the following Westgard rules: 13s 12s Apply the Westgard rules listed above to decide whether an analytic run is accepted or rejected. List which Westgard rules detect systematic versus random error. 22s 6x 7t Quality Control: Samples, Charts and Rules

Definitions Control materials: Protein-based materials made or manufactured with a composition similar to patient samples that have or have not been analyzed for concentration so they have expected or no expected values. The International Federation of Clinical Chemistry defines a control solution or control material as a "specimen or solution which is analysed solely for quality control purposes, not for calibration" . Quality Control: Samples, Charts and Rules

Definitions Certified reference material (CRM) “A reference material that has one or more values certified by a technically valid procedure and is accompanied by, or is traceable to, a certificate or other document that is issued by a certifying body.” [CLSI] NBS (National Bureau of Standards) Calibrator: purified solution used to adjust electrical output of an instrument and should trace back to a CRM. The International Federation of Clinical Chemistry defines a control solution or control material as a "specimen or solution which is analysed solely for quality control purposes, not for calibration" . Quality Control: Samples, Charts and Rules

Definitions Levey-Jennings Chart: A visual presentation of daily quality control values plotted on a chart using mean, standard deviation, and +/- 3 Std Dev range criteria Westgard Single Rule and Multi-Rule Criteria: A set of rules used to evaluate quality control values as acceptable or not The International Federation of Clinical Chemistry states that calibrators are purified solutions used to adjust the output of an instrument to a particular concentration of an analyte. Quality Control: Samples, Charts and Rules

Definitions Systemic Errors: Errors in the test system usually caused by a malfunction that affects all tests Random Errors: Errors that statistically occur unpredictably and do not indicate system malfunction but do indicate an individual test or component malfunction

Need for Quality Control Samples Patient results are “unknowns”. Analysis of the patient sample produces a number. But…is the result valid? Patient may have previous result. But… Change from previous may be expected No change from previous result may be expected The product of a testing process is a numerical result. Unlike a physical product that can be inspected to assess whether it looks good or bad, you can't look at a test result and tell whether it's valid. You could look to see if that patient had the same test measured recently but you don’t know whether to expect it to change because the patient has improved or worsened or if you expect it to stay the same. Quality control samples are like patient samples only they have a known amount of analyte. QC samples (knowns) can be analysed along with patient samples (unknowns) to test the validity of all results in the run. Quality Control: Samples, Charts and Rules

Need for Quality Control Samples Quality control samples are “knowns.” They have been made or manufactured with a composition similar to patient samples and have been analyzed for concentration before they are put into use so they have expected results. Therefore, if Quality Control samples are analyzed and expected results are obtained, we can assume our system is operating correctly. or If Quality Control samples are analyzed and do not give expected results, what should be done?

Quality Control Samples Are Preserved to maintain accurate and precise results Assayed –provided with mean and s for analytes Unassayed – you must determine mean and s The purpose of a statistical quality control procedure is to monitor the analytical quality of the measurement during stable operation, detect changes from the stable operation, and eliminate reporting of results with medically important errors. Control samples should be similar to patient specimens. They can be purchased from manufacturers who make them from donated patient samples that have been preserved. It is important to treat the control materials carefully. Attributes of controls are the stability, low vial to vial variability, assayed versus unassayed, and how to reconstitute or treat the control material. Quality Control: Samples, Charts and Rules

Why is it advisable to run controls at different levels? Interactive Opportunity Why is it advisable to run controls at different levels? Gather input from participants. One acceptable response is that running a control that has a result falling in the reference range and at least one control that falls at the medical decision level is value to test the analytical phase. Quality Control: Samples, Charts and Rules

Quality Control Sample Use Analyzed solely for quality control purposes Not for calibration Should be used for qualitative and quantitative testing as known results to check if method to measure patient results is reliable. The International Federation of Clinical Chemistry defines a control solution or control material as a "specimen or solution which is analysed solely for quality control purposes, not for calibration" . Even with assayed controls you should assay the selected control materials under routine operating conditions to characterize the expected measurement variation and establish your own expected values. This usually involves obtaining at least 20 values and calculating the mean and standard deviation. There are a number of pitfalls from using bottle values or other estimates of the means, standard deviations, and control limits without verifying the results first. Quality Control: Samples, Charts and Rules

Interactive Opportunity Is it sufficient to use only one calibrator to set the output of an analyzer? Gather input from participants. One acceptable answer is that calibration should initially be with standards that test the lowest, mid and highest level of linearity. Validation of the curve using one calibrator is sufficient followed by analysis of quality control materials. Quality Control: Samples, Charts and Rules

Qualitative Versus Quantitative Results What are some examples of qualitative tests in the laboratory? What quality control samples would you use to verify the test is working so that patient results may be reported? Take answers from participants and put on flipchart. Examples: HIV with + and neg controls, RPR or VDRL with reactive and NR controls, ABO with A1 cells (positive for A), B cells (positive for B) O cells or antibody screen cells (negative for A and B), Gram stain (E coli for GNB and Staph for GPC), many others may apply. Most urine testing and hematology testing is quantitative and should be reminded of need for analyzing known control samples or blood films even for manual microscopy to check technician results.

QC Results and Expectations for Qualitative Tests Quality Controls will be a known positive and known negative material The results will be recorded in a log book with the date, techs initials, and either ‘positive’ or ‘negative’ written in by the tech Reason for running controls with qualitative tests is to check that the reagent(s) are working as they are supposed to and give accurate results Depending on the test, the (+) and (-) controls will be run once a day (usually every morning) or each time a patient specimen is analyzed. When running a urine dipstick test, the controls are usually run only in the morning and recorded; for other tests, HIV, RPR, etc. the (+) and (-) controls must be run along with the patient to ensure reagents have been added correctly. (with the HIV kits, the controls are build in to the test strip) If the (=) and (-) controls do not give the expected result you must discard the patient test and repeat both the controls and the patient. Quality Control: Samples, Charts and Rules

QC Results and Expectations for Quantitative Tests Gaussian distribution In the laboratory, control charts are used to make it simple to compare today's observed value with what is expected based on past history. Assuming a Gaussian or normal bell-shaped distribution, it would be expected that about 68% of the points fall within 1 SD of the mean, 95% within 2 SD of the mean and 99.7% within 3 SD of the mean. A control chart also has the mean and standard deviations indicated but they are shown on the y axis (as if you turned this Gassian distribution on its side.) Quality Control: Samples, Charts and Rules

What does the “Internal” Quality Control on a test cartridge indicate? Interactive Opportunity What does the “Internal” Quality Control on a test cartridge indicate? In some test systems it indicates only that sufficient sample has reached all necessary parts of the test cartridge. If you add sufficient water you would get the indication for + and -. Some qualitative immunoassay methods such as rapid HIV tests have internal quality control to test for the ability of the test strip to detect a positive result. Quality Control: Samples, Charts and Rules

QC Sample Analysis as an Alarm System QC sample result should: Indicate when the test system is not working properly Not indicate a false alarm It should be 100% reliable Quality control is run to see if you can get the right answer for a known sample. The right answer is actually a range of values that are calculated from the mean and standard deviation of past results. That mean and control limits can be shown on a control chart to make it simple to plot new control measurements and see how they compare with the expected range of values. From the technologist’s standpoint, the objectives of the control procedure are simply to "alert me when the method has a problem" and "don't alert me when the method is working okay." These are "true alarm" and "false alarm" situations. Technologists want to know about real problems, but can't afford to waste time when the method's working okay. Quality Control: Samples, Charts and Rules

Common QC Sample Rules Single Rule: Accept the analytical run if both quality control results fall within mean +/- 2 standard deviations range. Reject the analytical run and troubleshoot the problem if one or both quality control results are outside +/- 2 standard deviations from the mean. Quality Control: Samples, Charts and Rules

QC Sample Result Outside of Expected Range QC value < +2 SD or > -2 SD from the mean Expected most of the time With 95% confidence. This means 95% of acceptable results are in this range. 5% could still be acceptable i.e. false alarm QC value > + 3 SD or < - 3 SD from the mean Usually indicates a problem and should be followed up on It would be very unexpected (0.3% chance) to observe a control value greater than 3 SD from the mean and such an observation usually indicates there is a problem with the method. It is somewhat unexpected to observe a control value greater than 2 SD from the mean, but this will happen at least 5% of the time when analysing 1 control per run, so it may indicate a real problem or it may be a false alarm. It is very common (32% chance) to see individual values beyond 1 SD from the mean, therefore this control limit is of no value for making a judgment about method performance based on a single control value. Quality Control: Samples, Charts and Rules

Historical Use of QC Sample Results Shewart developed statistical tools to monitor quality Levey and Jennings – quality standards for the laboratory Westgard - established rules for monitoring quality control Walter A. Shewhart was a statistician at Bell Telephone Laboratories who developed the scientific basis for statistical process control. He set up limits within which the results must lie and found that deviations in the results of a routine process outside such limits indicate that the routine has broken down. Levey and Jennings introduced statistical control methods in clinical laboratories, calculating the average and range (maximum difference), then plotting the average and the range on two different control charts. Levey and Jennings also proposed making duplicate measurements on a patient specimen. Others suggested using a stable reference sample and plotting answers directly on a control chart, the Levey-Jennings chart. Quality Control: Samples, Charts and Rules

Levey-Jennings Chart A Control chart is a graphical method for displaying control results and evaluating whether a measurement procedure is in-control or out-of-control. Control results are plotted versus time or sequential run number; lines are generally drawn from point to point to accent any trends, systematic shifts, and random excursions. Control limits are lines drawn on a control chart to provide graphical criteria for assessing whether a measurement procedure is in-control or out-of-control. Quality Control: Samples, Charts and Rules

Levey-Jennings Chart

Quality Control Result Limits Mean is the central line. One standard deviation above = + 1 s One standard dev below = - 1s 2 standard dev above = + 2s 2 standard dev below = -2s 3 standard dev above = + 3s 3 standard dev below = -3s These control limits are usually calculated from the mean and standard deviation (SD, or s) determined for a given control material. Typically the interpretation is based on a specified number of results or points exceeding a certain control limit when in-control patient test results are reported. When out-of-control, the run is rejected and no test results can be reported. Quality Control: Samples, Charts and Rules

Calculating Confidence Limits on the Chart For example, Control 1 has a mean of 200 and a standard deviation of 4 mg/dL. What is the 95% confidence limit or mean + 2 s limit? The upper control limit, mean + 2s would be: 200 + 2*4 = 208 mg/dL The mean – 2s, lower control limit would be: 200 - 2*4 = 192 mg/dL 95% confidence limit is 192-208 mg/dL This should be a practice problem for participants to set 95% confidence limits. Click to get answers after participants have offered up their answers. Quality Control: Samples, Charts and Rules

Quality Control Sample Rules Decision Criteria In control versus out of control Control limits Mean + 2s 95% limits 5% false rejection If 1 result exceeds 2s but not 3s and no other shifts or trends, it is usually a warning but not a reason to reject the results. Control rule means decision criteria for judging whether an analytical run is in-control or out-of-control. It is defined by the number of control measurements, and control limits, often specified as the mean plus or minus a multiple of the standard deviation (s) or sometimes by a specified probability for false rejection. Mean +/- 2s give 95% change of correct decision but 5% false rejection. Quality Control: Samples, Charts and Rules

QC Sample Measured with each Analytical Run How long is a run? Manual Automated method Run, analytical run, or run length refer to the interval, which could be a period of time or group of samples, for which a decision on control status is to be made. Each batch is a run for a manual procedure. 2 controls should be run with each batch. 24 hours is usually the longest run time for automated chemistry methods; 2 controls should be run at least once every 24 hr. 8 hours is the longest run time for automated hematology methods; 3 controls should be run at least once every 8 hr. Quality Control: Samples, Charts and Rules

QC Sample Analysis Procedure Laboratory Quality Control Policy (Quality Manual) stating your laboratory will not report results without appropriate QC. Quality control Analysis SOP How many control samples How often run How documented How problems are solved What Westgard rules are used. You should prepare written guidelines to define the QC procedure in detail including how many controls to run per analyte, how often they are run, how to document results and problems and how to solve the problems. This written document is important for teaching laboratory techs the QC procedure and establishing a uniform practice. It is also necessary for meeting regulatory requirements. Quality Control: Samples, Charts and Rules

Lab Results Need to Be... Accurate Precise Quality Control Rules should help to assess accuracy and precision. After verification with QC Rules, lab results also need to be reported timely.

Accuracy Closeness of the measured result to the true value Your control values may show precision but be inaccurate Instrument can repeatedly get the same value, but it’s not the right value This diagram is imprecise, but accurate

Precision Precision is: The reproducibility or closeness of results to each other The degree of fluctuation on repeated measurements is indicative of the “precision” of the assay. The closeness of measurements to the true value is indicative of the “accuracy” of the assay. This is inaccurate, but is precise

Interactive Opportunity How can we check for accuracy and precision? Can you have precision without accuracy? Can you have accuracy without precision? STRESS to participants that Quality Control: Should detect random errors (Imprecision) Should detect systematic errors (Inaccuracy) Shift (6 results that settle around new mean) Trend (7 results upward or downward ) Quality Control: Samples, Charts and Rules

Possible Causes for Random Errors Can be due to variations in line voltage, pipetting, dispensers, contamination, volume dispensed, bubbles in lines of reagents, etc*… A random error can be either positive or negative The direction and exact magnitude of the random error cannot be exactly predicted Random Errors affect reproducibility or precision of a test system *NOTE: These variations should be detected during morning instrument start-up and maintenance *NOTE: Pipettes with poor accuracy can cause systematic errors also. *NOTE: When you see a random error on QC results, the technologist should look for the possible causes listed in the slide to begin the troubleshooting before re-running controls.

Random Errors and Possible Causes 1 3s Rule Violation: Usually indicates a Random Error When reviewing statistical data remember when using +/- 3 SD, there is a 0.3% chance of getting a control value outside the +/- 3 SD range Before running patient specimens you must troubleshoot AND then rerun the control and the values must be within +/- 2SD Check that controls are in date and well mixed* *NOTE: Remind participants to include detailed steps to correctly reconstitute control material in their policy statement and procedure for Section 5 (Equipment) in their Quality Manual Quality Control: Samples, Charts and Rules

Westgard Rules Random error detected by 1 3s This means that 1 QC value exceeds -3s Could be below -3s or above + 3s 13s refers to a control rule that is commonly used with a Levey-Jennings chart when the control limits are set as the mean plus 3s and the mean minus 3s. A run is rejected when a single control measurement exceeds the mean plus 3s or the mean minus 3s control limit and it indicates a random error. This indicates the need to stop and look for the cause of the random error, solving the problem before reporting the patient results. How can you determine it is random error on the first violation? May be a sudden shift or trend? Quality Control: Samples, Charts and Rules

Possible Causes for Random Errors 2 2s Rule Violation: Usually indicates a Systematic Error Remember, the first time single control value is > or < 2 SD it serves as a warning and you can run and report out your patient values BUT, the second sequential time the control value is > or < 2 SD, you must trouble shoot and find the reason for the error BEFORE you can run patient specimens and report out patient values Check that controls are in date and well mixed *NOTE: 12s is only a warning while 22s rule violation within 20 days signals the run cannot be accepted and the test must undergo troubleshooting to detect the error BEFORE patient specimens may be analyzed. Quality Control: Samples, Charts and Rules

Westgard Rules Systematic error detected by 2 2s Could be below -2s or above + 2s

Systematic Errors Systematic Errors include bias/shifts, & trends These errors affect accuracy of the test system. Usually 6x and 7t Rule Violations Can be due to calibration lot changes, temperature changes in incubator unit, light source deterioration, electronics, reagent lot changes, etc. Systematic errors are always in one direction on a L J chart.

Bias/Shift: 6x Rule Violation A systematic difference or systematic error between an observed value and some measure of the truth. 6x Rule violations: Usually indicate an error with your test system On day 7, after 6 days of one or more control values above or below the mean*, you must trouble shoot and find/correct error before running patient specimens *Note the values of the 6 days are within +/- 2sd FYI: Generally used to describe the inaccuracy of a method relative to a comparative method in a method comparison experiment. It also has a specific meaning in the statistical t-test, where bias equals the difference between the mean values of the two methods being compared or the average of all the differences between the paired sample values.

Westgard Rules Systematic Error detected with 6x: shift 6 results are on the same side of the mean If a laboratory uses this rule, on the 6th run in which the result settles around a new mean such as shown here with 6 results below mean and above -2s, a shift has occurred. This indicates the need to stop and look for the cause of the systematic error, solving the problem before reporting the patient results. //// OR, does troubleshooting this have any medical relevance for this particular test? Discuss Platelet counts vs K+ levels Quality Control: Samples, Charts and Rules

Trend: 7t Rule Violation Systematic error when a series of control values are treated as a time series 7t Rule violations: Usually indicate an error with your system On day 8, after 7 days of one or more control values moving either upward or downward, you must trouble shoot and find/correct error before running patient specimens

Westgard Rules Systematic error detected with 7t: trend 7 results falling in a pattern, up or down On the 7th run when an upward or downward trend is discovered even within the 95% limits of mean +/- 2 standard deviations, a systematic error has occurred. This indicates the need to stop and look for the cause of the systematic error, solving the problem before reporting the patient results. Quality Control: Samples, Charts and Rules

Levey Jennings Charts: Evaluation of Results

Westgard Rules: Single or Multi Rule QC Single rule QC Procedure uses a single criterion or single set of control limits, such as a Levey-Jennings chart with control limits set as either the mean plus or minus 2 standard deviations (2s) or the mean plus or minus 3s. There is more than 1 single rule but each rule is assessed individually. Multi-rule QC Uses a combination of decision criteria, or control rules, to decide if an analytical run is in-control or out-of-control Uses 5 different control rules to judge acceptability of an analytical run Now for a more technical description. Multirule QC uses a combination of decision criteria, or control rules, to decide whether an analytical run is in-control or out-of-control. The well-known Westgard multirule QC procedure uses 5 different control rules to judge the acceptability of an analytical run. By comparison, a single-rule QC procedure uses a single criterion or single set of control limits, such as a Levey-Jennings chart with control limits set as either the mean plus or minus 2 standard deviations (2s) or the mean plus or minus 3s. "Westgard rules" are generally used with 2 or 4 control measurements per run, which means they are appropriate when two different control materials are measured 1 or 2 times per material, which is the case in many chemistry applications. Some alternative control rules are more suitable when three control materials are analyzed, which is common for applications in hematology, coagulation, and immunoassays.

Westgard Rules Many laboratories follow the Westgard Single-Rule policy. The rules are assessed individually to check for random and systematic errors.

Westgard Multirule System These rules work best together in a multi-rule system with automated analyzers Rules evaluated in a certain order Starts with the warning rule (12s ) that triggers the need to look for other violations: The first time a result exceeds mean + 2s or mean -2s but is within 3 SD it is a warning to check back for shifts or trends or other rule violations. It should indicate the need to verify this occurs only rarely (5% of the time or no more than once in 20 days) and that a 6 x or 7t aren’t also present. A multirule system means that many rules are used to verify patient results are correct, ruling out both random errors, such as 1 3s and systematic errors such as 2 2s, 6x, 7t or even 4 1s rule violations. Quality Control: Samples, Charts and Rules

Westgard Multirules If 1 control exceeds 2 s (1 2s rule is violated) this decision tree says this is a warning to look for other rule violations such as the 2nd control exceeding 2 s (2 2s) or R4s to decide whether to accept or reject the run. Rejection means that you should turn to the procedure or instrument operating manual to solve the problem before reporting patient results. This decision tree helps to decide whether to reject or accept a run. If you answer yes, it means that rule has been violated. Quality Control: Samples, Charts and Rules

Key Points Describe a control in terms of Assayed (mean and s provided) Unassayed (mean and s not provided) Use (validation of run not calibration) Calculate 95% limits for quality control samples Detect errors in quality control using 95% and Westgard rules Quality Control: Samples, Charts and Rules

Key Points Levey-Jennings chart include values and markings on the x (days or time) and y axis (concentration mean and s) Quality Control: Samples, Charts and Rules

Key Points Westgard rules increase ability to detect systemic errors. Quality Control: Samples, Charts and Rules

Key Points Westgard Rules increase ability to detect random errors. Quality Control: Samples, Charts and Rules

Key Points Westgard Rule, 1 2s is provided as a Warning Quality Control: Samples, Charts and Rules

References Shewhart WA. Economic Control of Quality of Manufactured Product. New York; D. Van Hostrand Company, Inc., 1931. Levey S, Jennings ER. The use of control charts in the clinical laboratory. Am J Clin Pathol 1950;20:1059-66. Henry RJ, Segalove M. The running of standards in clinical chemistry and the use of the control chart. J Clin Pathol 1952;27:493-501. Westgard JO, Groth T, Aronsson T, Falk H, deVerdier C-H. Performance characteristics of rules for internal quality control: probabilities for false rejection and error detection. Clin Chem 1977;23:1857-67. Quality Control: Samples, Charts and Rules

References Westgard JO, Barry PL, Hunt MR, Groth T. A multi-rule Shewhart chart for quality control in clinical chemistry. Clin Chem 1981;27:493-501. Westgard JO, Barry PL. Cost-Effective Quality Control: Managing the Quality and Productivity of Analytical Processes. Washington, DC:AACC Press www.westgard.com Quality Control: Samples, Charts and Rules