Laboratory QC: Bridging the Gap Between Theory and Practice 11th Annual HuQAS Scientific Conference Kenya Institute of Education Nairobi, Kenya September 29th, 2011

Post-analytical Results filing – correct result for correct vial Result reported to correct number of decimals Results filed after deadline Results changed after reporting RI reported with result Comments reported

The Forces of Change Evidence based medicine - improve outcomes Standardize medical management – Clinical Guidelines – tied to discrete test results (NGSP, NCEP-ATP, NKDEP) Greater awareness of the costs associated with laboratory error and non-standardized testing Electronic medical record – “the holy grail” Accreditation Tied to Quality Management Systems (ISO 15189, 17025, 17011)

Appreciating Today’s Reality “trueness” is now becoming the analytical goal (JCTLM) Peer group monitoring of performance rewards precision – “as long as I am consistently wrong I am ok” How good do we need to be? (patient centric clinically relevant performance goals) How can we harmonize/standardize testing country wide?

Appreciating Today’s Reality Assess laboratory performance using commutable test samples with target values assigned by reference methods For success there is a need for laboratories to work together and to openly share their performance data on a network wide basis Move towards the “ideal” as the performance of laboratory testing within the country improves

Are your methods good enough? What is good enough? How do you know?

“Performance is a measure of how we are doing” “Quality is a measure of performance against a defined standard or requirement” “Setting a performance standard is a pre-requisite for measurement of quality”

Clinical Laboratory Improvement Act (USA) Laboratories must establish or verify Performance Standards for each test. They may get these either from the manufacturer (for unmodified methods) or establish them (for modified methods) (Section 493.1253(b)(3)). Laboratories are required to make these same performance standards available to clients upon request. (493.1291(3)).

Quality Management of the Testing Process A definition of what is expected (performance standards, SOP, process maps) A system that measures variation from the expected (PT/EQA, IQC, internal auditing systems) A strategy for effecting and sustaining desired change (non-conformance, root cause analysis, corrective action, preventive action) A system for measuring and monitoring the impact of change (PT, internal auditing, outcome analysis, continual improvement) over time

Setting Quality Specifications Global strategy for setting quality specifications in laboratory medicine (IUPAC, WHO, IFCC – Stockholm 1999 - Scan J Clin Lab Invest 1999;57:475-585) 1. Quality specifications based upon an assessment of the effect of analytical performance on specific clinical decisions (NCEP-ATP) 2. Quality specifications based on biological variation (NKDEP- eGFR)

Setting Quality Specifications 3.Quality specifications based on medical opinions 4. Quality specifications based on guidelines from national/international expert groups 5. Quality specifications based on guidelines from expert individuals or institutional groups 6.Quality specifications as required by regulatory authorities 7. Quality specifications as laid down by PT/EQAS providers 8.Quality specifications as derived from published data on the state of the art

Setting Quality Specifications 1. Quality specifications based upon an assessment of the effect of analytical performance on specific clinical decisions Example: NCEP-ATP recommended performance specifications for the measurement of lipids

Setting Quality Specifications NCEP Quality Specifications - Measurement of Total Cholesterol Bias = 3% or less (relative to A-K reference method) Precision - CV = ≤ 3 % TE (%) = bias + 1.96 x CV = 9% (rounded)

NCEP Performance goal (TE) An International study examining the “trueness” and precision of lipid measurements in 27 countries Analyte NCEP Performance goal (TE) % Failure Total cholesterol 9 % 22% Triglycerides 15% 7% HDL 13% 48% LDL 12% 33%

Comparison of Quality Specifications NCEP Criteria CLIA Cholesterol RV +/- 9% PG +/- 10% HDL RV +/- 13% PG +/- 30% LDL RV +/- 12% Not Regulated Triglycerides RV +/- 15% PG +/- 25%

Setting Quality Specifications 2. General quality specifications based on biological variation Example National Kidney Disease Education Program – routine reporting of eGFR by the MDRD formula (NKDEP- eGFR)

Creatinine – Performance Goals Recommended Total Error performance goals (TEa) for the measurement of creatinine as calculated on the basis of biological variation: Minimum 11.4 % Desirable 7.6% Optimal 3.8% Clin Chem 52:5-18 (2006)

Creatinine (CLIA Performance Criteria) Peer group mean +/- 15% or 0.26 mg/dL (23.4 umol/L) whichever is greater Under CLIA – given a peer mean of 1.13 mg/dL (100 umol/L) labs will get a “pass” if they are within +/- 23.4% Clinically relevant TE performance goal for measurement of creatinine: Minimum 11.4 %; desirable 7.6%; optimal 3.8%

Performance data from an accuracy based EQA program for the measurement of creatinine Performance Assessment Criteria RV +/- 7.6% N= 176 IDMS Reference (umoles/L) (mg/dL) 113.5 1.28 97.2 1.10 65.4 0.74 % of labs failing 7.4% 14.8% 38.2%

Setting Quality Specifications 3. General quality specifications based on medical opinions "If the true value for this test is "x", how close to this value must the reported result be in order for you to make a meaningful decision?" You will be lucky if you can get two clinicians to agree!

Setting Quality Specifications 4. Guidelines from national/international expert groups “the clinical decision levels for albuminuria have been set and applied without regard to the analytical method that is being used in the determinations” The performance specifications for the methods used in these guidelines are typically not reported.

Setting Quality Specifications 5. Quality specifications laid down by regulation (CLIA) “CLIA quality specifications have more to do with what is achievable as opposed to what is actually needed to meet the clinical need “

Setting Quality Specifications (6) Quality specifications as defined by EQA/PT providers

Assessment Criteria from Six PT/EQA Providers Quantitative – Urine B C D E F Albumin Peer Group Mean +/- 30% or 3 SD +/-10 mg/L 25% 30% +/- 3 SD +/- 2.5 SD Creatinine +/-30 mg/dL 15% +/- 2 SD +/- 0.3 mg/dL 17% Albumin : Creatinine Ratio Not Assessed

Fall-back Position Tonks' Rule Tonks'’ Rule was one of the first published approaches for the setting of performance standards. He suggested that the variation of a method should be less than 1/4 the span of the reference interval, divided by the mean of the reference interval.

“Biological Variation: from Principles to Practice” Callum G “Biological Variation: from Principles to Practice” Callum G. Fraser AACC Press (2001) The most recent and extensive data on biological variation may be found at: www.westgard.com/guest17.htm

Setting Performance Criteria on the Basis of Biological Variation “What is biological variation?” Daily biological rhythms Monthly cycles Seasonal rhythms Random biological variation

Biological Variation “measure creatinine repeatedly in ten healthy male subjects over a period of 14 days”

Creatinine Reference Interval (64-120 umoles/L or 0.72-1.36 mg/dL)

Observations on Biological Variation The range of values for anyone individual covers a small part of the reference interval The mean values of all individuals lie within the reference interval and differ from each other Within-subject variation is the quantitative estimate of homeostasis in man Ample evidence to indicate that within-subject variation is constant Little evidence that within-subject variation changes with age

Observations on Biological Variation Homeostatic set points can change with age and pathology but the variation around these set points does not change In general, estimates of CVi are constant irrespective of the number of subjects, the time scale of the study, the methodology and the country in which the study was done Performance goals for precision and bias can be defined on the basis of within and between subject biological variation

Performance Criteria - Biological Variation Three levels of performance can be defined: minimum, desirable and optimal Performance Criteria for Precision CVa < 0.25 Cvi (Optimal) (3% increase in Result Variability) CVa < 0.50 CVi (Desirable) (12% increase in Result Variability) CVa < 0.75 CVi (Minimum) (25% increase in Result Variability) Desirable Performance Criteria for Bias BA = <0.125 (CVi2 + CVg2)1/2 (2% increase in # people outside RI) BA = <0.250 (CVi2 + CVg2)1/2 (16% “ “ “ “ RI) BA = <0.375 (CVi2 + CVg2)1/2 (34% “ “ “ “ RI)

Performance Criteria - Biological Variation The precision and bias goals can be combined to produce a desirable total error performance goal (TEa). TEa <0.250 (CVi2 + CVg2)1/2 + 1.65(0.50 CVi) In the case of sodium, the desirable TEa performance goal is 0.9%

“Six Sigma Quality Design & Control” Second Edition Desirable Precision and Requisite QC for Laboratory Measurement Processes James O. Westgard, PhD Westgard QC, Inc.

Six Sigma “ A six sigma process is world class – with an expected error rate of 3.4 DPM (defects per million)” Requires that method bias plus 6s < total error allowable Total cholesterol (TEa = 9%) Bias from EQA/PT program = 1% CV = 1.5% Calculate sigma value: TEa – Bias /CV Sigma = 5.3 The performance of this method requires QC of only 2 controls per run with 3s or 2.5s control limits

Benchmarking with Six Sigma “ A six sigma process is world class – with an expected error rate of 3.4 DPM (defects per million)” “Five sigma – is the initial goal for quality improvement - method requires QC of only 2 controls per run with 3s or 2.5s control limits” “Four sigma – method requires QC of 4 controls per run with multi-QC rules” (4000 DPM – 0.4% error rate; airline baggage handling – 4.15) “Three sigma – minimal acceptable quality - method requires maximum QC by laboratory – often a level that is not affordable for routine operations”

Failed PT – Now What? Manager to staff: “To address this problem we must use root cause analysis. I’ll begin the discussion by saying that its not my fault”

Problem Solving – Root Cause Analysis Define the problem Gather data and evidence – pre-analytical, analytical, post-analytical Ask "why" - identify the causes associated with each step in the process that contributed to the problem Classify causal factors that relate to an event in the process, and root causes, that if applied can be agreed to have interrupted the process Identify corrective action(s) that will with certainty prevent recurrence of the problem Identify solutions with group consensus that will effectively prevent recurrence of the problem with reasonable certainty Implement the recommended root cause correction Confirm effectiveness by auditing the recommended solutions

Pre-analytical Sample received on time Sample stored properly Sample condition Sample matrix appropriate Sample ID logged properly Tests to be ordered Method registration – check Reagent dating, shelf-lie, storage Reporting Units Check maintenance logs, trouble shooting logs Check non-conformities for the method involved

Analytical Method change recently Change in reagent lot Recent maintenance on the analyzer IQC running mean – has it shifted Has CV of the method changed? Is current CV meeting the set performance standard

Post-analytical Results filing – correct result for correct vial Result reported to correct number of decimals Results filed after deadline Results changed after reporting RI reported with result Comments reported

David W Seccombe MD, PhD, FRCPC Thank you David W Seccombe MD, PhD, FRCPC Department of Pathology and Laboratory Medicine, University of British Columbia Canadian External Quality Assessment Laboratory (CEQAL) DigitalPT International EQA Collaboration (DigitalPT) dseccombe@ceqal.com 604-222-3907