User verification of performance Zahra Khatami/Robert Hill BHR University Hospitals/Sheffield Teaching Hospitals Focus 2013

Aim and objective Provide an overview of measurement uncertainty Offer practically achievable procedures Recommend the minimum testing requirements also ensuring reliability Work through pre-programmed spreadsheets

Communication language “International Vocabulary of Metrology” (VIM) JCGM (BIPM, IEC, IFCC, IUPAC, IUPAP, ISO, OIML, ILAC)

VIM definition Verification: Provision of objective evidence that a given item fulfils specified requirement Example: Confirmation that performance properties or legal requirements of a measuring system are achieved

Common quandary Which performance criteria? What type of material? How many samples/measurements? How many days? What statistical tools?

Performance properties In the context of clinical chemistry this equates to the uncertainty associated with the measurement

VIM definition: Measurement Uncertainty Non-negative parameter characterizing the dispersion (imprecision) of the quantity values being attributed to a measurand (an interval of values within which the true value lies with a stated probability)

VIM definition: Trueness Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value (in absence of reference method the difference between specified methods)

Which performance criteria? Precision (Imprecision) Trueness (Bias)

Estimating the measurement uncertainty Guide to the expression of uncertainty measurement (GUM) Bottom–up approach which derives the uncertainty of a measurement result by combining the uncertainties related to the uncertainty sources of the measurement process

Estimating the measurement uncertainty Guide to the expression of uncertainty measurement (GUM) Alternatively Top-down approach which uses the reproducibility as uncertainty estimate

Reference publications EN/ISO (Requirements for quality & competence specific to the quality management system requirements particular to medical Laboratories) CLSI (Clinical Laboratory Standards Institute) Former NCCLS IFCC ECCLS (European Council for Clinical Laboratory Standards) obsolete since mid 90’s

Reference publications CLSI EP15-A2 (User verification of performance for Precision and Trueness 2005) CLSI EP9-A2 (Method Comparison and Bias Estimation Using Patient Samples 2002)

Reference publications Thorough document Lengthy document Large number of tests Detailed statistical processing Lack of accessible pre-programmed spreadsheets

Authors Zahra Khatami Robert Hill Catherine Sturgeon Edward Kearney Peter Breadon Anders Kallner

Aim Provide practical recommendations Minimum requirements/ensuring reliability User friendly Full pre-programmed spreadsheets Free access for all Regular review

Measurement verification in the clinical Laboratory: A guide to assessing analytical performance during the acceptance testing of methods (quantitative examination procedures) and /or analysers

Assumptions Adequate verification by manufacturer, accessible by the user (IVD directive) Established performance specifications (precision, bias, linearity, measurement interval, interference, etc)

Imprecision What type of material? Patient Pool or IQC material How many measurements? Five replicates at two concentrations How many days? Five days

The statistical test ANOVA Tests for a difference between the means of a number of groups of observations It partitions the total variance into the components: variation between the within and between each series of measurements It offers an estimate of within and between series variation

Refer to Calcium data word file

Bias: Patient comparison How many measurements? 20 patient samples How many days? Paired samples should be measured within a close time interval

The statistical tests T-test Ordinary linear regression Deming regression Difference plot

Refer to amylase word file

Bias: using reference material What type of material? Reference material with assigned value (CRM) How many measurements? Two concentrations, duplicate measurements How many days? 3-5 occasions/days What statistical tool? T-test

Bias: using EQA material What type of material? Material with peer group values from EQA organizers How many measurements? 7-10 concentrations, duplicate measurements How many days? N/A The statistical tool? T-test

Thank you

Everything should be made as simple as possible, but no simpler