1. User verification of performance Zahra Khatami/Robert Hill BHR University Hospitals/Sheffield Teaching Hospitals Focus 2013

2. Aim and objective Provide an overview of measurement uncertainty Offer practically achievable procedures Recommend the minimum testing requirements also ensuring reliability Work through pre-programmed spreadsheets

3. Communication language “International Vocabulary of Metrology” (VIM) JCGM 2008 www.bipm.org/utils/en/pdf/JCGM (BIPM, IEC, IFCC, IUPAC, IUPAP, ISO, OIML, ILAC)

4. VIM definition Verification: Provision of objective evidence that a given item fulfils specified requirement Example: Confirmation that performance properties or legal requirements of a measuring system are achieved

5. Common quandary Which performance criteria? What type of material? How many samples/measurements? How many days? What statistical tools?

6. Performance properties In the context of clinical chemistry this equates to the uncertainty associated with the measurement

7. VIM definition: Measurement Uncertainty Non-negative parameter characterizing the dispersion (imprecision) of the quantity values being attributed to a measurand (an interval of values within which the true value lies with a stated probability)

8. VIM definition: Trueness Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value (in absence of reference method the difference between specified methods)

9. Which performance criteria? Precision (Imprecision) Trueness (Bias)

10. Estimating the measurement uncertainty Guide to the expression of uncertainty measurement (GUM) Bottom–up approach which derives the uncertainty of a measurement result by combining the uncertainties related to the uncertainty sources of the measurement process

11. Estimating the measurement uncertainty Guide to the expression of uncertainty measurement (GUM) Alternatively Top-down approach which uses the reproducibility as uncertainty estimate

12. Reference publications EN/ISO 15189 (Requirements for quality & competence specific to the quality management system requirements particular to medical Laboratories) CLSI (Clinical Laboratory Standards Institute) Former NCCLS IFCC ECCLS (European Council for Clinical Laboratory Standards) obsolete since mid 90’s

13. Reference publications CLSI EP15-A2 (User verification of performance for Precision and Trueness 2005) CLSI EP9-A2 (Method Comparison and Bias Estimation Using Patient Samples 2002)

14. Reference publications Thorough document Lengthy document Large number of tests Detailed statistical processing Lack of accessible pre-programmed spreadsheets

15. Authors Zahra Khatami Robert Hill Catherine Sturgeon Edward Kearney Peter Breadon Anders Kallner

16. Aim Provide practical recommendations Minimum requirements/ensuring reliability User friendly Full pre-programmed spreadsheets Free access for all Regular review

17. Measurement verification in the clinical Laboratory: A guide to assessing analytical performance during the acceptance testing of methods (quantitative examination procedures) and /or analysers http://www.acb.org.uk/An%20Ver/Measurement1.asp

18. Assumptions Adequate verification by manufacturer, accessible by the user (IVD directive) Established performance specifications (precision, bias, linearity, measurement interval, interference, etc)

19. Imprecision What type of material? Patient Pool or IQC material How many measurements? Five replicates at two concentrations How many days? Five days

20. The statistical test ANOVA Tests for a difference between the means of a number of groups of observations It partitions the total variance into the components: variation between the within and between each series of measurements It offers an estimate of within and between series variation

21. Refer to Calcium data word file

26. Bias: Patient comparison How many measurements? 20 patient samples How many days? Paired samples should be measured within a close time interval

27. The statistical tests T-test Ordinary linear regression Deming regression Difference plot

28. Refer to amylase word file

32. Bias: using reference material What type of material? Reference material with assigned value (CRM) How many measurements? Two concentrations, duplicate measurements How many days? 3-5 occasions/days What statistical tool? T-test

34. Bias: using EQA material What type of material? Material with peer group values from EQA organizers How many measurements? 7-10 concentrations, duplicate measurements How many days? N/A The statistical tool? T-test

36. Thank you

37. Everything should be made as simple as possible, but no simpler