1. BMTA July 2005: 1 Valid Analytical Measurement Studies of Proficiency Testing scheme performance S Ellison LGC Limited, Teddington The work described in this paper was supported under contract with the Department of Trade and Industry as part of the Valid Analytical Measurement programme

2. BMTA July 2005: 2 or 63 routes to the wrong result... and what to do about it

3. BMTA July 2005: 3 Introduction PT in analytical chemistry Why study mistakes? How does the UK do? –PT results compared to international performance What goes wrong? (and why) –Web-based study of causes of poor PT scores

4. BMTA July 2005: 4 PT in analytical chemistry - organisation Typical rounds comprise: –test sample preparation, characterisation and distribution –analysis by participants –data collection and processing –preparation and distribution of the report Frequency –Typically 6-12 rounds per year Analytes (measured quantities) –1-30 per sample per round Participants –Typically 30-100 per round, but strongly scheme-dependent

5. BMTA July 2005: 5 The aims of proficiency testing Primary aim: “To provide the infrastructure for a laboratory to monitor and improve the quality of its routine analytical measurements” Other aims –Provide information on the state-of-the-art in analytical measurements –Compare performance of analytical methods –Assist a laboratory in the validation of new methods

6. BMTA July 2005: 6 Principle of performance assessment Observed error –difference between laboratory result (x) and assigned value (X) ‘Target range’ –usually a standard deviation ( ) or uncertainty Compare….. …..using an acceptability criterion

7. BMTA July 2005: 7 Performance Scoring: z-scores xsubmitted resultX assigned value standard deviation for proficiency assessment  Z   2Satisfactory performance 2  Z   3Questionable performance  Z   3Unsatisfactory performance Interpretation of z is consistent across schemes but depends on

8. BMTA July 2005: 8 Typical analytical performance data. Collected food analysis data: Various analytes

9. BMTA July 2005: 9 PT data for benchmarking Three studies of UK performance –Clinical –Food –Environment Clinical: Backed by IMEP-17 study (20 analytes: 35 countries) Food: FAPAS PT scheme data (6 representative analytes; 2000 labs; ca. 250 countries and regions) Environment: CONTEST and CoEPT project data

10. BMTA July 2005: 10 UK performance: Clinical UK performance: Consistent with others; Rarely poor

11. BMTA July 2005: 11 UK Performance: Food GMO measurement

12. BMTA July 2005: 12 UK Performance: Food Aflatoxins

13. BMTA July 2005: 13 UK Performance: Food Pirimphos-Me (pesticide residue) UK Other

14. BMTA July 2005: 14 Problem analytes: Arsenic UK

15. BMTA July 2005: 15 Problem analytes: Arsenic All countries

16. BMTA July 2005: 16 UK Performance: Environment Total polycyclic aromatics

17. BMTA July 2005: 17 UK Performance: Summary Broadly comparable to other countries No problems unique to the UK Some problems (e.g. Arsenic) shared with other countries

18. BMTA July 2005: 18 Part 2: Causes of error VAM Project KT2.4/3: Causes of poor PT performance Aim: Study "…the principal causes of poor performance in laboratories and... the effectiveness of the steps taken by Participants in PT to improve the reliability of their results” Methodology –Web-based questionnaire –Focussed on documented problems identified via PT scores –Lead questions with follow-up for positive responses

19. BMTA July 2005: 19 Why study poor scores in PT? Why PT? –PT participants are already committed to quality improvement –Participants follow up poor PT scores Why only poor scores? –Acceptable scores give poor information about problems –Correlation of scores with general methodology is not very effective –Every good lab has documented problems and corrective actions

20. BMTA July 2005: 20 Top causes of poor scores Sample preparation Equipment problem Human error Calibration Selection of method Calculation error Reporting problem 111 respondents 230 causes

21. BMTA July 2005: 21 Top causes of poor scores Sample preparation Extraction/recovery Dilution to volume

22. BMTA July 2005: 22 Top causes of poor scores Equipment problem Equipment failure

23. BMTA July 2005: 23 Top causes of poor scores Human error Training/experience Transcription error Reporting error

24. BMTA July 2005: 24 Top causes of poor scores Calibration No reference material Defective RM Incorrect procedure Calibration range Calibration

25. BMTA July 2005: 25 Top causes of poor scores Reporting problem Reporting problems Value correct but not in customer units Transcription/typographical error Incorrect units

26. BMTA July 2005: 26 Top causes of poor scores Calculation error Commercial software problem Spreadsheet problem Spreadsheet user error Calculator error Arithmetic error Value mis-entered Software mis-applied Other

27. BMTA July 2005: 27 Corrective action Training New procedures Revalidation Method documentation New equipment Additional calibration Method change RM change Other Detailed information showed problem-specific responses

28. BMTA July 2005: 28 Corrective action - efficacy No significant difference in efficacy across different corrective actions Only 50% of actions were marked as ‘fully effective’ Monitoring of efficacy tended to use local/immediate methods –Monitor QC results –Internal audit

29. BMTA July 2005: 29 Causes of error: Summary Most PT errors were caused by basic lab operations –Incorrect dilution to volume –Transcription and reporting errors –Data and spreadsheet formula entry errors Equipment failure is perceived as a problem Extraction/recovery problems important Commercial software faults caused no problems Corrective actions are problem-specific and ‘multifactor’ –More than one action generally required.

30. BMTA July 2005: 30 Conclusions UK analytical labs perform similarly to international partners, and share similar problems The most common cause of PT performance failures are not technical, but simple human errors such as incorrect volumetric operations and transcription errors Time to look harder at human factors? Study web page: via http//www.vam.org.uk - surveys link