1. Data of Known and Documented Quality
Investigating Improper Laboratory Practice: Tools used by the Contract Laboratory Program
Data of Known and Documented Quality
Presented By: Charlie Appleby
Office of Superfund Remediation and Technology Innovation
Analytical Services Branch

2. U.S. Environmental Protection Agency
Agenda
CLP Infrastructure
CLP’s Tool Set
Warning Signs / Examples
Discussion
I will be talking about what I do every day, and that is monitoring laboratory data quality through the use of a variety of tools to achieve the mission of the CLP. EPA’s mission is protection of human health and the environment; CLP’s mantra is data of known and documented quality.
Some items to mention:
CLP’s cost effectiveness provides an economy of scale for laboratory analyses; eliminates duplication of efforts for procurements, sample tracking, and invoice processing.
CLP’s central financial control systems collect and disseminate financial information to EPA management for budgetary and cost recovery activities
CLP documentation of data quality elements has become a standard in the industry.
The focus of the CLP on extensive documentation and evaluation of data quality, led early on to complaints about how slowly the data turned around. However, we have moved into the 21st century with the electronic data delivery system (EXES/EDM), which facilitates the timely processing of laboratory EDDs, including automated evaluation of data package completeness, review of basic QC parameters, qualification of data as necessary, and generation of a uniform data deliverable that is available in several formats and importable into Regional LIMS, Scribe, and EQUIIS.
August 10, 2017
U.S. Environmental Protection Agency

3. U.S. Environmental Protection Agency
CLP Infrastructure
Level 4 Data Package
Performance Testing and Lab Reference Samples
On-site Audits
Staged Electronic Data Deliverable (SEDD)
Electronic/Manual Contract Compliance Screening (CCS)
Electronic NFG Data Review by EXES
Data Package and Electronic Media Audits
National Functional Guidelines for Data Review (NFG)
Eyes-on Data Validation
We will go through the importance of these tools in the next few slides
August 10, 2017
U.S. Environmental Protection Agency

4. U.S. Environmental Protection Agency
CLP’s Tool SET
Contract Requirements
All electronic files associated with SDG
New Electronic Tools
Electronic Data Mining Using EXES
Laboratory Data Processing Software
Data Investigation Tools
Communication with Regional reviewers
Communication with OIG
We have already toughened up the procurement process and have made some adjustments to the language in our contracts, including those recently awarded, to limit vulnerabilities. The new contracts include new contract base period requirements for meeting method and deliverables requirements prior to receipt of field samples, as well as Contract criteria requiring better method performance, like GC resolution criteria, and a stage III SEDD for the inorganic program.
ASB is considering new EXES Data tools, including scans used in the case study above, of MIs of QC samples that subsequently were only a few % within criteria. Another use of surveying manual integration information ASB has used was to identify labs that were using manual integration in lieu of regular maintenance.
The amount of time spent looking at data packages has decreased as electronic data review has become more sophisticated. But a combination of electronic data evaluation tools and good old eyes on the data is needed. All of us who review data have to be ever vigilant. If you see something in the data that doesn’t look right, say something.
August 10, 2017
U.S. Environmental Protection Agency

5. Data Mining Tools
August 10, 2017

6. Warning Signs - Organic
Multiple manual integrations performed on QC samples or calibrations
Manual Integration in lieu of instrument maintenance
Poor peak shapes
Short run times
Calibration Outliers
GC/MS tune issues
Inappropriate manual integrations: peak shaving and enhancement
Unnecessary baseline adjustment
Our office identified potential data anomalies through a routine data package/electronic media audit. The issue identified was potentially inappropriate manual integrations. It appeared the laboratory had manipulated data to make it appear to meet CLP quality assurance criteria, when otherwise it would have failed. ASB initiated routine processes for handling audit deficiencies with requesting laboratory response and corrective action.
August 10, 2017
U.S. Environmental Protection Agency

7. Warning Signs - Inorganic
Time gaps in calibration sequences
Interference Check Sample outliers
Making up % solids results to meet contract requirements
Poor precision caused by dirty systems
Running rinse blanks before QC samples
Expired standards or gaps in traceability
Our office identified potential data anomalies through a routine data package/electronic media audit. The issue identified was potentially inappropriate manual integrations. It appeared the laboratory had manipulated data to make it appear to meet CLP quality assurance criteria, when otherwise it would have failed. ASB initiated routine processes for handling audit deficiencies with requesting laboratory response and corrective action.
August 10, 2017
U.S. Environmental Protection Agency

8. U.S. Environmental Protection Agency
Things Worth Noting
Data processing methods tailored to make QC pass
Final results just within technical acceptance limits
Inappropriate inclusion of area including adjacent noise and even other peaks for area enhancement
A large number of manually integrated results just within technical acceptance limits
Reprocessing of raw data produces acceptable integrations that do not meet technical acceptance criteria
Electronic audit trail files show multiple integrations of many analytes
Our office identified potential data anomalies through a routine data package/electronic media audit. The issue identified was potentially inappropriate manual integrations. It appeared the laboratory had manipulated data to make it appear to meet CLP quality assurance criteria, when otherwise it would have failed. ASB initiated routine processes for handling audit deficiencies with requesting laboratory response and corrective action.
August 10, 2017
U.S. Environmental Protection Agency

9. Examples of Improper Manual Integration
Here is what you may notice from looking at the data for a calibration verification standard. The typical chromatographic data system will sense a change in the slope of the detector response and integrate the peaks as shown on the left. But what you see is the one on the right, and you ask yourself, “why did they do that?” If it happens only once, you may be tempted to qualify the associated sample data for delta BHC as estimated and move on. But you should stay vigilant! When you see it repeatedly, only in standards or QC samples (which should be problem-free), and always with the result that the peak passed criteria when it otherwise would have failed, that is another story.
Other examples include: Selectively background subtracting spectra from other peaks to make tuning criteria pass in GC/MS analysis.
Delta BHC Delta BHC
12.45 min ng/ml min ng/ml m
response = response =
%D = %D = 19.6%

10. Improper Manual Integration
Printed from Laboratory-provided audit trail file
We are now looking at the audit trail for the peak we saw on the previous slide. You can obtain the audit trail by asking the lab for all raw and processed data files associated with the analysis. This is a text file that is associated with each injection on a GC or GC/MS instrument.
On slide 49, we saw that the initial percent difference between the delta-BHC peak and the initial calibration was 23.2%. The method performance criterion is < = 20%. From this trail, we can see that the peak was manually re-integrated four times in 12 seconds to create the result which meets the 20% criterion.

11. Proper Integration Before (Automatic Integration)
Abundance
Ion ( to ): B05048_Before.D\data.ms
110000
Ion ( to ): B05048_Before.D\data.ms
Ion ( to ): B05048_Before.D\data.ms
Before (Automatic Integration)
VSTD00549 (opening CCV)
100000
1,2,4-Trichlorobenzene
90000
Produced by the QATS auditor from laboratory-submitted electronic data by quantitating the data file with the laboratory-supplied method file
80000
15.70| 0
70000
60000 |
50000
Start Integration |
End Integration
40000
30000 |
End ID Window
20000
Start ID Window |
10000 | 7d 4d
3d| 1|
| 2d
5d 6d 8d |
Time-->

12. Improper Manual Integration
Abundance
Ion ( to ): B05048_Submitted.D\data.ms
110000
Ion ( to ): B05048_Submitted.D\data.ms
Ion ( to ): B05048_Submitted.D\data.ms
After (Manual Integration)
1,2,4-Trichlorobenzene
100000
VSTD00549 (opening CCV)
90000
Reproduced from the laboratory-submitted electronic data
80000
15.70| 0
70000
60000 |
50000
Start Integration
End Integration |
40000
30000 |
End ID Window
20000
Start ID Window |
10000 | 7d 4d | 3d 1| 2d
5d 6d
8d | |
Time-->
15.35
ENCLOSURE 1B

13. Improper Manual Integration
Printed from the laboratory-submitted audit trail file
Modified : Sat Aug 13 12:25: Event : Manual Integration
Message : Changed peak amount for 1,2,4-Trichlorobenzene from ug/L to ug/L
QuantFile: CLPT05044.RES Severity: 1
Modified : Sat Aug 13 12:26: Event : Manual Integration
Message : Changed peak amount for 1,2,4-Trichlorobenzene from ug/L to ug/L
QuantFile: CLPT05044.RES Severity: 1
Modified : Sat Aug 13 12:26:
Event : Manual Integration
Message: Changed peak amount for 1,2,4-Trichlorobenzene from ug/L to ug/L
QuantFile: CLPT05044.RES Severity: 1

14. Proper Manual Integration
An enlargement of the SICPs of the manually integrated 2378-TCDF ions m/z and m/z from the ICAL standard CS12I as presented in the hardcopy data before manual integration.

15. Improper Manual Integration
An enlargement of the SICPs of the manually integrated 2378-TCDF ions m/z and m/z from the ICAL standard CS12I as presented in the hardcopy data after manual integration.

16. Obtained from hard-copy data package
Example of Time Travel
Obtained from hard-copy data package
In this example, a CCV from a semivolatile GC/MS run, everything appears to be in order until you pay attention to the times. This analysis was quantitated before it was injected!

17. Improper Laboratory Practices
Obtained from hard copy data package
For this example, we obtained the audit trail text file. The quant report looks fine, the times make sense. However the reviewer, who was looking for the audit trail from the run on the previous slide, found that the audit trail file was created for this standard two days prior to the injection time!
If you do see something like this, you should not assume it is accidental, but you should investigate further, and consider providing it to someone who can open an official investigation. In the case of EPA, contact the Inspector General’s office.
Printed from the laboratory-submitted audit trail file

18. Discussion

19. Thank you!
Charlie Appleby, CLP COR and Program Manager for Hi-Res and Organic Methods, ASB:
Shari Myer, Analytical Services Branch Chief