Guide to Selecting Qualified WET Laboratories Robert N. Brent, Ph.D. DynCorp Science and Engineering Group

Why is Lab Selection Important?  You, the permittee, are responsible for: Meeting monitoring frequency requirements Certifying the quality of data – “I certify under penalty of law …the information submitted is, to the best of my knowledge and belief, true, accurate, and complete…” Living with the test results – Test failures can initiate additional testing, TIEs, TREs, enforcement actions, fines, public scrutiny

Why is Lab Selection Important?  You are the one paying In a 1998 WERF Study – C.dubia chronic test: $100 - $2,300 – Fathead minnow chronic test: $238 - $5,500 With multiple outfalls, monthly testing requirements, and multi-species monitoring, test costs can be significant – 3 outfalls X 12 samples/yr X 3 species X $1000/test = $108,000 Know what you are paying for!

Why is Lab Selection Important?  Quality does matter There is a wide range of laboratory performance – Ex: EPA’s Variability Guidance Document Percentiles of Intralaboratory CVs (coefficient of variation) for C. dubia Chronic Test

A Tale of Two Labs Ex: WET Interlaboratory Variability Study

A Tale of Two Labs Ex: WET Interlaboratory Variability Study $936 / test $500 / test

Obstacles to Selecting Qualified Labs  Procurement Regulations Many permittees are required to select lowest responsive and responsible bidder This doesn’t always mean selecting lowest bidder – Must be responsive to everything that you request in the solicitation – Must be responsible for delivering exactly what you specify in the contract Solution: You determine the exact specifications for what you want – Government Ex: $600 hammer – WET Ex: lab must demonstrate intralab variability of <27% Talk to your procurement specialist about incorporating detailed specifications into the lab solicitation process

Obstacles to Selecting Qualified Labs  Insufficient knowledge and resources for identifying qualified laboratories Solution: This workshop will provide – Practical tools for evaluating laboratory quality – Things to look for in a qualified lab – Questions to ask – Sample test data sets to quiz your lab

General Evaluation Criteria  Capacity Will the lab be able to test your samples when you need them tested? What is the labs maximum and typical capacity (tests per week)? – Look for maximum capacity to greatly exceed typical capacity – Quality usually decreases when operating near capacity limits What limits capacity? (space, staff, organisms, equipment) – Look for limits that can be easily remedied

General Evaluation Criteria  Staff What is the level of experience and education of staff from top to bottom? – Remember: it is the staff at the bottom (technicians) that will have the most contact and interaction with your samples – Experienced and well-trained staff is critical to consistently generating high quality data What is the turnover rate? – Look for low turnover at the top – Turnover at the bottom may be high, but look for rigorous training program

General Evaluation Criteria  Organizational Structure What is the chain of command? Who will you be interacting with? – Look for a dedicated account manager – Look for a laboratory contact as well  Certification Does the lab carry any State or national certifications? – Not all States have lab certification programs, but beware of labs without certification in a State where there is a certification program

General Evaluation Criteria  Historical Performance How many tests does the lab run annually? – Look for 50 – several hundred (for common test methods) What percentage of tests were successfully completed (met TAC) without retesting? – Look for % How many other clients do they have with similar waste streams (municipal, industrial, etc.) to yours? – Familiarity with similar waste streams may lead to better, quicker problem resolution

General Evaluation Criteria  Reporting Does the lab’s reporting standards and format meet the requirements of your regulatory authority? Does the lab’s reporting format meet your needs – Look for simple but detailed reports s Ex: You should be able to find the test result (NOEC, IC25, etc.) within 60 seconds s Ex: Report should be detailed enough for someone to completely recalculate the test results s Ex: You should be able to find the temperature of treatment A on Day 4 of the test

General Evaluation Criteria  Awareness Is the lab aware of recent developments in WET testing? – Look for lab to be aware of: s Proposed WET method changes and upcoming new versions of WET manuals s WET Method Guidance Document (EPA 821/B-00/004) s WET Variability Guidance Document (EPA 833/R-00/003)

Quality Control Evaluation Criteria  Organism Source and Quality Where does the lab get their organisms? – In-house cultures vs. commercial suppliers – Look for consist source How many organisms are available on a daily basis? – Look for capacity to exceed usage What QC measures are used to assess organism health? – Look for reference toxicant testing and other measures (reproduction and survival in cultures) How often are culture crashes experienced?

Quality Control Evaluation Criteria  Dilution Water Source and Quality What dilution water types does the lab use? Where does the dilution water come from or how is it prepared? – Look for consistent high quality source – Look for experience with different dilution water types How does the lab assess dilution water quality? – Look for chemical testing and toxicity testing of new dilution water batches

Quality Control Evaluation Criteria  Reference Toxicant Control Charts Reference toxicant control charts are one of the best ways to assess laboratory performance (if you know what to look for) Labs are required to conduct monthly reference toxicant tests Test results are plotted on a chart to show variability over time

Evaluating Reference Toxicant Control Charts  Look for control charts of LC50 (for acute methods) or IC25 (for chronic methods), not NOECs  Look for 20 points  Look for curved limits

Evaluating Reference Toxicant Control Charts  Look at scale Different Scale Same Scale

Evaluating Reference Toxicant Control Charts  Look at CV CV = Standard deviation / Mean * 100% Compare CV to national percentiles (Variability Guidance Doc Table B-1) CV = 17% CV = 40%

Evaluating Reference Toxicant Control Charts  Look for results beyond control limits 1 in 20 will exceed limits by chance alone Beware of >2 exceedances in 20

Quality Control Evaluation Criteria  Control Charts for Additional QC Measures Look for control charts of: – Control performance over time (i.e., control reproduction) – Control CV over time – PMSDs (percent minimum significant difference) over time Compare with national percentiles from Variability Guidance Document (Tables B-7 to B-8)

Evaluation of PMSD Control Charts  Look for PMSD mean versus national percentiles  Look for PMSDs that exceed 90 th percentile

Quality Control Evaluation Criteria  Statistical Analysis and Endpoint Calculation Can the laboratory correctly perform the recommended statistics and properly calculate test endpoints? – EPA found that a large percentage of laboratories in the WET Interlaboratory Variability Study made one or more errors in the calculation of test results Test laboratories with sample data sets – 3 sample data sets provided – Ask laboratories to calculate LC50, survival NOEC, sublethal NOEC, and IC25 using the recommended statistical flowcharts and concentration-response evaluation guidance (where appropriate)

Sample Test Data Set #1

Sample Test Data Set #2  Requires use of concentration-response guidance

Sample Test Data Set #3  Requires use of concentration-response guidance

Conclusions  It’s your money and your reputation at stake: Choose a quality laboratory  When you find a quality laboratory, stick with them  When you don’t know, ask