1. Precision, Bias, and Total Error (Accuracy)
Guest Speaker:
Bill Frietsche
US EPA

2. Overall Course Overview:
April 7: QA Systems, EPA definitions, PQAOs and common sense – Mike Papp
April 14: Routine Quality Control and Data Management (1-pt QC, flow rate, and instrument stability checks) – Travis Maki
April 21: Audits Overview (NPAP, PEP, Annual PE, Flow Rate Audits) – Jeremy Howe
April 28: Calculating Bias and Precision and AQS reports – Bill Frietsche
May 5: 40 CFR 58 App. A- Gaseous Pollutants – Glenn Gehring
May 12: 40 CFR 58 App. A- Ozone – Brenda Jarrell
May 19: 40 CFR 58 App. A- PM filter and continuous methods –Brandy Toft

3. …the difference between your answer and the “truth”
Error
…the difference between your answer and the “truth”
Two components of total error (accuracy):
Bias (jump)
Precision (wiggle)

4. Precision Error Some imprecision is unavoidable
Sometimes up, sometimes down– “random”
Difference divided by best estimate of the truth
“truth” is:
For gas QC checks: known conc.
For PM flow rate: audit (known) FR
For PM2.5 collocated: their average

5. 1st way of assessing precision:
Repeated measurement of the same thing
di = the difference between a known value and your value (flow rate, conc., voltage, that should remain the same)
Sequence of check (date, time, check number)

6. 2nd way of assessing precision:
Pairs of simultaneous measurements (collocated instruments)
di (difference between collocated values)
date

7. Bias = calibration error:
Can be sudden or due to slow drift over time:
di = the difference between the true value (audit value) and the analyzer
date

8. Accuracy = total error Includes both bias and precision
At any one time, an audit value could be close to the analyzer value due to precision errors in both
Or, audit value could be far from analyzer value due to precision errors adding
Audits estimate Accuracy
EPA uses audit results to verify that ongoing QC checks do represent total error

9. Accuracy = total error:
Auditor’s see only one day, while your QC checks see overall big picture

10. Precision and total error both are estimated from the QC checks:
1-pt QC checks no longer called precision checks, because the results are used (by YOU) to calculate both precision and to estimate total error
Each check <= 7% is the CRITICAL criteria for each set of data since last passing check
The audits basically verify your precision and bias that have been calculated all along from your QC checks (concentration and flow rate, for PM)

11. QC Checks are used to estimate precision*:
Method
Pollutants
Frequency
MQO
One-Point QC
SO2, NO2, O3, CO
Every 2 Weeks
O3 :
Precision = 7%
SO2, NO2, CO :
Precision = 10%
Flow Rate Verification (QC check)
PM10 (lo-Vol) PM2.5
Once every 4 weeks
<= 4% of Standard
PM10 (high-Vol), TSP
Once per quarter
<= 7% of Standard
Collocated Sampling
PM10, TSP, PM2.5
15% of Network Every 12 Days
Precision CV= 10% (which means RPD =14%)
*and total error, more on this later….

12. Why is tracking precision important?
Estimates your overall system uncertainty, if you were perfectly calibrated
Changes over time can point to operator error, lab error, poorly written procedures, equipment/standards going bad
Changes can be fixed sooner rather than at audit
EPA compares precision by site and agency

13. di Control charts: Useful to track precision (or bias) over time
See this example (and you fill in your values) at:
Another example in the DASC tool: di

14. Control chart in PBDASC tool:
The DASC tool automatically writes your values from the columns into the chart:
Measured value is your analyzer
Audit value is the known di
You may want to add dates to the x-axis, or use my example if you want to set up control charts in your program

15. EPA estimates precision:
In their annual QA reports, from your RP transactions in AQS: di

16. Invalidation criteria for NAAQS:
Recommendations in redbook say:
Critical criteria invalidate every hour that is not met
Operational criteria means something probably wrong, go check it
Systematic criteria mean as a set (day? year?) data is not usable for NAAQS, but individual hours or more may be valid

17. Critical QC invalidation criteria for gas:
(NO2 and SO2 are the same)
THERE IS A REASON that the checks are no longer called precision checks: the results are used to calculate both precision and bias.

18. Critical invalidation criteria for PM:
Method
Pollutants
Frequency
MQO
Flow Rate Verification (QC check)
PM10 (lo-Vol), PM2.5
Once every 4 weeks
<= 4% of Standard
PM10 (high-Vol), TSP
Once per quarter
<= 7% of Standard

19. Operational precision invalidation criteria for PM:
Method
Pollutants
Frequency
MQO
Collocated Sampling
PM10, TSP, PM2.5
15% of Network Every 12 Days
Precision as CV < = 10%
(meaning the relative percent diff must be less than 14% for conc > 3 ug/m3)

20. Total Error
Keep the bias component minimized by calibrating and verifying your equipment against a standard
Keep precision component low by consistency
You can work to keep bias down, while precision is often out of your control below a certain limit
EPA calculates from your RA transactions

21. Total error requirements for NAAQS are all operational criteria (investigate, but not necessarily invalid):
Method
Pollutants
Frequency
MQO
Annual Performance Evaluation (Audit)
SO2, NO2, O3, CO
Once per Year
<= 15% for each audit concentration— OPERATIONAL
Semi-Annual Flow Rate Audit
PM10, PM2.5
Every 6 Months
<= 4% of Standard
OPERATIONAL
PM10 (high-Vol), TSP
<= 10% of Standard
PM2.5 PEP Program
NPAP
PM2.5
Quarter
Year (see QA Requirements.xls)
Bias = 10% -- OPERATIONAL

22. Bias, precision, and total error:
Bias: systematic difference, or “jump”
Precision: random error, or “wiggle”
Accuracy = total error, a combination of both
Simple temp bath illustrations at:
A101_Resources/AllDownloadableMovies/

23. How do I calculate precision and total error?
DASC (Data Assessment Statistical Calculator)
s/QA101_Resources/DASC%20EPA%20Prec %20and%20Bias%20Calculator/
AQS: Data Quality Indicators Report (AMP255)

24. Before we cut to Bill: Minimize bias by regular calibrations
both accuracy (total error) and precision are first estimated by di on an ongoing basis, by you
Audits verify total error estimates
Course website: A101_Resources/
Our s:
Bill Frietsche:

25. AMP255 and other AQS QA tools

26. Appendix A to Part 58
Table A-2 of App A
AQS TTN web site
AQSP&A Spreadsheet
Discoverer
AMTIC web site
AQS Helpline
AQS AMP255 Data Quality Indicator Report

27. Define assessments for each criteria pollutant 1 Pt QC check for gases
Appendix A to Part 58
Regulations that define QA reporting requirements for criteria pollutants
Define assessments for each criteria pollutant
1 Pt QC check for gases
Annual performance evaluation for gases
Flow rate verification for particulate matter
Semi-annual flow rate audit for particulate matter
Collocated sampling requirements for particulate matter
Pb audit strips for laboratory analysis QA
Performance Evaluation Program for PM fine, PM coarse, and Pb
Formerly called precision and accuracy data
– still use these terms on AQS transactions

28. This document available at: http://itep68. itep. nau

29. AQS TTN web site http://www.epa.gov/ttn/airs/airsaqs/
AQSP&A Spreadsheet
AQS Discoverer
AMTIC web site
AQS Help Line
EPA (4372)

30. Example run of AMP255 shown onscreen

31. AQS Precision and Accuracy Conclusions:
All QC data helps EPA balance your costs of QC with needed information to protect health
Questions to AQS are welcomed
EPA committed to improving user friendliness
Course website: A101_Resources/
Our s:
Bill Frietsche: