1. Calculating Statistics: Concentration Related Performance Goals James W. Boylan Georgia Department of Natural Resources PM Model Performance Workshop Chapel Hill, NC February 11, 2004

2. Outline Performance Statistic –Standard Bias and Error Calculations Model Performance Goals for PM –Speciated Bias and Error Goals –Relative Proportions Goals

3. Performance MetricsEquation Mean Bias (  g/m 3 ) Mean Error (  g/m 3 ) Mean Normalized Bias (%) (-100% to +  ) Mean Normalized Error (%) (0% to +  ) Normalized Mean Bias (%) (-100% to +  ) Normalized Mean Error (%) (0% to +  ) Mean Fractional Bias (%) (-200% to +200%) Mean Fractional Error (%) (0% to +200%)

4. Example GT showed a positive bias of 11 points NB = 14.3% FB = 13.3% North Carolina 77 Georgia Tech 88

5. Performance Metrics Mean Normalized Bias and Error –Usually associated with observation-based minimum threshold Some components of PM can be very small making it difficult to set a reasonable minimum threshold value without excluding a majority of the data points –Without a minimum threshold, very large normalized biases and errors can result when observations are close to zero even though the absolute biases and errors are very small A few data points can dominate the metric –Overestimations are weighted more than equivalent underestimations

6. Performance Metrics Normalized Mean Bias and Error –Biased towards overestimations Mean Fractional Bias and Error –Bounds maximum bias and error –Gives additional weight to underestimations and less weight to overestimations

7. Example Calculations Mean Normalized Bias and Error –Most biased and least useful of the three metrics Normalized Mean Bias and Error Mean Fractional Bias and Error –Least biased and most useful of the three metrics Model  g/m 3 ) Obs.  g/m 3 ) MB  g/m 3 ) NMB (%) MNB (%) MFB (%) ME  g/m 3 ) NME (%) MNE (%) MFE (%) 0.051.0-0.95-95-180.95+0.95+95+180.95 1.00.05+0.95+1900+180.95+0.95+1900+180.95 1.00.01+0.99+9900+196.04+0.99+9900+196.04 0.6830.353+0.33+93.4+3901.7+65.30.96272.93965.0186.0

8. SAMI Model Performance Summary Species# Obs Mean  g/m 3 MB  g/m 3 NMB (%) MNB (%) MFB (%) ME  g/m 3 NME (%) MNE (%) MFE (%) SO4 1346.71-1.16-17.31.1-22.72.4837.055.150.2 NO3 1340.63-0.30-47.66.8-73.60.5281.8112.8107.2 NH4 1342.70-1.25-46.4-27.4-57.41.4353.161.670.0 NH4 Bi 1341.440.010.434.2-2.60.6242.970.444.4 ORG 1323.41-0.27-7.815.8-6.01.3740.453.843.9 EC 1320.56-0.05-8.615.1-12.70.2748.361.950.4 Soils 1350.550.2546.2171.621.90.57102.9207.472.5 PM2.5 13017.05-4.79-28.1-9.1-28.86.839.848.947.6 PM10 13023.44-5.21-22.2-6.2-21.09.1839.144.243.5 PMC 1266.98-0.48-6.943.97.83.8655.278.754.1 bext 132133.1-27.91-21.0-10.2-23.743.7032.840.040.4

9. Proposed Performance Goals Based on Mean Fractional Error (MFE) and Mean Fractional Bias (MFB) calculations Performance goals should vary as a function of species concentrations –More abundant species should have a MFE  + 50% and MFB  ± 30% –Less abundant species should have less stringent performance goals Goals should be continuous functions with the features of: –Asymptotically approaching + 50% MFE and ± 30% MFB when the concentrations (mean of the observed and modeled concentrations) are greater than 2.5  g/m 3 –Approaching + 200% MFE and ± 200% MFB when the concentrations (mean of the observed and modeled concentrations) are extremely small

10. Proposed Mean Fractional Error and Bias Goals

11. Example Calculations Species X Model  g/m 3 )Obs.  g/m 3 ) FB (%)FE (%) Day 1 – Site A2.01.0+66.7 Day 1 – Site B1.02.0-66.7+66.7 Day 2 – Site A1.00.4+85.7 Day 2 – Site B0.51.5-100.0%+100.0% Average1.1251.225-3.6%79.8% Average C O + C M = 0.5*(1.125 + 1.225) = 1.175 MFE performance goal for “Species X” = 81.3% MFB performance goal for “Species X” = ± 46.2%

12. Mean Fractional Error Goal

13. Mean Fractional Bias Goal

14. SAMI – 6 Episodes

16. VISTAS – July 1999 Episode

18. VISTAS – January 2002 Episode

20. Relative Proportions (RP) PERF Goals EPA draft guidance (2001) –“For major components (i.e., those observed to comprise at least 30% of measured PM 2.5 ), we propose that the relative proportion predicted for each component averaged over modeled days with monitored data agrees within about 20% of the averaged observed proportion. For minor observed components of PM, we suggest a goal that the observed and modeled absolute proportion of each minor component agree within 5%.”                      N i o N i m m Total component Total component CoCo C NC C N Bias 11 11 0.2 RP% (5%)

21. Example Calculation Calculating component proportions based on concentrations averaged over multiple days can hide poor model performance Observed RP (%)Modeled RF (%) Day 150%95% Day 250%95% Day 350%5% Day 450%5% Average50%

22. ObservedSimulated Relative Proportions for SAMI

24. Proposed Relative Proportions Performance Goals Propose to use an equation that accounts for the day- to-day variability of species relative proportions: RP  30%, Error  10% RP  15%, Error  5% RP 15% - 30%, Error  [RP]/3

25. Proposed Relative Proportions Performance Goals

26. Concluding Remarks Recommended performance values are model goals, not model criteria –Failure to meet proposed performance goals should not prohibit the modeling from being used for regulatory purposes Help identify areas that can be improved upon in future modeling If performing episodic modeling, performance evaluation should be done on an episode-by- episode basis If performing annual modeling, performance evaluation should be done on a month-by- month basis

27. Concluding Remarks (cont.) As models mature, performance goals can be made more restrictive by simply: –Adjusting the coefficients in the MFE and MFB goal equations – Lowering the relative proportion error goals Q: Is there a need for performance goals for gaseous precursors or wet deposition species? –“One-atmosphere” modeling system –If not, still should be evaluated to help identify potential problems with PM model performance

28. Questions?