Calculating Statistics: Concentration Related Performance Goals James W. Boylan Georgia Department of Natural Resources PM Model Performance Workshop Chapel Hill, NC February 11, 2004

Outline Performance Statistic –Standard Bias and Error Calculations Model Performance Goals for PM –Speciated Bias and Error Goals –Relative Proportions Goals

Performance MetricsEquation Mean Bias (  g/m 3 ) Mean Error (  g/m 3 ) Mean Normalized Bias (%) (-100% to +  ) Mean Normalized Error (%) (0% to +  ) Normalized Mean Bias (%) (-100% to +  ) Normalized Mean Error (%) (0% to +  ) Mean Fractional Bias (%) (-200% to +200%) Mean Fractional Error (%) (0% to +200%)

Example GT showed a positive bias of 11 points NB = 14.3% FB = 13.3% North Carolina 77 Georgia Tech 88

Performance Metrics Mean Normalized Bias and Error –Usually associated with observation-based minimum threshold Some components of PM can be very small making it difficult to set a reasonable minimum threshold value without excluding a majority of the data points –Without a minimum threshold, very large normalized biases and errors can result when observations are close to zero even though the absolute biases and errors are very small A few data points can dominate the metric –Overestimations are weighted more than equivalent underestimations

Performance Metrics Normalized Mean Bias and Error –Biased towards overestimations Mean Fractional Bias and Error –Bounds maximum bias and error –Gives additional weight to underestimations and less weight to overestimations

Example Calculations Mean Normalized Bias and Error –Most biased and least useful of the three metrics Normalized Mean Bias and Error Mean Fractional Bias and Error –Least biased and most useful of the three metrics Model  g/m 3 ) Obs.  g/m 3 ) MB  g/m 3 ) NMB (%) MNB (%) MFB (%) ME  g/m 3 ) NME (%) MNE (%) MFE (%)

SAMI Model Performance Summary Species# Obs Mean  g/m 3 MB  g/m 3 NMB (%) MNB (%) MFB (%) ME  g/m 3 NME (%) MNE (%) MFE (%) SO NO NH NH4 Bi ORG EC Soils PM PM PMC bext

Proposed Performance Goals Based on Mean Fractional Error (MFE) and Mean Fractional Bias (MFB) calculations Performance goals should vary as a function of species concentrations –More abundant species should have a MFE  + 50% and MFB  ± 30% –Less abundant species should have less stringent performance goals Goals should be continuous functions with the features of: –Asymptotically approaching + 50% MFE and ± 30% MFB when the concentrations (mean of the observed and modeled concentrations) are greater than 2.5  g/m 3 –Approaching + 200% MFE and ± 200% MFB when the concentrations (mean of the observed and modeled concentrations) are extremely small

Proposed Mean Fractional Error and Bias Goals

Example Calculations Species X Model  g/m 3 )Obs.  g/m 3 ) FB (%)FE (%) Day 1 – Site A Day 1 – Site B Day 2 – Site A Day 2 – Site B %+100.0% Average %79.8% Average C O + C M = 0.5*( ) = MFE performance goal for “Species X” = 81.3% MFB performance goal for “Species X” = ± 46.2%

Mean Fractional Error Goal

Mean Fractional Bias Goal

SAMI – 6 Episodes

VISTAS – July 1999 Episode

VISTAS – January 2002 Episode

Relative Proportions (RP) PERF Goals EPA draft guidance (2001) –“For major components (i.e., those observed to comprise at least 30% of measured PM 2.5 ), we propose that the relative proportion predicted for each component averaged over modeled days with monitored data agrees within about 20% of the averaged observed proportion. For minor observed components of PM, we suggest a goal that the observed and modeled absolute proportion of each minor component agree within 5%.”                      N i o N i m m Total component Total component CoCo C NC C N Bias RP% (5%)

Example Calculation Calculating component proportions based on concentrations averaged over multiple days can hide poor model performance Observed RP (%)Modeled RF (%) Day 150%95% Day 250%95% Day 350%5% Day 450%5% Average50%

ObservedSimulated Relative Proportions for SAMI

Proposed Relative Proportions Performance Goals Propose to use an equation that accounts for the day- to-day variability of species relative proportions: RP  30%, Error  10% RP  15%, Error  5% RP 15% - 30%, Error  [RP]/3

Proposed Relative Proportions Performance Goals

Concluding Remarks Recommended performance values are model goals, not model criteria –Failure to meet proposed performance goals should not prohibit the modeling from being used for regulatory purposes Help identify areas that can be improved upon in future modeling If performing episodic modeling, performance evaluation should be done on an episode-by- episode basis If performing annual modeling, performance evaluation should be done on a month-by- month basis

Concluding Remarks (cont.) As models mature, performance goals can be made more restrictive by simply: –Adjusting the coefficients in the MFE and MFB goal equations – Lowering the relative proportion error goals Q: Is there a need for performance goals for gaseous precursors or wet deposition species? –“One-atmosphere” modeling system –If not, still should be evaluated to help identify potential problems with PM model performance

Questions?