1. AN EVALUATION OF THE ETA-CMAQ AIR QUALITY FORECAST MODEL AS PART OF NOAA’S NATIONAL PROGRAM CMAQ AIRNOW AIRNOW Brian Eder* Daiwen Kang * Ken Schere* Ken Schere* Robert Gilliam* Jonathan Pleim* Atmospheric Modeling Division Atmospheric Modeling Division Air Resources Laboratory, NOAA August 26,2003 August 26,2003 * On assignment to NERL EPA * On assignment to NERL EPA RTP, NC 27711 RTP, NC 27711

2. Forecast Configuration -Eta Meteorology -CBIV Mechanism -SMOKE Emissions (Offline) -12 km grid resolution -22 Vertical Layers 48 Hr. Forecast (12Z Initialization) 7 July – 31 September, 2003 7 July – 31 August (shown) 7 July – 31 August (shown) 48 Hr.Forecast (Corrected Land-use) 12 - 19 August 48 Hr. Forecast (Corrected Land-use) 12 - 19 August Domain Models-3 CMAQ

3. This evaluation used: Hourly O 3 concentrations (ppb) from EPA’s AIRNOW network 521 stations 7 July - 31 August A suite of statistical metrics for both: discrete forecasts and categorical forecasts for the: hourly, maximum 1-hr, maximum 8-hr O 3 simulations

4. Two Forecast / Evaluation Types - Discrete Forecasts [Observed] versus [Forecast] - Category Forecasts (Two Category) Observed Exceedances, Non-Exceedances versus Forecast Exceedances, Non-Exceedances

5. Discrete Forecast / Evaluation Statistics Statistics - Summary - Regression - Biases - Errors AIRNOW [Observed] versus [Forecast]

6. Category Forecast / Evaluation - Two Category Forecasts Observed Exceedances, Non-Exceedances versus Forecast Exceedances, Non-Exceedances ab cd Forecast Exceedance No Yes Observed Exceedance a b c d

7. Category Forecast A ccuracy Percent of forecasts that correctly predict event or non-event. B ias Indicates if forecasts are under-predicted (false negatives) or over-predicted (false positives ) F alse A larm R ate Percent of times a forecast of high ozone did not occur a b c d

8. C ritical S uccess I ndex How well the high ozone events were predicted. P robability O f D etection Ability to predict high ozone events Category Forecast a b c d

9. a c a b c d a= 151 b= 1 c= 24,227 d= 4 n= 24,383 CMAQ = 34.9 + 0.65(AIRNOW) Max 1-hr O 3 7 July – 31 August

10. Summary Statistics Discrete Evaluation Categorical Evaluation [ppb]CMAQAIRNOWCMAQ = 34.9 + 0.65 (AIRNOW) Ozone  125 ppb Mean71.656.5r0.60A99.4% SD18.116.6n24,383B25.5 CV25.229.3 Max182.9132BIASES 95 th 103.484MB15.1FAR99.4% 75 th 83.067NMB26.9%CSI0.6% 50 th 70.556 25 th 58.045ERRORS 5 th 45.430RMSE21.9POD16.7% Min01NME31.7% Max 1- hr O 3

11. Temporal Evaluation – Max 1 hr O 3 7 July 1 August 31 August

12. Spatial Evaluation Max 1- hr O 3 Correlation 0.00 – 0.25 0.25 – 0.50 0.50 – 0.75 0.75 – 1.00 Mean = 0.60

13. Max 1- hr O 3 Mean Bias Spatial Evaluation -10 – 10 10 - 20 20 - 30 30 – 40 40 – 50 Mean = 15.1

14. Spatial Evaluation Max 1- hr O 3 Root Mean Square Error 0 – 10 10 - 20 20 - 30 30 – 40 40 – 50 Mean = 21.9

15. CMAQ = 35.5 + 0.64(AIRNOW) a b c d a= 3276 b= 149 c= 20,979 d= 65 n= 24,469 Max 8-hr O 3

16. Summary Statistics Discrete Evaluation Categorical Evaluation [ppb]CMAQAIRNOWCMAQ = 35.5 + 0.64 (AIRNOW) Ozone  85 ppb Mean67.249.6r0.57A86.3% SD16.514.7n24,469B16.0 CV24.5%29.7% Max162.2108.4BIASES 95 th 95.873.9MB17.6FAR95.6% 75 th 78.059.8NMB35.8%CSI4.2% 50 th 66.449.5 25 th 54.739.1ERRORS 5 th 43.326.1RMSE23.0POD69.6% Min01NME39.1% Max 8- hr O 3

17. Temporal Evaluation – Max 8 hr O 3 1 August 31 August 7 July

18. Spatial Evaluation Max 8- hr O 3 Correlation 0.00 – 0.25 0.25 – 0.50 0.50 – 0.75 0.75 – 1.00 Mean = 0.57

19. Spatial Evaluation Max 8- hr O 3 Mean Bias -10 – 10 10 - 20 20 - 30 30 – 40 40 – 50 Mean = 17.6

20. Spatial Evaluation Max-8 hr O 3 Root Mean Square Error 0 – 10 10 - 20 20 - 30 30 – 40 40 – 50 Mean = 23.0

21. Land-Use Error Land-use fields associated with Eta were being post-processed incorrectly. As a result : - Most of the domain was classified as water. - Dry deposition was greatly under simulated This error was discovered/corrected by NCEP on Sept. 9 th. This error was discovered/corrected by NCEP on Sept. 9 th. - An eight day period (12-19 August) was re-simulated. - Positive biases were cut in half, errors reduced also.

22. Runr MB (ppb) NMB (%) RMSE (ppb) NME (%) A (%) BFAR (%) CSI (%) POD (%) Initial0.6416.227.523.031.799.0-100.00.0- Corrected0.667.613.016.621.799.6-100.00.0- Max 1-hr O 3 Max 8-hr O 3 Comparison Between Initial and Corrected Simulations August 12 –19 2003 Runr MB (ppb) NMB (%) RMSE (ppb) NME (%) A (%) BFAR (%) CSI (%) POD (%) Initial0.6219.237.224.639.976.2-100.00.0- Corrected0.6410.420.117.126.390.73.592.06.628.0

23. Temporal Evaluation (Corrected August 12 –19) – Max 1 hr O 3 – Max 8 hr O 3

24. Summary The Eta-CMAQ modeling system performed reasonably well, in this, its first attempt at forecasting ozone concentrations: Correlation:0.57 - 0.60 Bias: 15.1 ppb (26.9%) - 17.6 ppb (31.7%) Error:21.9 ppb (31.7%) - 23.0 ppb (39.1%) Accuracy:86.3 - 99.4% An error was discovered in Eta’s post processed land-use designation that resulted in the: – under-estimation of dry deposition and – hence over-simulation of concentrations Once corrected, the positive biases and errors were greatly reduced: Correlation:0.64 - 0.66 Bias:7.6 ppb (13.0%) - 10.4 ppb (20.1%) Error:16.6 ppb (21.7%) - 17.1 ppb (26.3%) Accuracy:90.7 - 99.6%

25. Contact: Brian K. Eder email: eder@hpcc.epa.gov