1. Preliminary Experiences with the Multi-Model Air Quality Forecasting System for New York State Prakash Doraiswamy 1, Christian Hogrefe 1,2, Winston Hao 2, Brian Colle 3, Mark Beauharnois 1, Ken Demerjian 1, J.-Y. Ku 2 and Gopal Sistla 2 1 Atmospheric Sciences Research Center, University at Albany, Albany, NY 2 New York State Department of Environmental Conservation, Albany, NY 3 School of Marine and Atmospheric Sciences, Stony Brook University, Stony Brook, NY 10/19/2009

2. Background The New York State Department of Environmental Conservation (NYSDEC) has been performing CMAQ model-based air quality forecasts daily since June 2005, based on the NCEP UTC 12z weather forecasts Beginning June 2008, NYSDEC, in collaboration with the University at Albany (SUNY-Albany) and Stony Brook University (SUNY-SB), has implemented an ensemble air quality forecasting system in an attempt to better quantify uncertainties associated with the ozone and PM 2.5 forecasts. SUNY-SB has established a short-range ensemble weather forecast system (SREF) consisting of 14 members that has been run over the Northeast US for nearly four years (http://chaos.msrc.sunysb.edu/NEUS/nwp_graphics.html) Funded by New York State Energy and Research Development Authority (NYSERDA) and NYSDEC through in-kind contributions

3. 12-member retrospective simulation Timeline of Ensemble Forecasting System May 2008 Nov 2008 Since June 2005 NCEP 12z June 2008 SUNYSB-F2 SUNYSB-F9 NCEP 00z NCEP 12z NCEP 00z SUNYSB-F2 SUNYSB-F9 NCEP 12z NCEP 00z NCEP 00z w/ DEC Emiss Mar 2009 SUNYSB-F2 SUNYSB-F9 NCEP 12z NCEP 00z NCEP 00z w/ DEC Emiss ASRC Aug 2008 12-member retrospective simulation Dec 2008

4. Daily Air Quality Forecast Ensemble Members Member Name Met.Emis. Inv.AQMGrid Res Initial- ize Start Date NCEP_t12zWRF- NMM EPACMAQ v4.6 12-km12z Summer 2004; Winter 2004- 2005; everyday since June 2005 NCEP_t00zWRF- NMM EPACMAQ v4.6 12-km00z May 2008 NYSDEC_3x _t00z WRF- NMM NYSDECCMAQ v4.6 12-km00z November 2008 SUNYSB_F2MM5NYSDECCMAQ v4.6 36-km, 12-km 00z June 2008 SUNYSB_F9WRF- ARW NYSDECCMAQ v4.6 36-km, 12-km 00z June 2008 ASRCWRF- ARW NYSDECCAMx v4.5.1 12-km00z March 2009

5. SUNYSB SREF Members Used in Retrospective CMAQ Simulations NameModelCloudPBLRadiationMicrophysicsInitialization F1MM5BMMYCCM2Simple IceGFS F2MM5GrellMRFCloudRadSimple IceWRF-NMM F3MM5GrellMYCloudRadReisner2WRF-NMM F5MM5GrellBlackadarCCM2Simple IceNOGAPS F6MM5KF2MYCCM2Simple IceCMC F7MM5KF2MRFCloudRadReisner2GFS F8WRFKF2MYRRTMWSM3CMC F9WRFBMMYRRTMWSM3WRF-NMM F10WRFKF2MYRRTMWSM3GFS F13MM5GrellBlackadarCCM2Simple IceGFS F14WRFBMYSURRTMWSM3NOGAPS F15WRFKFEMYRRTMThompsonGFS F2 and F9 were used to drive CMAQ forecasts each day since June 1, 2008. They were selected based on temperature and wind verification results for summer 2007 and operational considerations Two additional SREF members use the Ferrier microphysics scheme that is currently not compatible with CMAQ

6. Analysis Examined the performance of the daily simulated ensemble system for the following time periods: – June - September 2008: 4 members – December 2008 – February 2009: 5 members – June – August 2009: 6 members Compared daily 8-hr maximum Ozone (O 3 ) and 24-hr average PM 2.5 model predictions against observations from the AIRNOW database and against the NYSDEC official (human) forecasts For the summer 2009 period, comparisons were also made against operational NOAA ozone forecasts that were made available to NYSDEC from June 2009 – 06z initialization providing same-day forecast: NOAA_t06z – 12z initialization providing next-day forecast: NOAA_t12z Retrospective simulations of CMAQ using 12 SUNYSB short-range ensemble forecasting system (SREF) along with the regular members for the summer and winter time periods – June 4, 2008 – July 22, 2008 – December 1, 2008 – February 28, 2009

7. Official DEC Forecasts & Air Quality Forecast Regions in NYS Official DEC forecasters use human judgment and a variety of products including this ensemble system while issuing their forecasts Model-based forecast guidance is issued and evaluated following the same region-based approach used for the official human-based air quality forecasts issued by NYSDEC Each forecast region contains one or more ozone monitor and one or more continuous PM 2.5 monitor For a given region and day, the forecasted/observed air quality value for ozone (PM 2.5 ) is defined as the maximum ozone (PM 2.5 ) value among the ozone (PM 2.5 ) monitor(s) in that region Model values are extracted for the locations of all monitors, and the model air quality value for a region for ozone (PM 2.5 ) is defined in the same way as for the observations

8. The Air Quality Index (AQI) Used by NYSDEC Non-dimensional index to communicate air quality forecasts to the public Concentrations of ozone and PM 2.5 are converted to AQI through piecewise linear functions AQILevel of Health Concern Daily Maximum 8-hr O 3 Concentration (ppb) 24-hr Average PM 2.5 Concentration (μg/m 3 ) 0-50Good0-590-15.4 51-100 Moderate 60-7515.5-35.4 101-150 Unhealthy for sensitive groups 76-9535.5-55.4 151-200Unhealthy96-11555.5-140.4 201-300Very Unhealthy116-374140.5-210.4

9. OZONE PERFORMANCE MAY (JUNE) - SEP 2008 Daily Forecast Simulations Members: 1.NCEP 12z 2.NCEP 00z 3.(NYSDEC_3x not in operation) 4.SUNY-SB F2 5.SUNY-SB F9 6.(ASRC not in operation)

10. Model predictions track the observations Over-prediction around Aug-Sep particularly in regions 5-8 Time Series of 8-hr Daily Max O 3 May – Sep 2008 Observations NCEP Members SUNY F2 (MM5) SUNY F9 (WRF) ASRC Ensemble Average Ensemble Median Official DEC Forecasts

11. Bias ~ 2 to 7 ppb NCEP-based models: lower bias in upstate regions Ensemble average not always the lowest bias All models have a RMSE of 9 to 12 ppb, with ensemble average showing similar or lower RMSE NCEP Members SUNY F2 (MM5) SUNY F9 (WRF) ASRC Ensemble Average Ensemble Median Official DEC Forecasts Mean Bias of 8-hr Daily Max O 3 Jun – Sep 2008 Official DEC forecasts showed similar or lower bias

12. Categorical Metrics Prob. Of Detection (POD): Fraction of observed exceedances that were predicted correctly False Alarm Ratio (FAR): Fraction of incorrect predicted exceedances Critical Success Index (CSI) = correct exceedance forecasts / (correct exceedance forecasts + false alarms + missed exceedance forecasts); range 0 (no skill) to 1

13. NCEP Members SUNY F2 (MM5) SUNY F9 (WRF) ASRC Ensemble Average Ensemble Median Official DEC Forecasts Prob. Of Detection (POD), False Alarm Ratio (FAR) & Critical Success Index (CSI): O 3, Jun – Sep 2008

14. PM 2.5 PERFORMANCE WINTER: DEC 2008 – FEB 2009 Daily Forecast Simulations Members: 1.NCEP 12z 2.NCEP 00z 3.NYSDEC_3x 4.SUNY-SB F2 5.SUNY-SB F9 6.(ASRC not in operation)

15. Time Series of 24-hr Average PM 2.5 Dec 2008 – Feb 2009 Model predictions track the observations Except for Region 2, no significant over- predictions were found at other regions Observations NCEP Members SUNY F2 (MM5) SUNY F9 (WRF) ASRC Ensemble Average Ensemble Median Official DEC Forecasts

16. Mean Bias of 24-hr Average PM 2.5 Dec 2008 – Feb 2009 Over-prediction in Region 2 (NY City) and under-prediction at other regions; Ensemble average similar or lower bias All models have a RMSE of 3 to 13 µg/m 3, with ensemble average showing similar or lower RMSE. SUNY members showed higher RMSE at upstate regions. NCEP Members SUNY F2 (MM5) SUNY F9 (WRF) ASRC Ensemble Average Ensemble Median Official DEC Forecasts

17. Exceedances in region 2 were picked up by all models, but there were false alarms as well, resulting in <15% critical success index Official DEC forecasts did not capture any of the observed exceedances Prob. Of Detection (POD), False Alarm Ratio (FAR) & Critical Success Index (CSI): PM 2.5 Dec 2008 – Feb 2009

18. RETROSPECTIVE SIMULATIONS – SUMMER: JUN – JUL 2008 WINTER: DEC 2008 – FEB 2009 Members: 1.NCEP 12z, 00z and NYSDEC_3x: Shades of green 2.SUNY-SB SREF Members: a.MM5-based: Shades of blue b.WRF-based: Shades of orange

19. Overall performance is similar to the 4-member system MM5-based members (blue) typically showed a negative bias, while WRF-based members showed a positive bias. (Not noticed in PM 2.5 predictions) Ensemble average is most often better than any of the individual models. Mean absolute error is 5-6 ppb compared to 7-11 ppb for the individual models Mean Bias of 8-hr Daily Max O 3 June – July 2008 (SUMMER) NCEP Members SUNY-SB MM5-based SUNY-SB WRF-based Ensemble Average Ensemble Median Official DEC Forecasts

20. Time Series of Ensemble Mean and Standard Deviation Ozone: JUNE - JULY 2008 PM 2.5 DEC 2008 -FEB 2009 Standard deviation (black) among the members often, but not always, appears to increase with increase in concentration, suggesting that a higher absolute uncertainty may be associated with episodes

21. PERFORMANCE DURING SUMMER 2009 (JUN- AUG 2009) Daily Forecast Simulations Members: 1.NCEP 12z 2.NCEP 00z 3.NYSDEC_3x 4.SUNY-SB F2 5.SUNY-SB F9 6.ASRC 7.NOAA Operational Ozone Forecasts

22. Ozone PM 2.5 Typical bias of 4-10 ppb; ASRC WRF/CAMx system was an outlier with a bias of 12–17 ppb Contrary to previous years, a positive bias was seen at all regions; The ASRC CAMx system was not an outlier for PM 2.5 Mean Bias: Jun – Aug 2009 Observations NCEP Members SUNY F2 (MM5) SUNY F9 (WRF) ASRC NOAA Operational Forecasts Ensemble Average Ensemble Median Official DEC Forecasts

23. False Alarm Ratio (FAR): O 3, Jun – Aug 2009 Observations NCEP Members SUNY F2 (MM5) SUNY F9 (WRF) ASRC NOAA Operational Forecasts Ensemble Average Ensemble Median Official DEC Forecasts FAR of ~50 -80%, for Ozone compared to 20- 60% during 2008

24. Notes on Summer 2009 Performance All models over-predicted ozone concentration during summer 2009, including the NOAA model FAR was higher than what was observed the previous year Contrary to previous summers for PM 2.5, model predictions were positively biased for all regions What is different this summer? – Meteorology ? – Emissions ?

25. Below Normal TemperatureAbove Normal Precipitation Courtesy: NCDC/NOAA plots compiled by Tom Downs of Maine Department of Environmental Protection Meteorology: Cooler and Wetter Summer

26. Emissions Cooler and wetter summer may have been less favorable to ozone formation in general The weather patterns alone may not fully explain the ozone over-prediction by the models. Even days with observed temperatures greater than 90+ °F did not always result in an observed ozone exceedance. So could the model over-predictions be related to differences in emissions between the model and the real-world? – Power plant (i.e., electric generating units, EGUs) emissions in the model are based on 2005 measured emissions with no adjustment. Based on the data from the continuous emission monitors, these emissions have decreased by an average of ~15% during the ozone season (May-Sep), and by ~20% on an annual emission basis between 2005 and 2008 in the northeast US – Any decrease in emissions due to the current economic recession?

27. Emissions Sensitivity Simulation To test this, we selected the NCEP member that uses the NYSDEC emission inventory, referred to as “NYSDEC_3x” Reduced all anthropogenic emissions of all pollutants from all source categories by 20% over the whole domain Reran the CMAQ model with the reduced emissions from August 7 to August 26, 2009, during which high ozone episodes were observed (08/10, 08/16, 08/17) in Regions 1 & 2 (Long Island and New York City). The rerun is referred to as “DEC_3x_20pctcut” in the following plots

28. Time Series of 8-hr Daily Max Ozone A 20% cut in anthropogenic emissions (blue) resulted in a maximum of ~7% reduction in the predicted 8-hr daily max ozone concentrations compared to the base case (green) simulation (4.7 ppb in region 5 to 7.3 ppb in region 1).

29. NYSDEC_3x Original Simulation NYSDEC_3x w/ 20% cut in anthropogenic emissions Normalized Mean Bias (NMB) Over the Whole Domain A 20% cut in emissions shifted the NMB by one color category (for example, 20->25% to 10-20%) at most locations in the Eastern US. May indicate that the significant over-prediction in ozone concentrations this summer could be partly related to an over-estimated emission inventory.

30. Summary The 4- to 6- member multi-model system predictions tracked ozone and PM 2.5 observations during summer and winter It appeared to capture the range of observed ozone concentrations during summer 2008, but under-predicted PM 2.5 concentrations for all regions except the NY City area Winter PM 2.5 concentrations were also under-predicted in most regions, except NY City area. Future work will compare PM 2.5 species concentrations with CSN speciation data. Retrospective simulations of a 14- or 15-member system showed similar results as the regular mini-ensemble system. Overall for the NY State region, the ensemble average (and median) often, but not always, showed similar or better performance than the individual models.

31. Summary… Daily variation between the members, as represented by the standard deviation of the ensemble mean, appeared to be mostly (but not always) larger on days with higher observed concentrations. This may suggest that episodic days may sometimes be associated with higher absolute uncertainty. On a relative basis, the daily variability in model-predictions based on the multi-model system was ~5 to 15% for 8-hr maximum Ozone in summer and 20-30% or greater for 24-hr average PM 2.5 concentrations in winter. Analysis of the summer 2009 season showed over-predictions for both ozone and PM 2.5. In addition to the cooler and wetter weather patterns that may have contributed partially to model over-predictions, an emissions sensitivity analysis suggests possible over-estimated emissions inventory.

32. Summary… This indicates the challenges associated with incorporating up-to-date emissions that are reflective of real-world activity in forecasting applications.

33. Supplementary Slides Differences in Emission Inventory between NYSDEC and EPA Inventory for PM 2.5

34. Year of inventory database Source Category NYSDECEPA On-road Mobile 2005 VMT: interpolation between 2002 and 2009 2007 VMT projected from 2001 data Point: EGU2005: from 2005 NIF2001, based on 2001 NEI Point: Non- EGU 2005: interpolation between 2002 and 2009 2001, based on 2001 NEI Canada: 1996 inventory, using 1995 data Area2005: interpolation between 2002 and 2009 Fugitive dust: 2001, based on 2001 NEI (no transport factor applied) Fire: 2010 inventory, projected from 2001 inventory Other area: 2001, based on 2001 NEI Non-Road2005: interpolation between 2002 and 2009 2001, based on 2001 NEI

35. Note: The pie charts do not include PM2.5 from on-road mobile sources VMTNYSDECEPA OnRd Gas 143,338 E06 miles/yr 132,523 E06 miles/yr OnRd Diesel 6,941 E06 miles/yr 11,311 E06 miles/yr New York State

36. NYSDEC PM2.5EPA PM2.5 RankCategory Emissions (tons/yr)Category Emissions (tons/yr) 1 Stationary Source Fuel Combustion Residential 38,475 Stationary Source Fuel Combustion Residential 40,631 2Industrial Processes Other 9,169Unpaved Roads 20,008 3 Stationary Source Fuel Combustion Industrial 5,311 External Combustion Boilers Electric Generation 14,667 4Unpaved Roads 4,479 Industrial Processes Construction: SIC 15 - 17 14,497 5NonRd Diesel 4,249Open Burning Waste Disposal 13,216 6Paved Roads 3,508Paved Roads 12,707 7 Stationary Source Fuel Combustion Commercial/Institutional 3,154Agriculture Production - Crops 7,850 8 Industrial Processes Construction: SIC 15 - 17 3,045Industrial Processes Other 6,674 9 External Combustion Boilers Electric Generation 2,185 Stationary Source Fuel Combustion Commercial/Institutional 2,992 10NonRd Gasoline 1,905 Stationary Source Fuel Combustion Industrial 1,994 Top 10 PM 2.5 Emission Source Categories in NYS

37. Note: The pie charts do not include PM2.5 from on-road mobile sources VMTNYSDECEPA OnRd Gas 34,689 E06 miles/yr 46,081 E06 miles/yr OnRd Diesel 1,415 E06 miles/yr 3,024 E06 miles/yr New York City (Region 2 of AQF)

38. Top 10 PM 2.5 Emission Source Categories in NYC NYSDEC PM2.5EPA PM2.5 RankCategory Emissions (tons/yr)Category Emissions (tons/yr) 1Industrial Processes Other 2,609 Stationary Source Fuel Combustion Residential 5,235 2NonRd Diesel 1,747 Industrial Processes Construction: SIC 15 - 17 2,513 3 Stationary Source Fuel Combustion Residential 1,601 External Combustion Boilers Electric Generation 2,109 4 Stationary Source Fuel Combustion Commercial/Institutional 1,477Paved Roads 1,844 5 Stationary Source Fuel Combustion Industrial 1,402 Stationary Source Fuel Combustion Commercial/Institutional 1,550 6 External Combustion Boilers Electric Generation 768Marine Vessels, Commercial 1,525 7 Stationary Source Fuel Combustion Electric Utility 760 Stationary Source Fuel Combustion Industrial 716 8 Industrial Processes Construction: SIC 15 - 17 481Other Waste Disposal 574 9NonRd Gasoline 466Industrial Processes Other 518 10Other Waste Disposal 456Internal Combustion Engines 410

39. Remarks In general, over NY State (NYS) EPA emissions were higher than NYSDEC for all source categories, except the non-road mobile sources – Fugitive dust emissions from paved and unpaved roads were 3.5 to 4.5 times higher in the EPA inventory than NYSDEC inventory – Higher contribution of emissions from open burning in the EPA inventory For the NY City (NYC) region (Region 2 of AQF), it appears that the EPA inventory has higher contribution from most source categories than NYSDEC inventory for the NY City region – 2 times higher PM 2.5 emissions from stationary source fuel combustion (all 3 subcategories together) than NYSDEC – 13 times higher emissions from paved road dust than NYSDEC – Emissions from marine vessels also appear to be high? For comparison, the NOx emissions are 37,000 tons/yr in EPA inventory versus 7,000 tons/yr in NYSDEC inventory. EPA has higher VMT (2007 yr) (a difference of 13,000 E06 miles) within NYC region than NYSDEC (2005 year) inventory. Even if we assume that it is because of growth in VMT between 2005 and 2007, it appears abnormally high (18% increase per year from the 2005 NYSDEC value)