CMAS Conference, October 6 – 8, 2008 The work presented in this paper was performed by the New York State Department of Environmental Conservation with partial support from the U.S. EPA under cooperative agreement CR and the New York State Energy Research and Development Authority (NYSERDA) under agreement # The views expressed in this paper do not necessarily reflect the views or policies of the New York State Department of Environmental Conservation or those of the sponsoring agencies. Multi-Model Air Quality Forecasting Over New York State For Summer 2008 Christian Hogrefe 1,2,*, Prakash Doraiswamy 2, Winston Hao 1, Brian Colle 3, Mark Beauharnois 2, Ken Demerjian 2, Jia-Yeong Ku 1, and Gopal Sistla 1 1 New York State Department of Environmental Conservation, Albany, NY 2 Atmospheric Sciences Research Center, State University of New York at Albany, Albany, NY 3 School of Marine and Atmospheric Sciences, Stony Brook University, Stony Brook, NY

Ensemble Forecasting Consists of using multiple models, multiple configurations of the same model, multiple initial/boundary conditions, multiple model inputs, or a combination thereof Aims at minimizing the effects of uncertainties in model inputs and model parameterizations on model predictions –In weather forecasting, ensemble-mean forecasts are often found to provide higher accuracy than forecasts from individual models –Initial applications for air quality forecast studies (McKeen et al., JGR, 2005; Pagowski et al., GRL, 2005) also show promise but were limited in temporal scope In addition, ensemble systems can be used to provide probabilistic forecasts of threshold exceedances

CMAQ-Based Air Quality Forecasting at NYSDEC Since 2004: CMAQ “next day” air quality forecast driven by 12z NCEP WRF-NMM (ETA-NAM prior to June 2006) (12 km) Since April 2008: additional CMAQ “same-day” air quality forecast driven by 00z NCEP WRF-NMM (12 km) Since June 2008: two additional CMAQ “same day” air quality forecast driven by two selected members of the SUNY-Stony Brook (SUNY-SB) 00z Short-Range Ensemble Forecast System (SREF) (nested 36km/12 km) –SUNY SB SREF: consists of a total of 14 MM5 or WRF weather forecasts different by their initial conditions and physics options –Two members were selected to drive daily CMAQ simulations based on temperature and wind verification results for summer 2007 and operational considerations –Two of the 14 SREF members use the Ferrier microphysics scheme that is currently not compatible with CMAQ June 1 – July 22, 2008: retrospective case study to perform twelve CMAQ “same day” air quality forecasts driven by all compatible members of the SUNY-SB 00z SREF (nested 36km/12 km) While there are differences in emission inventories, projection parameters, and horizontal and vertical grid setup, the horizontal grid spacing over New York State is 12 km in all simulations

Air Quality Forecast Regions in NYS Model-based forecast guidance is issued and evaluated following the same region-based approach used for the official human-based air quality forecasts issued by NYSDEC Each forecast region contains one or more ozone monitor and one or more continuous PM 2.5 monitor For a given region and day, the forecasted/observed air quality value for ozone (PM 2.5 ) is defined as the maximum ozone (PM 2.5 ) value at any ozone (PM 2.5 ) monitor in that region Model values are extracted for the locations of all monitors and the model air quality value for ozone (PM 2.5 ) is defined in the same way as for the observations

The Air Quality Index (AQI) Used by NYSDEC Non-dimensional index to communicate air quality forecasts to the public Concentrations of ozone and PM 2.5 are converted to AQI through piecewise linear functions Some PM 2.5 thresholds lower than those used in AIRNOW AQILevel of Health Concern Daily Maximum 8-hr O 3 Concentration (ppb) 24-hr Average PM 2.5 Concentration (μg/m 3 ) 0-50Good Moderate Unhealthy for sensitive groups Unhealthy Very Unhealthy

Discrete and Categorical Evaluation of Ozone and PM 2.5 Predictions From the Four Member CMAQ Forecast Guidance System Summer 2008 (June 4 – August 31)

Daily Maximum 8-hr Ozone June 4 –August 31 for NYS Forecast Regions 1-8 Red: observations Blue: 4-Model Average Grey: 4-Model Range (minimum to maximum)

Blue/green represent individual model runs, red is the ensemble mean, orange is the ensemble median The four member mean or median forecasts often but not always have lower RMSE than individual forecasts RMSE of predicted 8-hr daily maximum ozone concentrations June 4 – August 31: Individual Models, Ensemble Average, and Ensemble Median

Left to Right: Observations, 4-Model Average, NCEP/CMAQ 12z, NCEP/CMAQ 00z, SUNY-SB F2 / CMAQ 00z, and SUNY-SB F9 / CMAQ 00z Relative frequency of AQI categories for 8-hr daily maximum ozone June 4 – August 31: observations, Ensemble Average, and Individual Models

CSI = correct exceedance forecasts / (correct exceedance forecasts + false alarms + missed exceedance forecasts); range 0 (no skill) to 1 Blue/green represent individual model runs, red is the ensemble mean, orange is the ensemble median Critical Success Index (CSI) for categorical forecasts of an ozone threshold of 75 ppb for predicted 8-hr daily maximum ozone concentrations June 4 – August 31

Daily Average PM 2.5 June 4 –August 31 for NYS Forecast Regions 1-8 Red: observations Blue: 4-Model Average Grey: 4-Model Range (minimum to maximum)

Bias of predicted 24-hr average PM 2.5 concentrations June 4 – August 31: Individual Models, Ensemble Average, and Ensemble Median Blue/green represent individual model runs, red is the ensemble mean, orange is the ensemble median For all regions except NYC (region 2), PM2.5 predictions tend to be biased low

Discrete, Categorical, and Probabilistic Evaluation of Ozone and PM 2.5 Predictions From the Retrospective Case Study With a Twelve Member CMAQ Forecast Guidance System Case Study June 4 – July 21 (period of highest O 3 during summer 2008 in New York State)

How much variability in meteorological and air quality variables is introduced by using the 12 meteorological SREF members to drive CMAQ?  Calculate coefficient of variation (standard deviation across models / mean across models); averaged June 4 – July 21 Daily Average WindDaily Maximum PBLDaily Maximum Temperature Daily Maximum 8-hr O 3 Daily Average PM 2.5 Variations in meteorology introduced by the twelve SREF members cause a typical ozone variability of 5 – 10% (with higher values in urban areas near land/sea interfaces) and a typical PM 2.5 variability of 20 – 25%

Daily Maximum 8-hr Ozone June 4 – August 31 for NYS Forecast Regions 1 (Long Island) and 2 (NYC Metro) Red: observations Blue: 12-Model Average Grey: 12-Model Range (minimum to maximum) (Increased compared to 4-member results)

The twelve member mean or median forecasts often but not always have better performance statistics than individual forecasts RMSE, correlation coefficient, and CSI of predicted 8-hr daily maximum ozone June 4 – July 21: Individual Models, Ensemble Average, and Ensemble Median

Example of Using the Ensemble to Predict Exceedance Probabilities: Predicted Probability of Daily Maximum 8-hr O 3 > 75 ppb for June 10, out of 12 ensemble members predicting O 3 > 75 ppb  probability = 0% 1 out of 12 ensemble members predicting O 3 > 75 ppb  probability = 8.33% 12 out of 12 ensemble members predicting O 3 > 75 ppb  probability = 100% Observed AQI

Given that the predicted exceedance probability was xx%, what was the observed exceedance probability? (i.e. if there were 10 days where 3-4 models predicted an exceedance, on how many of these days was there an observed exceedance?) For a good probabilistic forecast system, the points fall close to the 1:1 line Example Probabilistic Evaluation of Ensemble Predictions of 8-hr Daily Maximum O 3 Exceedances For Regions 1 and 2 Caveat: sample size in this example is very small (48 days, ~10 observed exceedances) for this type of analysis

Each day, rank-order the 12 forecasts Depending on whether the observations are lower than the lowest forecast, fall between the lowest and second-lowest forecast, … are higher than the highest forecast, assign that day to one of 13 bins Repeat the analysis for all days and create a histogram based on the number of days assigned to each of the 13 bins Ideal shape – flat U-shape (inverted U-shape): ensemble is underdispersed (overdispersed) L-shape – ensemble is biased Talagrand Diagram (Rank Histogram) for 8-hr Daily Maximum O 3

The ensemble predictions are underdispersed and often biased Talagrand Diagram (Rank Histogram) for 24-hr Average PM 2.5

Summary and Next Steps Over NYS, the summer of 2008 was characterized by several ozone episodes from early June through late July and relatively low ozone thereafter As measured by discrete and categorical metrics, the four model system utilized since June 1, 2008 appears to have provided good ozone forecast guidance for the summer 2008, especially for Long Island and the NYC metro area As reported in earlier studies, forecasts for summertime PM 2.5 are characterized by a negative bias for all areas except the NYC metro area  the upcoming CMAQ release with updated SOA module may help (or not …) The four member mean or median forecasts often but not always have better statistics than individual forecasts  weighting or bias correction approaches prior to averaging may provide improved forecasts

Summary and Next Steps For a 48 day retrospective case study, CMAQ simulations were performed using twelve SUNY-SB SREF weather forecasts As for the four member system, the twelve member mean or median forecasts also often but not always have better statistics than individual forecasts Variations in meteorology introduced by the twelve SREF members cause a typical ozone variability of 5 – 10% (with higher values in urban areas near land/sea interfaces) and a typical PM 2.5 variability of 20 – 25% The 12-member system provides a realistic spread of ozone concentrations for regions 1 and 2 but tends to be underdispersed for other regions and for PM 2.5, indicating that uncertainties from other factors such as emissions or chemistry need to be included for a better representation of uncertainty Next steps: –Increase the number of daily forecast members (add CAMx) –Perform retrospective 12-member case studies for additional seasons –Investigate and implement weighting and bias correction approaches