of Temperature in the San Francisco Bay Area Statistical Evaluation of High-resolution Numerical Weather Model Forecasts of Temperature in the San Francisco Bay Area Yilin Lu & Dave Dempsey Department of Earth & Climate Sciences Introduction Why are weather predictions not always accurate? Inspired by this question, I pursued this research with the ultimate goal of improving a forecast model by first spotting its potential defects. In particular, I used the Weather Research & Forecasting (WRF) model. The Weather Research & Forecasting (WRF) model can make weather forecasts on user-specified bounded regions in space (domains) with high spatial and temporal resolution. In this project, we begin by looking to see whether forecast errors worsen the further ahead in time that the model makes forecasts. We used weather station observations to evaluate the accuracy of the model temperature forecasts in the San Francisco Bay Area by posing and testing the following statistical hypothesis: Strategy: Test H0 statistically at the 95% confidence level Statistical Evaluation How do we evaluate forecast accuracy? Mean Absolute Error (MAE) = Problem 1: Model grid points and observations are at different locations (Fig.1). Solution: Interpolate forecasts from grid points to observation points Problem 2: In a plot of MAE against forecast hour, two patterns appear: a 6 hour cyclic pattern and an increasing trend (Fig. 2). The non-random cyclic pattern interferes with our hypothesis test of statistical significance of the trend. (The linear regression residuals must be random.) Solution: Remove the cyclic pattern by applying a 6-hour “moving average” to mean absolute errors (Fig. 4). Problem 3: MAEs at successive hours, like both observations and forecasts, are autocorrelated (i.e., not entirely independent). This dependence will interference with our hypothesis test of statistical significance of the trend. (The linear regression residuals must be independent.) Solution: Calculate the autocorrelation function for MAE vs. forecast hour. Use it to identify a sampling interval to ensure sample independence. Result: Sample every 5 forecast hours. Problem 4: In Figs. 2 & 4, each MAE comprises an average from 4 forecasts/day for 111 days, which sacrifices independent information from individual forecasts. We want to use this information, but MAEs at successive days are also autocorrelated. Solution: (a) Average MAEs for each day (4 forecasts/day) to remove the diurnal cycle; (b) Calculate the autocorrelation function for daily average MAE vs. day. Use it to identify a sampling interval to ensure sample independence. Result: Sample one-day MAE average every 3 days. (Fig.5 shows the results of solutions to both Problems 3 & 4) We are now ready to test our hypothesis about the trend of MAE vs. forecast hour. Statistical t-test & Results Null Hypothesis: The trend (slope) of MAE vs. forecast hour ≤ 0 oC/hr Select confidence level: 95% Simple size of MAE = 297 (degrees of freedom = 295) Estimate the slope using least squares linear regression (result: 0.005 oC/hr) Calculate a sample t-statistic: (estimated slope – hypothesized slope) / standard error of the slope Table 1: Results of right-tailed t-test Figure 5: Mean absolute error in forecast temperature, sampled every 5 forecast hours and every 3 days, in San Francisco Bay Area from January 1 to April 22 Research Question: Do WRF model temperature forecasts become less and less accurate as forecast time increases? Null Hypothesis (H0): No, the model won’t make worse forecasts as forecast time increases. Figure 2: Mean absolute error (MAE) in temperature from 4 runs/day for 111 days, vs. forecast hour. Figure 3: MAE in temperature from 111 days for the10 pm forecasts only, vs. forecast hour. Why there is a 6 hour cyclic pattern? Because there is a 12 hour semi-diurnal pattern in forecast errors, and model is run every 6 hours. Methods Model Configuration WRF = Initial Weather Conditions + Changes over a period Forecast (Solving a set of equations) (at discrete (Lower-resolution grid points) National Weather Forecast boundary Service forecast model) conditions on the domain Our model forecast runs: Daily frequency: 4 times/day (4 am, 10 am, 4 pm, & 10 pm PST) Forecast length: 48 hours Output interval: 1 hour Test period: 111 days from January 1 to April 22 Weather Observations Source: Meteorological Assimilation Data Ingest System (MADIS) Numbers of stations: About 90 in San Francisco Bay Area domain Quality Control: Reject missing and unreliable data provides Confidence Level Critical t Sample t-Statistic Result of test 95% 1.65 2.461 Reject H0 required for provides < Discussion & Conclusion The result indicates that we can reject the null hypothesis and conclude with 95% confidence that the WRF model shows a statistically significant positive trend in forecast errors vs. forecast hour. Although the trend seems relatively small (0.24oC in 48 hours), the impacts of decreasing accuracy with respect to forecast hour will depend on the application. Additional observations leading to future research questions: Forecast errors appear at the initialization time. We don’t know how this might affect the trend. The semi-diurnal pattern in MAE was a surprising discovery. Understanding the cause could lead to improvement of model forecasts. Many more! Figure 1: Low-resolution model grid (right, ), and the high-resolution WRF model grid for our San Francisco Bay Area domain (left, ), where there is one model forecast temperature at each grid point. Color-filled contours show an example of a forecast temperature pattern. Blue stars (left, ) show locations of weather stations used to evaluate the model forecasts. Figure 4 (right): Temperature MAE smoothed with a 6-hour moving average (compare with Fig. 2). A least squares linear regression line is fit to the data. Acknowledgements Atmospheric Sciences Education and Research Grants (ASERG) Reference Wilks, D.S. (2006). Statistical Methods in the Atmospheric Sciences. Elsevier Inc.