1. of Temperature in the San Francisco Bay Area
Statistical Evaluation of High-resolution Numerical Weather Model Forecasts
of Temperature in the San Francisco Bay Area
Yilin Lu & Dave Dempsey
Department of Earth & Climate Sciences
Introduction
Why are weather predictions not always accurate? Inspired by this question, I pursued this research with the ultimate goal of improving a forecast model by first spotting its potential defects. In particular, I used the Weather Research & Forecasting (WRF) model.
The Weather Research & Forecasting (WRF) model can make weather forecasts on user-specified bounded regions in space (domains) with high spatial and temporal resolution. In this project, we begin by looking to see whether forecast errors worsen the further ahead in time that the model makes forecasts. We used weather station observations to evaluate the accuracy of the model temperature forecasts in the San Francisco Bay Area by posing and testing the following statistical hypothesis:
Strategy: Test H0 statistically at the 95% confidence level
Statistical Evaluation
How do we evaluate forecast accuracy?
Mean Absolute Error (MAE) =
Problem 1: Model grid points and observations are at different locations (Fig.1).
Solution: Interpolate forecasts from grid points to observation points
Problem 2: In a plot of MAE against forecast hour, two patterns appear: a 6 hour cyclic pattern and an increasing trend (Fig. 2).
The non-random cyclic pattern interferes with our hypothesis test of statistical
significance of the trend. (The linear regression residuals must be random.)
Solution: Remove the cyclic pattern
by applying a 6-hour “moving average”
to mean absolute errors (Fig. 4).
Problem 3: MAEs at successive hours, like both observations and forecasts, are autocorrelated (i.e., not entirely independent).
This dependence will interference with our hypothesis test of statistical significance of the trend. (The linear regression residuals must be independent.)
Solution: Calculate the autocorrelation function for MAE vs. forecast hour. Use it to identify a sampling interval to ensure sample independence. Result: Sample every 5 forecast hours.
Problem 4: In Figs. 2 & 4, each MAE comprises an average from 4 forecasts/day for 111 days, which sacrifices independent information from individual forecasts. We want to use this information, but MAEs at successive days are also autocorrelated.
Solution: (a) Average MAEs for each day (4 forecasts/day) to remove the diurnal cycle; (b) Calculate the autocorrelation function for daily average MAE vs. day. Use it to identify a sampling interval to ensure sample independence. Result: Sample one-day MAE average every 3 days.
(Fig.5 shows the results of solutions to both Problems 3 & 4)
We are now ready to test our hypothesis about the trend of MAE vs. forecast hour.
Statistical t-test & Results
Null Hypothesis: The trend (slope) of MAE vs. forecast hour ≤ 0 oC/hr
Select confidence level: 95%
Simple size of MAE = 297 (degrees of freedom = 295)
Estimate the slope using least squares linear regression (result: oC/hr)
Calculate a sample t-statistic:
(estimated slope – hypothesized slope) / standard error of the slope
Table 1: Results of right-tailed t-test
Figure 5: Mean absolute error in forecast temperature, sampled every 5 forecast hours and every 3 days, in San Francisco Bay Area from January 1 to April 22
Research Question:
Do WRF model temperature forecasts become less and less accurate as forecast time increases?
Null Hypothesis (H0):
No, the model won’t make worse forecasts as forecast time increases.
Figure 2: Mean absolute error (MAE) in temperature from 4 runs/day for 111 days, vs. forecast hour.
Figure 3: MAE in temperature from 111 days for the10 pm forecasts only, vs. forecast hour.
Why there is a 6 hour cyclic pattern?
Because there is a 12 hour semi-diurnal pattern in forecast errors, and model is run every 6 hours.
Methods
Model Configuration
WRF = Initial Weather Conditions Changes over a period
Forecast (Solving a set of equations)
(at discrete (Lower-resolution
grid points) National Weather Forecast boundary
Service forecast model) conditions on the domain
Our model forecast runs:
Daily frequency: 4 times/day (4 am, 10 am, 4 pm, & 10 pm PST)
Forecast length: 48 hours
Output interval: 1 hour
Test period: 111 days from January 1 to April 22
Weather Observations
Source: Meteorological Assimilation Data Ingest System (MADIS)
Numbers of stations: About 90 in San Francisco Bay Area domain
Quality Control: Reject missing and unreliable data
provides
Confidence Level
Critical t
Sample
t-Statistic
Result of test
95%
1.65
2.461
Reject H0
required for
provides
<
Discussion & Conclusion
The result indicates that we can reject the null hypothesis and conclude with 95% confidence that the WRF model shows a statistically significant positive trend in forecast errors vs. forecast hour. Although the trend seems relatively small (0.24oC in 48 hours), the impacts of decreasing accuracy with respect to forecast hour will depend on the application.
Additional observations leading to future research questions:
Forecast errors appear at the initialization time. We don’t know how this might affect the trend.
The semi-diurnal pattern in MAE was a surprising discovery. Understanding the cause could lead to improvement of model forecasts.
Many more!
Figure 1: Low-resolution model grid (right, ), and the high-resolution WRF model grid for our San Francisco Bay Area domain (left, ), where there is one model forecast temperature at each grid point. Color-filled contours show an example of a forecast temperature pattern.
Blue stars (left, ) show locations of weather stations used to evaluate the model forecasts.
Figure 4 (right): Temperature MAE smoothed with a 6-hour moving average (compare with Fig. 2). A least squares linear regression line is fit to the data.
Acknowledgements
Atmospheric Sciences Education and Research Grants (ASERG)
Reference
Wilks, D.S. (2006). Statistical Methods in the Atmospheric Sciences.
Elsevier Inc.

2. WRF = Initial Weather Conditions + Changes over a period Forecast (Solving a set of equations) (at discrete (Lower-resolution grid points) National Weather Forecast boundary Service forecast model) conditions on the domain
provides
required for
provides

3. California
Terrain Elevation
(oC)