Are Numerical Weather Prediction Models Getting Better? Cliff Mass, David Ovens, and Jeff Baars University of Washington

Getting Better? The overall performance of numerical weather prediction has gotten better over the past several decades. But such improvements could have come from a number of sources: Increased resolution Better data assimilation Improved global data assets (e.g., satellite data) Better models and associated parameterizations.

The Question of This Talk To what degree are our modeling systems IMPROVING in their ability to deliver better simulations and forecasts?

Additional Question To What Degree Do We Actually KNOW how much our models are improving?

Most modeling systems have shown a flattening of the improvement curve the last 5 years or so—even with improving data for initialization and increasing resolution

ECMWF Flat

GFS and NAM Heavy Precipitation Threat Scores: Flat-lined

500 hPa Day 2 Most Global Models Show Minimal Improvement the Last 5 Years

What about mesoscale models like WRF?

Developmental Testbed Center (DTC) Reference Configuration Tests and Comparisons (3.6 versus 3.4) Purple: 3.6 Green: 3.4 Upper Air Temperature

No real overall improvement between WRF 3.4 and 3.6

Let’s Check Out the Long-Term Verification of the Northwest Regional Prediction System Frozen MM5 driven by operational National Weather Service NAM Continuous upgraded WRF driven by operational NWS GFS and recently by RAP So we are driving the local models with supposedly improving national guidance.

July 1, 2006 June 1, 2015

Bottom Line: No evidence of significant improvement in UW WRF over 9 years, even with synoptic models getting slightly better

But let’s do this more rigorously Run a history of WRF models from substantially different “eras” using the SAME large scale forcing. Jan 1-Feb 28, 2010, Pacific Northwest Domain Attempted to keep physics parameterizations consistent (but they are being updated as the modeling system changes)

WRF Versions Tested October 31, July 31, April 18, April 20, 2015

A stroll through WRF history WRF version with Kain-Fritsch cumulus parameterization, simple ice physics, YSU PBL, Noah LSM, and RRTM longwave and Dudhia shortwave radiation WRF version with Kain-Fritsch cumulus parameterization, Thompson microphysics, YSU PBL, Noah LSM, and RRTM longwave and Dudhia shortwave radiation WRF version 3.5 with Kain-Fritsch cumulus parameterization, Thompson microphysics, YSU PBL, Noah LSM, and RRTMG longwave and shortwave radiation WRF version 3.7 with Kain-Fritsch cumulus parameterization, Thompson microphysics, YSU PBL, Noah LSM, and RRTMG longwave and shortwave radiation

Some Results

Sea Level Pressure (24h Forecast) WINS Forecasts GET WORSE Over Time

Wind Speed 3.5 and 3.7 Better Than Early Versions 3.5 lower MAE than 3.7

2-m Temperature has lowest MAE, Worst MAE and ME

Wind Direction 3.5 best, followed by better than 3.11

Conclusions (based on many more stats) The WRF modeling system does not get steadily better, but staggers forward in a halting manner A new version of the modeling system is not necessarily better. Some statistics have generally improved, others have not. Precipitation (not shown above) has been one of the winners (improvement over the last decade). Positive Definite advection helped.

Other modeling systems are no better than WRF

Why Slow Progress? Some Possibilities NWP is now a mature technology and we should not expect significant advances There is no organized, coherent system in place to improve physics parameterizations and model numerics/structure. There is a lack of organized verification/evaluation of model simulations to find problems.

Without detail evaluations of the strengths of weaknesses of our modeling system are we driving blind?

Our Challenge We need a completely new paradigm on how we develop and fund NWP model development, and how we apply the enormous resources of our community more effectively. Need to bring together the huge, uncoordinated NWP research effort in the U.S Reduce the number of models, determine the key weaknesses in NWP systems, and direct resources to rectify them.

The END