A Multimodel Streamflow Forecasting System for the Western U.S. Theodore J. Bohn, Andrew A. Wood, and Dennis P. Lettenmaier University of Washington, U.S.A. EGU Conference, Spring 2006 Session HS23/NP5.04

Outline Background Multi-model vs Individual Models UW West-Wide Forecasting System Bayesian Model Averaging Multi-model vs Individual Models Deterministic Retrospective Forecasts ESP Retrospective Forecasts

Background UW West-Wide Stream Flow Forecast system (Wood and Lettenmaier, in review; Wood et al, 2002) Developed in partnership with USDA/NRCS NWCC Long-lead-time (1-12 months) stream flow forecasting for western U.S. Main component: Variable Infiltration Capacity (VIC) large-scale hydrological model Probabilistic forecasts Uses forecasts from multiple climate models to take into account climate uncertainty Does not yet take into account uncertainty in hydrologic model physics

Background Immediate goal: improve forecast skill at long lead times (1-12 months) Problems: Uncertainty grows with lead time Greater uncertainty when making forecasts before the snow pack has accumulated How much of this uncertainty is due to hydrologic model physics?

Seasonal Hydrologic Forecast Uncertainty in Western US Importance of uncertainty in ICs vs. climate varies with lead time … IC error low climate fcst error high IC error high climate fcst error low Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Uncertainty Forecast actual perfect data, model streamflow volume forecast period model + data uncertainty low high ESP addresses climate uncertainty, but the single model/calibration framework doesn’t address IC uncertainty -- ignoring calibration issues at moment – assuming reasonably well calibrated models can be further adjusted via bias-correction -- issues regarding estimating inputs as an ensemble? … hence importance of model & data errors also vary with lead time.

How to quantify uncertainty and reduce bias? Multi-model ensemble Average the results of multiple models – reduces bias Ensemble mean should be more stable than a single model Combines the strengths of each model - generally as good as the best model at all times/locations Provides estimates of model uncertainty

Expansion to multiple-model framework Seasonal Climate Forecast Data Sources CCA NOAA CAS OCN CPC Official Outlooks SMLR CA Coupled Forecast System VIC Hydrology Model NASA NSIPP/GMAO dynamical model ESP ENSO UW ENSO/PDO

Expansion to multiple-model framework Multiple Hydrologic Models CCA NOAA CAS OCN CPC Official Outlooks Model 1 SMLR CA Coupled Forecast System Model 2 NASA NSIPP/GMAO dynamical model Model 3 ESP weightings calibrated via retrospective analysis ENSO UW ENSO/PDO

Averaging of Forecasts Bayesian Model Averaging (BMA) (Raftery et al, 2005) Ensemble mean: E(y|f1,…fK) = Σwkfk where y = observation fk = forecast of kth model wk = weight of kth model = expected fraction of data points for which kth model forecast is best of the ensemble Ensemble variance, for forecast at time t: Var(yt|f1t,…,fKt) = Σwk(fkt - Σwifit)2 + Σwkσk2 σk2 = uncertainty of kth model, conditional on kth model being the best = weighted mean square error (MSE), favoring data points for which kth model forecast is best of the ensemble Spread due to model uncertainty Spread among models

Averaging of Forecasts wk, σk reflect uncertainty due to model physics p(y|f1,…f3) Model 1 Model 2 Model 3 σ1 σ2 σ3 p(y|f1) f1 f2 f3 p(y|f2) p(y|f3) Σwkfk w1f1 + w2f2 = Multimodel Average + w3f3

Computing Model Weights Parameters wk and σk wk and σk depend on each other computed via iterative maximum likelihood method Currently: determined from model performance in a retrospective deterministic simulation Future: determine from performance of retrospective probabilistic forecasts The σk help define a distribution about the multimodel average Reflect model uncertainty This method assumes normally-distributed data Discharge in W. U.S.A tends to have positive skew Therefore: Generate monthly wk and σk from log-transformed discharge Form multimodel average from log-transformed forecasts Transform multimodel average (and distribution) back to flow domain Emphasize that w_k and sigma_k both reflect a model’s BEST performance

UW West-Wide Forecast Ensemble Models: VIC - Variable Infiltration Capacity (UW) SAC - Sacramento/SNOW17 model (National Weather Service) NOAH – NCEP, OSU, Army, and NWS Hydrology Lab Model Energy Balance Snow Bands VIC Yes Yes SAC No Yes NOAH Yes No SAC does not compute PET; it uses PET computed by NOAH Data: Calibration parameters from NLDAS 1/8 degree grid (Mitchell et al 2004) – no further calibration performed Meteorological Inputs: Maurer et al. (2002), 1949-1999

Three Test Basins Salmon R. Feather R. Colorado R. (Above Snake R.) Drainage area: 33600 km2 Feather R. (Above Oroville Res.) Drainage area: 8600 km2 Colorado R. (Above Grand Junction) Drainage area: 19900 km2

Deterministic Retrospective 1956-1995 Training Period: Even Years Model Weights Monthly Mean Discharge Monthly RMSE 1.0 0.0 0.5 Salm. Colo. Feat.

Deterministic Retrospective 1956-1995 Validation Period: Odd Years Model Weights Monthly Mean Discharge Monthly RMSE 1.0 0.0 0.5 Salm. Colo. Feat.

Deterministic Retrospective Results Individual Models VIC is best in general Best at capturing autumn-winter base flow (all basins) → high weights Best estimate of snowmelt peak in Colorado basin Generally Lowest RMSE SAC is second Low autumn/winter base flow → low weights In Salmon basin, snowmelt peak flow is early but magnitude is close to observed in May → high weight Best estimate of snowmelt peak in Feather basin → high weight NOAH is last No autumn/winter base flow → low weights In Salmon and Colorado basins, snowmelt peak flow is 1-2 months early and far too small (high snow sublimation, lack of elevation bands) → low weights Competitive in Feather basin (snowmelt is less dominant here) Generally highest RMSE and lowest weights

Deterministic Retrospective Results Multimodel Ensemble Prediction In general, ensemble bias and RMSE are at least as small as those of the best individual model Notable exceptions: June in Salmon Basin, February in Feather Basin SAC beats the ensemble RMSE in both the training and validation sets – how? Model weights reflect each model’s best performance SAC consistently good here VIC not as consistent, but when it is good, it is very good → gets equal weight to SAC This warrants further investigation

ESP Forecasts Extended Streamflow Prediction Start with I.C. of forecast year Run model with ensemble of historical meteorological forcings (climatology) The distribution of results indicates uncertainty due to climate (but implicitly contains uncertainty due to the model) ESP forecast distribution Save state vector here ESP forecast typically includes median and quartile values Retrospective simulation Forecasts using climatology, starting from saved ICs

ESP Forecasts and Multimodel Approach 1: FIRST form multimodel average of all models for each forcing THEN form ESP distribution of the multimodel averages Use weights determined in the training period One multimodel distribution for each forcing Multimodel Forcing 1 Forcing 1 Forcing 2 Model 1 Add these distributions Model 2 Model 3 ESP distribution of multimodel distributions (“grand distribution”) Forcing 2

ESP Forecasts and Multimodel Approach 2: FIRST form the ESP distribution for each model THEN form multimodel average of the ESP distributions Determine wk and σk based on each model’s ESP distribution σ1 Model 1 ESP 1 w1ESP1 + σ2 Model 2 ESP 2 = w2ESP2 ESP distribution of multimodel distributions (“grand distribution”) + σ3 Model 3 ESP 3 w3ESP3

ESP Forecasts and Multimodel Approach 2 incorporates model forecast performance into the computation of wk, σk Should be more accurate Approach 1 is simpler We will start with approach 1

Example ESP Forecast, 1966-1967 S. C. F. ESP Distribution of multimodel averages S. Distributions of Individual models Spread of multimodel averages is similar to individual model ESP spreads - mainly reflects uncertainty in climatological forcings the average reflects which model is more reliable but does not quantify model uncertainty Oct Dec Feb Apr Jun Aug C. Oct Dec Feb Apr Jun Aug F. Oct Dec Feb Apr Jun Aug

Now add multimodel “grand distribution” Oct Dec Feb Apr Jun Aug Grand distribution has larger spread than distribution of multimodel averages, due to addition of model uncertainty C. ESP distribution of multimodel averages (Note: grand distribution error bars extend from 1%-ile to 99 %-ile) F.

S. C. F. Aggregate ESPs, Odd years 1956-1995 Oct Dec Feb Apr Jun Aug Mean 25th – 75th %-ile Range Grand distribution has larger range between 25th-75th %-ile range than that of the distribution of multimodel means alone. This difference reflects the contribution of model uncertainty. Grand distribution’s 25-75 range is larger than most individual models during spring snowmelt peak (May-June), reflecting range of model snow formulations. S. C. F.

Conclusions Multimodel averaging can Future work: reduce the bias of a hydrological forecast but not always – depends on the weighting scheme help quantify model uncertainty and/or identify where model uncertainty is important model snow formulation in snowmelt-driven basins Future work: Weights based on forecast performance Multimodel’s influence on dependence of skill on forecast start date

References Wood, A.W., Maurer, E.P., Kumar, A. and D.P. Lettenmaier, 2002. Long Range Experimental Hydrologic Forecasting for the Eastern U.S. J. Geophysical Research, VOL. 107, NO. D20, October. Raftery, A.E., F. Balabdaoui, T. Gneiting, and M. Polakowski, 2005. Using Bayesian Model Averaging to Calibrate Forecast Ensembles. Monthly Weather Review, 133, 1155-1174.

Model Averaging: Process Flow