Evaluation of Skill and Error Characteristics of Alternative Seasonal Streamflow Forecast Methods Climate Forecast and Estimated Initial Soil Moisture Forecast Ensemble Lead time = 8 months ENSO PDO Run Initialized Hydrologic Model Ensemble Streamflow Forecast Select Temperature and Precipitation Data from Historic Record Associated with Forecast Climate Category Climate Forecast Schematic for Streamflow Forecasting Methodology Using VIC Hydrologic Model and Resampled Observed Data Elevation (m) The Dalles, OR Effects of the Pacific Decadal Oscillation (warm and cool epochs) and ENSO (red El Niño, blue La Niña, green ENSO Neutral) on Columbia River Summer Streamflows at The Dalles, OR. Cool Warm Interpreting Ensemble Streamflow Forecasts The figures above show a time line for the long range streamflow forecasts, and a climatology based on simulated streamflows from To produce a forecast, a categorical climate forecast and initial soil conditions are specified on October 1 of each water year. Then meteorological sequences are resampled from the historic record from according to the climate forecast, and the initialized hydrologic model is used to simulate the natural streamflow for each of these resampled climate records. The forecasts are informed both by initial conditions and the climate forecasts. The primary objective of the forecast is to predict April-September average runoff for the Columbia River at The Dalles, OR, an important quantity in the Columbia’s management scheme. The lead time from the forecast to peak flows in mid summer is about eight months in this case study. Longer lead times of up to 12 months are also possible with very similar results. Overview of Streamflow Forecasting Methods Long-lead forecasts of the Pacific Decadal Oscillation (PDO) and ENSO, global climate phenomena which exert strong controls on Pacific Northwest winter climate, are used to produce streamflow forecasts with 8- month lead times using techniques shown in the schematic below. A long retrospective streamflow forecast was made for water years using the VIC hydrologic model. A perfect categorical forecast of winter average ENSO state (warm, neutral, cool) is assumed, and heuristic methods of predicting relatively uncertain decadal changes in the PDO are used. Simulated soil moisture storage on October 1 is quantified using the hydrologic model driven by observed meteorological data. The climate forecasts contain intentional misclassifications of the PDO for some years based on methods described in Hamlet and Lettenmaier (1999, 2002). 1998, 1999, 2000 are classified as cool PDO based on the high flow event in 1997, although this classification remains uncertain. Schematic and Timeline for Streamflow Forecasts Dept. of Civil Engineering, UW, Box , Seattle, WA PDO/ENSO Based Long-Range Streamflow Forecasts JISAO Center for Science in the Earth System Climate Impacts Group and the Department of Civil and Environmental Engineering, University of Washington Key to Forecast Graphics Year shown on each plot is water year (Oct-September). Y axis is monthly mean streamflow in cfs. Gray lines are the ensemble forecast members, light black lines are the highest and lowest values from the simulated climatology, the red lines are simulated natural streamflows for the forecast year (from VIC), and the blue lines are the ensemble mean of the forecast ensemble in each case. Skill of the forecast for April-September streamflow relative to the 50-year climatology (shown at left) is displayed in each figure. Positive values are superior to the climatology. Negative values are inferior to the climatology. April-September. Note that all values are hydrologic simulations. Quantifying Forecast Performance Brier Based Skill Score Quantitative performance of the ensemble forecasts relative to the 50-year simulated climatology is assessed using a Brier- based skill score defined as follows: Skill = 1 - [ 1/N1 *  (obs-forecast) 2 / 1/N2 *  (obs-climatology) 2 ] where the summations in the second term are over all forecast/observation pairs in each case. This performance metric rewards accuracy but punishes higher variability in the forecast ensemble. Using this metric, a perfect forecast would have a skill of 1.0, whereas forecast performance equivalent to using a climatological ensemble would have a skill of 0. Negative values of the skill score indicate inferior performance in comparison to a climatological forecast. In the results shown at the right, the skill in predicting cumulative streamflow from April-September is assessed. Conclusions PDO/ENSO long-lead forecasts for the Columbia basin are more skillful than climatology during warm and cool ENSO events in 20 out of 34 realizations, but are neither as skillful nor as robust as January 1 forecasts based on persistence of hydrologic state (soil moisture and snow). Jan 1 ESP forecasts are more skillful than climatology in 29 out of 31 realizations. The preliminary results suggest that in ENSO neutral years there is little forecast skill in comparison with climatology. Sensitivity of the performance metrics to small sample size and the presence of one extremely high flow year (1997) in the ENSO neutral ensemble are probably exacerbating this effect. The crude epochal PDO forecasts used in this study improve and degrade the skill of the Oct 1 ENSO forecasts with about equal frequency, which suggests that interannual PDO forecasts should be considered as an alternative approach. Streamflow (cfs) Alan F. Hamlet Dennis P. Lettenmaier VIC Simulated Climatology Overview Streamflow forecasts in the Columbia River basin based on statistical relationships between snow accumulation and spring and summer streamflow have been used operationally for decades. More recently, experimental long-lead streamflow forecasting systems that use climate information associated with the El Niño Southern Oscillation (ENSO) and the Pacific Decadal Oscillation (PDO), have been developed. In retrospective evaluations, the potential utility of these long-lead (~ 12 months) forecasts for water resources management has been demonstrated, largely because they can provide information about future hydrologic conditions prior to the winter snow accumulation season. Aside from the obvious differences in useful lead-time in the two types of forecasts (which has value in itself in many applications), the error characteristics of the two kinds of forecasting systems are also different. Statistical forecasts, for example, generally do not have substantial skill until mid winter, but are robust to large forecast errors, because much of the precipitation that will ultimately produce runoff in spring and summer is stored in the snowpack at the time the forecast is made. To better understand the implications of these differences in skill and error characteristics, three sets of retrospective ensemble forecasts for the Columbia River at The Dalles, OR are produced and evaluated. The first set of forecasts are simulated using an assumption of perfect advance knowledge of ENSO, but are made on Oct 1, prior to accumulation of the winter snowpack, so in situ observations are used only to specify the soil moisture initial condition. The second forecast further conditions the Oct 1 forecast by including PDO information. The third set of forecasts represents a forecast that could be made on January 1 if perfect information were available about the snowpack accumulated to that time, but with no forecast of future climate. The relative skill in comparison with climatology (using a skill score defined below), and sources of error in these forecasts are compared quantitatively for a 50-year period. ENSOPDO/ENSOJanuary 1 ESP Warm ENSO Water Years ENSOPDO/ENSOJanuary 1 ESP Cool ENSO Water Years Summary and Discussion of Results For the skill metric used here, Oct 1 forecasts based primarily on climate forecast information are generally of lower skill than forecasts based on simulated hydrologic state variables available on Jan 1. For warm ENSO years, for example, PDO/ENSO forecasts on Oct 1 have a skill score greater than 0.25 in 7 out of 17 realizations, whereas the Jan 1 ESP forecasts have skill greater than 0.25 in 16 out of 17 realizations. The error statistics for cool ENSO years are comparable. In ENSO neutral years, PDO/ENSO forecasts on Oct 1 are shown to have lower skill than climatology almost all the time, whereas Jan 1 ESP forecasts in ENSO neutral years (for which data are not yet available) are presumably equally skillful in comparison with warm and cool ENSO years. The skill of the PDO/ENSO Oct 1 forecasts in ENSO neutral years is strongly influenced by small sample size and the inclusion of 1997 in the ENSO neutral ensemble, which increases the mean square error. This limitation in the analysis suggests that future work to better represent the probability distribution of the PDO ENSO forecasts in using Monte Carlo techniques may alter these preliminary results significantly by improving the robustness of the skill scores of the Oct 1 climate-based forecasts. The very crude PDO “forecasts” employed here (with intentional errors in classification) improve or degrade the skill of ENSO based forecasts on Oct 1 with about equal frequency. For warm and cool ENSO years, for example, including PDO information improves the skill of the forecast in 18 out of 34 realizations. These apparent changes in forecast skill are difficult to interpret in that the skill metrics are strongly influenced by small sample sizes and the presence of “outliers” in the ensemble (as discussed above). Including the PDO frequently alters the probability distribution of the Oct 1 forecast in a useful manner, but also reduces the sample size, which in turn limits the ability to properly interpret the skill metrics employed. Alternate methods of forecasting and incorporating PDO information in climate forecasts is an active area of research, and there appear to be opportunities to forecast variations in the PDO on interannual time scales, as opposed to the epochal, decadal-scale variations used to represent the PDO in this study. In any case, the need for Monte Carlo experiments to improve the representative sample size of the Oct 1 PDO ENSO forecasts is apparent in these experiments. Results