Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory Roux, Wanli Wu, Will Cheng, Jason Knievel, Sue Haupt

Army Test and Evaluation Command: Dugway Proving Ground

Dugway Proving Grounds, Utah e.g. T Thresholds Includes random and systematic differences between members. Not an actual chance of exceedance unless calibrated.

Xcel Energy Service Areas Wind Farms (50+) ~3200 MW Northern States Power (NSP) Public Service of Colorado (PSCO) Southwestern Public Service (SPS) 3.4 million customers (electric) Annual revenue $11B Copyright 2010 University Corporation for Atmospheric Research

WRF RTFDDA Model Domains Ensemble System (30 members) D1 = 30 km D2 = 10 km 0-48 hrs Real Time Four Dimensional Data Assimilation (RTFDDA) 41 vertical levels Vary: Multi-models Lateral B.Cs. Model Physics External forcing Yubao Liu -- for further

Goals of an EPS Predict the observed distribution of events and atmospheric states Predict uncertainty in the day’s prediction Predict the extreme events that are possible on a particular day Provide a range of possible scenarios for a particular forecast

Outline I.Brief overview of: 1) quantile regression (QR), 2) logistic regression (LR), 3) umbrella post- processing procedure, 4) “analog Kalman filter” (ANKF) II.2 nd moment calibration via rank histograms III.Skill score comparisons and improvements with increased hindcast data III.Example of blending approaches IV.Conclusions

Example of Quantile Regression (QR) Our application Fitting T quantiles using QR conditioned on: 1)Ranked forecast ens 2)ensemble mean 3)ensemble median 4) ensemble stdev 5) Persistence Hopson and Hacker 2012

Logistic Regression for probability of exceedance (climatological thresholds)

Probability/°K Temperature [K] Probability/°K Temperature [K] Forecast PDF climatological PDF Step I: determine climatological quantiles Probability Temperature [K] Step 3: use conditioned CDF to interpolate desired quantiles prior posterior Step 2: calculate conditional probs for each climat quan Probability Temperature [K] Final result: “sharper” posterior PDF represented by interpolated quans Hopson and Hacker 2012

T [K] Time forecastsobserved Regressor set: 1. reforecast ens 2. ens mean 3. ens stdev 4. persistence 5. LR quantile (not shown) Probability/°K Temperature [K] climatological PDF Step I: Determine climatological quantiles Step 2: For each quan, use forward step-wise cross-validation to select best regress set Selection requires: a) min QR cost function, b) binomial distrib at 95% confidence If requirements not met, retain climatological “prior” Step 3: segregate forecasts based on ens dispersion; refit models (Step 2) for each range Time forecasts T [K] I.II.III.II.I. Probability/°K Temperature [K] Forecast PDF prior posterior Final result: “sharper” posterior PDF represented by interpolated quans Hopson and Hacker 2012

National Security Applications Program Research Applications Laboratory Significant calibration regressors 3hr Lead-time 42hr Lead-time Station DPG S01

National Security Applications Program Research Applications Laboratory RMSE of ensemble members 3hr Lead-time 42hr Lead-time Station DPG S01

Time t = 0 day -1 day -2 day -6 day -5 day -4 day -3 day -7 OBS PRED KF-weight KF Delle Monache et al Analog Kalman Filter (ANKF) Deterministic method applied to each individual ensemble KF weighting run in analog space

Time t = 0 day -1 day -2 day -6 day -5 day -4 day -3 day -7 OBS PRED KF-weight KF ANKF AN “Analog” Space day -4 day -7 day -5 day -3 day -2 day -1 day -6 PRED OBS farthest analog closest analog NOTE This procedure is applied independently at each observation location and for a given forecast time Delle Monache et al. 2010

Outline I.Brief overview of: 1) quantile regression (QR), 2) logistic regression (LR), 3) umbrella post- processing procedure, 4) “analog Kalman filter” (ANKF) II.2 nd moment calibration via rank histograms III.Skill score comparisons and improvements with increased hindcast data III.Example of blending approaches IV.Conclusions

42-hr dewpoint time series Before Calibration After Calibration (QR) Station DPG S01

Original ensemble QR LRANKF Rank Histograms 15hr lead wind forecasts

Skill measures used: 1)Rank histogram (converted to scalar measure) 2)Root Mean square error (RMSE) 3)Rank Probability Score (RPS) 4)Relative Operating Characteristic (ROC) curve Skill Scores Comparing to original ensemble forecast, but with bias removed => “reference forecast”

Blue - QR Red - ANKF Green - LR Skill Score Comparison For wind farm CEDC, 3hr lead forecasts Reference forecast: original wind speed ensemble w/ bias removed Data size: 900pts

Rank Histogram scalar QR LR RMSE Skill Scores Dependence on Training Data Size Upper dashed – 900pts Solid – 600pts Lower dashed – 300pts Reference Forecast: Original wind speed ensemble w/ bias removed

ROC QR LR RPSS Skill Scores Dependence on Training Data Size (cont) Reference Forecast: Original wind speed ensemble w/ bias removed Upper dashed – 900pts Solid – 600pts Lower dashed – 300pts

23 RPS ROC RMSE Brier Score Wind farm TWBT ANKF QR QR + ANKF

24 original ANKF QR QR + ANKF TWBT 6-h

Summary  “step-wise cross-validation”-based post-processing framework provides a method to ensure forecast skill no worse than climatological and persistence  Also provides an umbrella to blend together multiple post-processing approaches as well as multiple regressors, and to diagnose their utility for a variety of cost functions  Quantile regression and logistic regression useful tools for improving 2 nd moment of ensemble distributions  See significant skill gains with increasing “hindcast data” amount for a variety of skill measures  Blending of post-processing approaches can also further enhance final forecast skill (e.g. ANKF and QR) by capturing “best of both worlds” Further questions: or