1. Ensembling Medium Range Forecast MOS GUIANCE By Richard H. Grumm National Weather Service State College PA 16803 and Robert Hart The Florida State University

2. Introduction Model Output Statistics (MOS) –Regression equations of parameters from models to make a forecast for a point. –statistical equations were specifically tailored for each location, taking into account factors such as local climate. MOS equations are based on output from a single model –Models have bias and errors –This will affect, though some statistical corrections, the MOS output –Timing errors and intensity errors will affect the outcome

3. Introduction-II Verification Issues –Base line the extended MOS and ENSEMBLE of extended MOS with more widely used products FWC - NGM based MOS MET – Eta based MOS MAV – GFS based MOS –Limited at this time to temperatures only Average (bias or avg ), Mean Absolute Error ( MAE ) and root-mean square error ( RMSE ). Primary Goal is to evaluate the longer term MOS guidance and an Ensemble –But the shorter term MOS helps illustrate the point –Helps us further learn about ensembles and ensemble strategies.

4. Medium Range Forecast MOS Produced from data from the Global forecast system (GFS) at the highest resolution of the model –Known as the Extended Range GFS MOS (MEX) –Message contains routine variables and a climate range of data at some locations at the end of the message –Most widely used extended range MOS product –Based on the highest resolution forecast model

5. MEX Ensemble MOS NCEP runs the GFS and has an GFS ensemble prediction system (EPS) –MOS is generated for each GFS EPS member –The control run is slightly coarser resolution than the operational GFS –The control is used to produce perturbed members There are 5 positive (P1-P5) and 5 negatively (N1-N5) perturbed members based on the control run This provides 12 GFS runs to produce MEX guidance the operational MRF MOS prediction equations are applied to the output from each of the ensemble runs

6. Why Ensemble MOS? The operation GFS and its MOS – are expected to be more accurate – Due to higher resolution which we expect to be more accurate – But may pay for timing and intensity errors as finer scale systems typically correlate less in space. The ENSEMBLE of the MOS…. –Will show a range and times of high disagreement ~ UNCERTAINTY

7. Ensembles help us quick review of ensemble concepts Deal with uncertainties in data (initial conditions) – the ability to properly resolve the feature Deal with uncertainties in data verse resolution of the model – 6  rule, we may under sample a system. Deal with uncertainties in physics – Current GFS system is based solely on the same model – Only variation is initial conditions

8. GFS EPS with varied initial Conditions Forecast Length Forecasts Initialized at most recent data time Envelope of solutions at single time Solution

9. Displaying uncertainties in forecasts In Model output: – spaghetti plots and probability charts (the most likely outcome) – consensus forecast charts, the middle ground, with dispersion (standard deviation about the mean) – to visualize these is to see limits of any single solution. MOS OUTPUT: – Unless plan view maps, does not lend itself well to spaghetti plots – In text bulletins, the dispersion about the mean may show uncertainty – Consensus and the range of possibilities are good candidates for display Why ensemble MOS output 

10. Consider this Consider this: Would you want to shoot one arrow at the bullseye or a quiver full of arrows at the bullseye. Or…pick the MEX and maybe miss the mark or use the ensembles and have a better chance of approximating something something.

11. Producing Ensemble MOS Assume all members of equal skill – Any single member may be most correct at any single time frame – No a priori knowledge as to which would be best member on any give day or forecast length Decode each product – Computer sums, sums of squares etc Text variables are assigned numbers – Clouds: CL = 0; SC: 33 BK: 66 and OV is 100 – Java Object can translate number to letter or vise/verse – Get range – Produce consensus and dispersion of forecasts about the mean. For select parameters show : – MEX value, consensus value, highest and lowest members value – Current system has no weights applied.

12. Extended MOS Formatting is lost from web

13. Short Term Ensemble MOS KAOO

14. Verifying Ensemble MOS At select locations compute – Mean errors (bias) – M ean a bsolute e rror (MAE), and – R oot m ean s quare e rror (RMSE) How to view data with 12 members per site and 150 sites verified for this study? Lots of graphs, currently at single sites.

15. Reference values CTP 1-4 day forecasts Dec 2003 MOSBIAS MAE MAV 0.742.96 ETA 0.783.73 FWC 0.143.58 CCF 0.683.33

16. Short Term MOS Verification Examine MAV, ETA, FWC –06 and 18 UTC MAV provide 18,30,42,54 and 66 hour forecasts –ETA,FWC,MAV at 0000 and 1200 UTC provide 24,36,48,60 and MAV 72 hour forecasts –So, 6 hour forecast intervals for MAV Not to belabor point, but each MAV update clearly improves on previous as skill decreases with time. The false belief that the 06 and 18UTC MAV is unfounded. Select sites are shown though data exists for about 100 sites No scoring of ensemble products is accomplished here

17. KMDT AVG-MAE FWC cold bias MAV lower MAE

18. KMDT RMSE MAV lower RMSE

19. KIPT AVG-MAE FWC cold bias MAV lower MAE

20. KIPT RMSE MAV lower RMSE

21. KBOS AVG-MAE FWC cold bias The ETA/FWC bias are opposite ~enemble potential ? MAV slightly lower MAE

22. KBOS RMSE MAV lower RMSE But Eta is close

23. Short Term MOS findings At sites examined (3 shown): – MAV is the most skillful temperature forecast MOS – FWC has cold bias at many (most) sites – Eta MOS has some regional/local variation and is more competitive with MAV as some sites with opposite bias from FWC  this may lend itself to ensembling. – Clear skill differences at some sites where MAV is far superior  May limit ensembling without weighting at these sites as straight blend would weaken impact of more accurate member

24. Medium Range MOS Similar display concepts Same observational data sets used Plot all 12 members –P members are RED N members are blue –Focus on the 3 best members: MEX with finest detail forecast (GREEn) Ensemble mean (BLACK) Control Run (YELLOW)

25. KMDT BIAS JA-FE 2004 MEX consensus MEX-Control

26. KMDT MAE JA-FE 2004 Early on MEX is far superior!

27. KMDT RMSE JA-FE 2004

28. Allentown MAE

29. Atlantic City MAE

30. Boston MAE

31. LGA MAE

32. KABE MAE

33. BFD MAE

34. Some findings RMSE at 24-48 hours comparable to those found in other MOS products 3 to 4 degrees – May be a bit better than MAV at 24 hours at KMDT MEX clearly outperforms all members and consensus at most sites 24-60 hours – Ensembling unskillful members is not helpful? MEX and Consensus both very good at 60+ hours – MEX more skillful at some sites than consensus, but consensus more skillful at others – Consensus, treated as a single forecast member is quite a skillful member at all locations after 60 hours Weighting the consensus toward the more skillful members might improve the forecasts

35. Ensemble Envelope Useful to know how often the warmest and coldest members captured the range of solutions…. At mid-range forecasts –50-60% of the time the observed temperature is within the forecast range of all 12 members –60% of the time at longer ranges, the observed temperature is outside of the range of the ensemble range –Artifact of untuned MOS for GFS EPS members?

36. KMDT when observed Temperature within range of Ensemble MOS forecasts

37. KBFD when observed Temperature within range of Ensemble MOS forecasts

38. KBOS when observed Temperature within range of Ensemble MOS forecasts

40. A few more items Examining just January –The control run was slightly more skillful at many sites than the MEX –The consensus was typically more skillful than the MEX –A pattern change likely contributed to this, however over 2 months this problem was mitigated.

41. January Potential Case Study Cold snap – caused timing errors in MEX  CTL run benefited as did Consensus from the changes

42. Short Term MOS findings At sites examined (3 shown): – MAV is the most skillful temperature forecast MOS – FWC has cold bias at many (most) sites – Eta MOS has some regional/local variation and is more competitive with MAV as some sites with opposite bias from FWC  this may lend itself to ensembling. – Clear skill differences at some sites where MAV is far superior  May limit ensembling without weighting at these sites as straight blend would weaken impact of more accurate member

43. Conclusions Short Term MOS findings: – MAV is the most skillful temperature forecast MOS – FWC has cold bias at many (most) sites – Eta MOS has some regional/local variation and is more competitive with MAV as some sites with opposite bias from FWC – Clear skill differences at some sites where MAV is far superior Medium Range Ensemble MOS was generated – From collections MOS forecasts from Various NCEP EPS model runs And NCEP short range forecast models Each model produced had different initial conditions – ensemble mean (consensus) temperature forecasts were skillful and competitive with the MEX forecasts at all sites after 60 hours

44. Conclusions-II Limitations – Ideally, the ensemble MOS would beat the MEX at all sites after the initial time periods. – It does not implying over the long haul: The MEX is more skillful than the other members – Especially at 24 to 60 hours! The MEX equations are tuned to the operational GFS and are not tuned to the GFS EPS members. Tuned equations for each member might improve the MOS guidance for each member and the ensemble MOS system in general. – Observed temperatures often falls outside the ensemble envelope 60% of the time at longer ranges Is this a problems? – What is the value and what are the limitations of ensembling unequally skillful members?

45. Conclusions-III Operational Applications – Consensus and the dispersion about the mean show times of large uncertainty – Forecasters need to apply knowledge of this uncertainty in forecasts – And this information needs to be conveyed to the users of these forecasts – Times of uncertainty are times of the ensemble providing more value to the forecast process. Plans – Apply same technique to verify the POP forecasts from these data – Experiment with weights to improve the consensus forecasts. – Improve the verification methods and software

46. NCEP EPS Breeding N SEEDS GIVE 2*N PERTURBATIONS Scaled + perturbation Initial random seed Opposite sign is negative perturbation Adjust magnitude to typical analysis errors 12-h forecast CONTROL-CTL Complete cycle forecast