1. Probability Forecasts from Ensembles and their Application at the SPC David Bright NOAA/NWS/Storm Prediction Center Norman, OK AMS Short Course on Probabilistic Forecasting January 9, 2005 San Diego, CA Where Americas Climate and Weather Services Begin

3. Outline Motivation for ensemble forecasting Ensemble products and applications –Emphasis on probabilistic products Ensemble calibration (verification) Decision making using ensembles

4. Outline Motivation for ensemble forecasting Ensemble products and applications –Emphasis on probabilistic products Ensemble calibration (verification) Decision making using ensembles

5. Daily weather forecasts begin as an initial- value problem on large supercomputers To produce a skillful weather forecast requires: –An accurate initial state of the atmosphere to begin the model forecast –Computer models that realistically represent the evolution of the atmosphere (in a timely manner) With a reasonably accurate initial analysis of the atmosphere, the state of the atmosphere at any subsequent time can be determined by a super- mathematician." (Bjerknes 1919)

6. Example: Determinism 60h Eta Forecast valid 00 UTC 27 Dec 2004 PMSL (solid); 10m Wind; 1000-500 mb thickness (dashed)

7. 60h Eta Forecast valid 00 UTC 27 Dec 2004 PMSL (solid); 10m Wind; 1000-500 mb thickness (dashed) Precip amount (in) and type (blue=snow; green=rain) Example: Determinism

8. 60h Eta Forecast valid 00 UTC 27 Dec 2004 “Truth” 00 UTC 27 Dec 2004 Example: Determinism

9. 60h Eta Forecast valid 00 UTC 27 Dec 2004 “Truth” 00 UTC 27 Dec 2004 Ignores forecast uncertainty Potentially misleading Oversells forecast capability ? Example: Determinism

10. - Ensemble forecasting can be traced back to the discovery of the "Butterfly Effect" (Lorenz 1963, 1965)… -Atmo a non-linear, non-periodic, dynamical system causes even tiny errors to grow upscale... resulting in forecast uncertainty and eventually chaos The Butterfly Effect

11. Discovery of the “butterfly effect” (Lorenz 1963) Simplified climate model… When the integration was restarted with 3 (vs 6) digit accuracy, everything was going fine until… Time

12. Solutions began to diverge Solutions diverge Time The Butterfly Effect

13. Soon, two “similar” but clearly unique solutions Solutions diverge Time The Butterfly Effect

14. Eventually, results revealed two uncorrelated and completely different solutions (i.e., chaos) Solutions diverge Time Chaos The Butterfly Effect

15. Ensembles can be used to provide information on forecast uncertainty Information from the ensemble typically consists of… (1)Mean (2) Spread (3) Probability Ensembles useful in this range! Solutions diverge Time Chaos The Butterfly Effect

16. Ensembles extend predictability… A deterministic solution is no longer skillful when its error variance exceeds climatic variance An ensemble remains skillful until error saturation (i.e., until chaos occurs) Solutions diverge Chaos Time Ensembles extend predictability Ensembles especially useful in this range! The Butterfly Effect

17. - NWP models... - Doubling time of small initial errors ~ 1 to 2 days - Maximum large-scale (synoptic to planetary) predictability ~10 to 14 days It’s hard to get it right the first time!

18. Example: Synoptic Scale Variability 7 day forecast – NCEP MREF 500 MB Height GFS “Control” Forecast GFS -12h “Control” GFS PertEuropean Model and Start

19. Reveals forecast uncertainty, e.g., se U.S. precip Sensible weather often mesoscale dominated Example: Mesoscale Variability 1.5 day forecast – NCEP SREF Precipitation

20. Sources of Uncertainty in NWP Observations –Density –Error –Representative –QC Analysis Models LBCs, etc. Satellite Land RMSD ECMWF-NCEP 500 mb Hght (5 winters)

21. Schematic Illustration: Ensemble Concepts Analysis, Model, and Subgrid-scale Errors... All equally-likely solutions All plausible atmospheric states All equally-likely ICs

22. Limit of single model skill Limit of ensemble skill Error Growth with Time: Idealized Forecast Error Expected climate variability We can use ensembles (e.g., probabilities, etc.) to extend predictability (~ 3 to 4 days for synoptic scale pattern) until the forecast becomes chaotic. No correlation to initial conditions…chaos!

23. 500 mb Hght (Dec. 2004; Greater U.S. Area) Climate SD 1.41 x Climate SD GFS Ens Means Limit of deterministic skill ~7.5 days Limit of ensemble skill ~10.5 days 12345 Days 789101112 RMSE 20 m 40 m 80 m 100 m 120 m Error Growth with Time: GFS

24. Determinism Ensemble Ensembles vs. Determinism Evaluating Weather Forecasts

25. Outline Motivation for ensemble forecasting Ensemble products and applications –Emphasis on probabilistic products Ensemble calibration (verification) Decision making using ensembles

26. Definitions SREF = NCEP Short Range Ensemble Forecast (5 Eta-BMJ; 5 EtaKF; 5 RSM) MREF = NCEP Medium-Range Ensemble Forecast (GFS) Mean = Arithmetic average of members Spread = Variance or Standard Deviation Probability = % of members meeting some condition Calibrated Probability = As above, but adjusted to reflect expected frequency of occurrence

27. SPC Approach to Ensembles Develop customized products based on a particular application (severe, fire wx, etc.) Design operational guidance products that… –Help blend deterministic and ensemble approaches –Facilitate transition toward probabilistic thinking –Aid in critical decision making Increase confidence Alert for rare but significant events

28. F15 SREF MEAN 500 MB HGHT,TEMP,WIND Ensemble Means

29. Synoptic-Statistical Relationships Mean + Spread Examples of simple relationships between dispersion patterns and synoptic interpretation can be defined. Obtain a quick overview of range of weather situations from ensemble statistics. Amplitude Location

30. F15 SREF MEAN/SD 500 MB HGHT Ensemble Mean + Spread

31. F000 F048 F096 F144 500 mb Mean Height (solid) and Standard Deviation (dashed/filled) Increased spreadLess predictabilityLess forecast confidence Ensemble Mean + Spread

32. F000 F048 F096 F144 500 mb Mean Height and Normalized Variance Normalize the ensemble variance by climatic variance Values approaching 2 (dark color fill) => Ensemble variance saturated based on climo 2 Ensemble Mean + Normalized Spread

33. F000 F048 F096 F144 500 mb Member Height “Spaghetti” - 5640 meter contour Another way to view uncertainty: Spaghetti

34. F63 SREF POSTAGE STAMP VIEW: PMSL, HURRICANE FRANCES Red = EtaBMJ Yellow= EtaKF Blue = RSM White = OpEta SREF Member

35. F15 SREF MEDIAN/RANGE CAPE At least 1 member has >= 500 J/kg All 16 members have >=500 J/kg CAPE Median Spatial Variability: Median + Range

36. F15 SREF MEDIAN/RANGE MLCAPE X 0-6 KM SHEAR Creation of Severe Wx Diagnositics - Calculated Craven-Brooks Significant Severe parameter for each member Median All 16 members have >= 10,000 m^3/s^3 At least 1 member has >= 10,000 m^3/s^3

37. Arithmetic mean… –Easy to compute and understand –Tends to increase coverage of light pcpn and decrease max values. 3-hr Total Pcpn NCEP SREF F63 Valid 09 Oct 2003 00 UTC Ways to view central value: Mean

38. Median… –If the majority of members don’t precip, will show large areas of no precip. Thus, often limited in areal extent. 3-hr Total Pcpn NCEP SREF Ways to view central value: Median

39. The blending of two PDFs, when one provides better spatial representation [e.g., ensemble mean QPF] and the other greater accuracy [e.g., QPF from all members]. See Ebert (MWR 2001) for more info. Rank Ens Mean Rank Member QPF 1  1 2  16 3  32 Ways to view central value: Probability Matching

40. Probability matching… –Ebert (2001) Found to be the best ensemble averaged QPF –Max values restored; pattern from ens mean 3-hr Total Pcpn NCEP SREF Ways to view central value: Probability Matching

41. Uncalibrated probabilities: Fraction of members meeting some condition

42. Probability 144h 2 meter Td <= 25 degF Probabilistic Output of Basic Products: 2 m Dewpoint

43. Probability 144h Haines Index > 5 Probabilistic Output of Derived Products: Haines Index

44. Probability Convective Pcpn >.01” Prob Conv Pcpn >.01” Valid 00 UTC 20 Sept 2003

45. Prob Conv Pcpn >.01” Valid 00 UTC 20 Sept 2003 Probability Convective Pcpn >.01”

46. Pcpn probs due to physics - No EtaBMJ members?! Red = EtaBMJ Yellow = EtaKF Blue = RSM Spaghetti: Different physics Note clustering by model

47. Spaghetti: Outliers F39 SREF SPAGHETTI (1000 J/KG) Red = EtaBMJ Yellow = EtaKF Blue = RSM White solid = 12 KM OpEta (12 UTC) 12 UTC operational Eta clearly an outlier from 09 UTC SREF - Is this the result of ICs or resolution? - Is this a better fcst (updated info) or an outlier

48. Extreme Values: Lowest RH 144h Minimum RH from any ensemble member “Worst case scenario”

49. MINIMUM F15 SREF MINIMUM 2 METER RH MAXIMUM F15 SREF MAXIMUM FOSBERG FIREWX INDEX Any member can contribute to the max or min value at a grid point Extreme Values

50. Combined Probability Charts Probability surface CAPE >= 1000 J/kg –Generally low in this case –Ensemble mean < 1000 J/kg (no gold dashed line) CAPE (J/kg) Green solid= Percent Members >= 1000 J/kg ; Shading >= 50% Gold dashed = Ensemble mean (1000 J/kg) F036: Valid 21 UTC 28 May 2003

51. Probability deep layer shear >= 30 kts –Strong mid level jet through Iowa 10 m – 6 km Shear (kts) Green solid= Percent Members >= 30 kts ; Shading >= 50% Gold dashed = Ensemble mean (30 kts) F036: Valid 21 UTC 28 May 2003 Combined Probability Charts

52. Convection likely WI/IL/IN –Will the convection become severe? 3 Hour Convective Precipitation >= 0.01 (in) Green solid= Percent Members >= 0.01 in; Shading >= 50% Gold dashed = Ensemble mean (0.01 in) F036: Valid 21 UTC 28 May 2003 Combined Probability Charts

53. Combined probabilities very useful Quick way to determine juxtaposition of key parameters Not a true probability –Not independent –Different members contribute Prob Cape >= 1000 X Prob Shear >= 30 kts X Prob Conv Pcpn >=.01” F036: Valid 21 UTC 28 May 2003 Combined Probability Charts

54. Severe Reports Red=Tor; Blue=Wind; Green=Hail Prob Cape >= 1000 X Prob Shear >= 30 kts X Prob Conv Pcpn >=.01” F036: Valid 21 UTC 28 May 2003 Combined probabilities a quick way to determine juxtaposition of key parameters Not a true probability –Not independent –Different members contribute Fosters an ingredients-based approach on-the- fly Combined Probability Charts

55. F15 SREF PROBABILITY P12I x RH x WIND x TMPF ( 30 mph x > 60 F) Ingredients for extreme fire weather conditions over the Great Basin Combined or Joint Probabilities - Not a true probability - An ingredients-based, probabilistic approach - Useful for identifying key areas Combined Probability Charts

56. F15 SREF PROBABILITY TPCP x RH x WIND x TMPF ( 30 mph x > 60 F) Ingredients for extreme fire weather conditions over the Great Basin Combined Probability Charts

57. Elevated Instability – General Thunder Elevated Instability – General Thunder NCEP SREF 30 Sept 2003 09 UTC F12 Mean MUCAPE/CIN (Sfc to 500 mb AGL)Mean LPL (Sfc to 500 mb AGL)

58. Parcel Equilibrium Level Parcel Equilibrium Level NCEP SREF 30 Sept 2003 09 UTC F12 Mean Temp (degC) MUEquilLvl (Sfc to 500 mb AGL) Prob Temp MUEquilLvl < -20 degC (Sfc to 500 mb AGL)

59. Lightning Verification Gridded Lightning Strikes 18-21 UTC 30 Sept 2003 (40 km grid boxes)

60. Dendritic Growth Potential Find SREF members with: –Saturated layers > 50 mb deep –Temp range –11 to –17 degC –Omega <= -3 microbar/s NCEP SREF 7 Oct 2003 21 UTC F15 Probability dendritic conditions (solid/shaded) Mean PMSL (white solid), Mean 1000-500 mb dZ (dashed), Mean 10m Wind

61. Microphysical Example Probability cloud top temps > -8 degC Probability cloud top temps < -12 degC Ice Crystals UnlikelyIce Crystals Likely NCEP SREF 7 Oct 2003 21 UTC F15

62. Banded Precipitation Combined Probabilities Probability MPV 1 NCEP SREF 7 Oct 2003 21 UTC F15

63. Banded Precipitation GOES 10 IR - 8 Oct 2003 1215 UTC

64. Identifying Rare Events Identifying Rare Events (Low end example: Wind/Small Craft Advisory) 9 Oct 2003 09 UTC F63 fcst Prob sfc winds > 30 mph (mean 10m wind vector shown) Difficult to forecast for every grid point Saturday afternoon forecast (11 Oct)

65. Identifying Rare Events Identifying Rare Events (Low end example: Wind/Small Craft Advisory) 9 Oct 2003 09 UTC F63 fcst Now consider an area +/- 90 mi of a point (see Legg and Mylne 2004) 30% chance small craft advy winds over Monterey Bay and offshore waters Saturday afternoon

66. Mode Most Common Precip Type (Snow = Blue); Mean Precip (in); Mean 32 o F Isotherm F015 SREF Valid: 00 UTC 21 December 2004

67. Probability Dendritic Layer > 50 mb F015 SREF Valid: 00 UTC 21 December 2004

68. Probability of Banded Precipitation Potential Probability MPV 1

69. Probability Omega <= -3 microbar/s Median Depth of Dendritic Layer F015 SREF Valid: 00 UTC 21 December 2004

70. Probability Omega <= -3 microbar/s F015 SREF Valid: 00 UTC 21 December 2004

71. Probability 6h Precip >=.25” F015 SREF Valid: 00 UTC 21 December 2004

73. Outline Motivation for ensemble forecasting Ensemble products and applications –Emphasis on probabilistic products Ensemble calibration (verification) Decision making using ensembles

74. Combine Thunderstorm Ingredients into Single Parameter Three first-order ingredients (readily available from NWP models): –Lifting condensation level > -10 o C –Sufficient CAPE in the 0 o to -20 o C layer –Equilibrium level temperature < -20 o C Cloud Physics Thunder Parameter (CPTP) CPTP = (-19 o C – T el )(CAPE -20 – K) K where K = 100 Jkg -1 and CAPE -20 is MUCAPE in the 0 o C to -20 o C layer

75. Example CPTP: One Member 18h Eta Forecast Valid 03 UTC 4 June 2003 Plan view chart showing where grid point soundings support lightning (given a convective updraft)

76. SREF Probability CPTP > 1 15h Forecast Ending: 00 UTC 01 Sept 2004 Uncalibrated probability: Solid/Filled; Mean CPTP = 1 (Thick dashed) 3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004

77. SREF Probability Precip >.01” 15h Forecast Ending: 00 UTC 01 Sept 2004 Uncalibrated probability: Solid/Filled; Mean precip = 0.01” (Thick dashed) 3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004

78. Joint Probability (Assumed Independence) 15h Forecast Ending: 00 UTC 01 Sept 2004 Uncalibrated probability: Solid/Filled P(CPTP > 1) x P(Precip >.01”) 3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004

79. Perfect Forecast No Skill Climatology P(CPTP > 1) x P(P03I >.01”) Uncalibrated Reliability (5 Aug to 5 Nov 2004) Frequency [0%, 5%, …, 100%]

80. Adjusting Probabilities 1)Calibrate based on the observed frequency of occurrence –Very useful, but may not provide information concerning rare or extreme (i.e., low probability) events 2)Use statistical techniques to estimate probabilities in the tails of the distribution (e.g., Hamill and Colucci 1998; Stensrud and Yussouf 2003)

81. Ensemble Calibration 1)Bin separately P(CPTP > 1) and P(P03M > 0.01”) into 11 bins (0-5%; 5-15%; …; 85-95%; 95-100%) 2)Combine the two binned probabilistic forecasts into one of 121 possible combinations (0%,0%); (0%,10%); … (100%,100%) 3)Use NLDN CG data over the previous 60 days to calculate the frequency of occurrence of CG strikes for each of the 121 binned combinations 4)Bin ensemble forecasts as described in steps 1 and 2 and assign the observed CG frequency (step 3) as the calibrated probability of a CG strike 5)Calibration is performed for each forecast cycle (09 and 21 UTC) and each forecast hour; domain is entire U.S. on 40 km grid

82. Before Calibration

83. Joint Probability (Assumed Independence) P(CPTP > 1) x P(Precip >.01”) 3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004 15h Forecast Ending: 00 UTC 01 Sept 2004 Uncorrected probability: Solid/Filled

84. After Calibration

85. Calibrated Ensemble Thunder Probability 15h Forecast Ending: 00 UTC 01 Sept 2004 Calibrated probability: Solid/Filled 3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004

86. Calibrated Ensemble Thunder Probability 15h Forecast Ending: 00 UTC 01 Sept 2004 Calibrated probability: Solid/Filled; NLDN CG Strikes (Yellow +) 3 hr valid period: 21 UTC 31 Aug to 00 UTC 01 Sept 2004

87. Perfect Forecast No Skill Perfect Forecast No Skill Calibrated Reliability (5 Aug to 5 Nov 2004) Calibrated Thunder Probability Climatology Frequency [0%, 5%, …, 100%]

88. Adjusting Probabilities 1)Calibrate based on the observed frequency of occurrence –Very useful, but may not provide information concerning extreme (i.e., low probability) events 2)Use statistical techniques to estimate probabilities in the “tails” of the distribution (e.g., Hamill and Colucci 1998; Stensrud and Yussouf 2003)

89. Consider 2 meter temperature prediction from NCEP SREF –Construct a “rank histogram” of the ensemble members (also called Talagrand diagram) Rank individual members from lowest to highest Find the verifying rank position of “truth” (RUC 2 meter analysis temperature) Record the frequency with which truth falls in that position (for a 15 member ensemble there are 16 rankings) Adjusting Probabilities

90. Adjusting Probabilities Adjusting Probabilities Uncorrected Talagrand Diagram Warm bias in 15h fcst of 12 UTC NCEP SREF Uniform Distribution 2m temperature ending 27 December 2004 Under-dispersive Truth is colder than all SREF members a disproportionate amount of time Clustering by model

91. Use 14-day bias to account for bias in forecast Members 1 through 15 of NCEP SREF

92. Adjusting Probabilities Adjusting Probabilities Bias Adjusted Talagrand Diagram Near neutral bias in 15h fcst of 12 UTC NCEP SREF Large bias eliminated but remains under-dispersive Uniform Distribution 2m temperature ending 27 December 2004

93. Build the pdf by using observed data to fit a statistical distribution (Gamma, Gumbel, or Gaussian) to the tails This produces a calibrated pdf based on past performance –“Past performance does not guarantee future results.” Adjusting Probabilities

94. Adjusting Probabilities Adjusting Probabilities Corrected Talagrand Diagram ~Uniform distribution in 15h fcst of 12 Z SREF Uniform Distribution SREF probabilities now reflect expected occurrence of event even in the “tails” 2m temperature ending 27 December 2004

95. Adjusted Temperature Fcst Max temp (50%) valid 12 UTC 5 Jan to 00 UTC 6 Jan 2004

96. Probabilistic Temperature Forecast Norman, OK (95% Confidence) 50.0% 2.5% Dec 27Dec 28Dec 29 Norman, OK Temp Forecast from SREF Actual mins & maxes indicated by red dots Temp (degF) Local Time  4 AM6 PMMid

97. Probabilistic Meteogram Probability of severe thunderstorm ingredients: OUN; Runtime: 09 UTC 21 April Information on how ingredients are evolving Viewing ingredients via probabilistic thinking

98. Probabilistic Meteogram Probability of severe thunderstorm ingredients: OUN; Runtime: 09 UTC 21 April Information on how ingredients are evolving Viewing ingredients via probabilistic thinking

99. Outline Motivation for ensemble forecasting Ensemble products and applications –Emphasis on probabilistic products Ensemble calibration (verification) Decision making using ensembles

100. Decision Making Probabilities from an uncalibrated, under- dispersive ensemble system are still useful in quantifying uncertainty Trends in probabilities (dprog/dt) may indicate less spread among members as t  0

101. 12h Prob Thunder12h Prob Severe Prob ThunderProb Severe Day 6 Day 5 Day 4 Day 3 Day 2 Increased probabilistic resolution as event approaches Run-to-run consistency Time-lagged members (weighted) add continuity to forecast Trend over 5 days from NCEP MREF (Valid: 22 Dec 2004)

102. Results…

103. Decision Making Probabilities from an un-calibrated, under- dispersive ensemble system are still useful to quantify uncertainty Trends in probabilities (dprog/dt) may indicate less spread among members as t  0 Decision theory can be used with or without separate calibration

104. Decision Theory Example Consider the calibrated thunderstorm forecasts presented earlier [see Mylne (2002) for C/L model]… User: Electronics store Critical Event: Lightning strike/surge Cost to protect: $300 Expense of a Loss: $10,000 YesNo Yes Hit $300 F.A. $300 No Miss $10,000 C.R. $0 Observed Forecast a = F.A. – C.R. Miss + F.A. – Hit – C.R. C/L = a = 0.03 If no forecast information is available, user will always protect if a < o, and never protect if a > o, where o is climatological frequency of the event

105. Decision Theory Example If the calibration were perfect, then user would seek protective action whenever forecasted probability is > a. But, forecast is not perfectly reliable…

106. Apply a cost-loss model to assist in the decision (prior calibration is unnecessary) Define a cost-loss model as in Murphy (1977); Legg and Mylne (2004); Mylne (2002) –This can be done without probabilistic calibration as the technique implicitly calibrates based on past performance V = E climate - E forecast E climate – E perfect Decision Theory Example

107. V = E climate - E forecast E climate - E perfect V = a general assessment of forecast value relative to the perfect forecast (i.e., basically a skill score). V = 1 indicates a perfect forecast system (i.e., action is taken only when necessary) V < 0 indicates a system of equal or lesser value than climatology

108. V = E climate - E forecast E climate - E perfect E climate = min[ (1-o)F.A. + oHit; (1-o)C.R. + oMiss ] E perfect = oHit E forecast = hHit + mMiss + fF.A.+ rC.R. o = climatological freq = h + m YesNo Yes Hit $300 F.A. $300 No Miss $10,000 C.R. $0 YesNo Yes Hit (h) F.A. (f) No Miss (m) C.R. (r) Decision Theory Example Observed Forecast Costs: Performance:

109. Decision Theory Example Never Protect Always Protect 10% Action probability for a =.03 is 7% with V =.64 Potential Value 0.0 0.5 0.001 0.010.10 a = Cost/Loss Ratio (log scale) 1.00 1.0 Maximum Potential Value of the Forecast and its Associated Probability.008.14

110. Summary Ensembles provide information on mean, spread, and forecast uncertainty (probabilities) Derived products viewed in probability space have proven useful at the SPC Combined or joint probabilities very useful When necessary, ensembles can be calibrated to provide reliable estimates of probability and/or aid in decision making

111. SPC SREF Products on WEB http://www.spc.noaa.gov/exper/sref/

112. References Bright, D.R., M. Wandishin, R. Jewell, and S. Weiss, 2005: A physically based parameter for lightning prediction and its calibration in ensemble forecasts. Preprints, Conference on Meteor. Appl. of Lightning Data, AMS, San Diego, CA (CD-ROM 4.3) Cheung, K.K.W., 2001: A review of ensemble forecasting techniques with a focus on tropical cyclone forecasting. Meteor. Appl., 8, 315-332. Ebert, E.E., 2001: Ability of a poor man's ensemble to predict the probability and distribution of precipitation. Mon. Wea. Rev., 129, 2461-2480. Hamill, T.M. and S.J. Colucci, 1998: Evaluation of Eta-RSM ensemble probabilistic precipitation forecasts. Mon. Wea. Rev., 126, 711-724. Legg, T.P. and K.R. Mylne, 2004: Early warnings of severe weather from ensemble forecast information. Wea. Forecasting, 19, 891-906. Mylne, K.R. 2002: Decision-making from probability forecasts based on forecast value. Meteor. Appl., 9, 307-315. Sivillo, J.K. and J.E. Ahlquist, 1997: An ensemble forecasting primer. Wea. Forecasting, 12, 809-818. Stensrud, D.J. and N. Yussouf, 2003: Short-range ensemble predictions of 2-m temperature and dewpoint temperature over New England. Mon. Wea. Rev., 131, 2510-2524. Wilks, D.S., 1995: Statistical Methods in the Atmospheric Sciences. International Geophysics Series, Vol. 59, Academic Press, 467 pp.