Presentation is loading. Please wait.

Presentation is loading. Please wait.

The 10th annual Northeast Regional Operational Workshop, Albany, NY Verification of SREF Aviation Forecasts at Binghamton, NY Justin Arnott NOAA / NWS.

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "The 10th annual Northeast Regional Operational Workshop, Albany, NY Verification of SREF Aviation Forecasts at Binghamton, NY Justin Arnott NOAA / NWS."— Presentation transcript:

1 The 10th annual Northeast Regional Operational Workshop, Albany, NY Verification of SREF Aviation Forecasts at Binghamton, NY Justin Arnott NOAA / NWS Binghamton, NY

2 The 10th annual Northeast Regional Operational Workshop, Albany, NY Motivation Ensemble information making impacts throughout the forecast process Ensemble information making impacts throughout the forecast process Examples SREFs, MREFs, NAEFS, ECMWF Ensemble Examples SREFs, MREFs, NAEFS, ECMWF Ensemble SREF resolution is reaching the mesoscale (32-45 km), a scale at which some aviation impacts may be resolvable SREF resolution is reaching the mesoscale (32-45 km), a scale at which some aviation impacts may be resolvable Can SREFs provide useful information in the aviation forecast process? Can SREFs provide useful information in the aviation forecast process?

3 The 10th annual Northeast Regional Operational Workshop, Albany, NYSREF 21 member multi-model ensemble 21 member multi-model ensemble 10 ETA(32 km, 60 vertical levels) 10 ETA(32 km, 60 vertical levels) 3 NCEP - NMM WRF (40 km, 52 vertical levels) 3 NCEP - NMM WRF (40 km, 52 vertical levels) 3 NCAR - ARW WRF (45 km, 35 vertical levels) 3 NCAR - ARW WRF (45 km, 35 vertical levels) 5 NCEP - RSM (45 km, 28 vertical levels) 5 NCEP - RSM (45 km, 28 vertical levels) Various IC/BCs, physical parameterizations Various IC/BCs, physical parameterizations

4 The 10th annual Northeast Regional Operational Workshop, Albany, NY SREF Aviation Forecasts Numerous Aviation parameters are created from the 9 and 21Z simulations Numerous Aviation parameters are created from the 9 and 21Z simulations For creating TAFs, For creating TAFs, CIG/VSBY fields may provide the most potential use http://wwwt.emc.ncep.noaa.gov/mmb/SREF/SREF.html Some outputted directly,Some outputted directly, some derived some derived Include: CIG, VSBY, icing,Include: CIG, VSBY, icing, turbulence, jet stream, shear, turbulence, jet stream, shear, convection, precipitation, convection, precipitation, freezing level, fog, etc. freezing level, fog, etc.

5 The 10th annual Northeast Regional Operational Workshop, Albany, NY SREF Aviation Forecasts Verification of SREF CIG/VSBY forecasts has been minimal Verification of SREF CIG/VSBY forecasts has been minimal Alaska Region has completed a study using SREF MEAN values Alaska Region has completed a study using SREF MEAN values No verification study has been conducted over the lower 48 No verification study has been conducted over the lower 48

6 The 10th annual Northeast Regional Operational Workshop, Albany, NY Expectations CIGS/VSBYS, can vary greatly on scales far less than the 32-45 km scale of the SREFs CIGS/VSBYS, can vary greatly on scales far less than the 32-45 km scale of the SREFs Some MVFR/IFR events Some MVFR/IFR events are more localized than others Summer MVFR/IFR tends Summer MVFR/IFR tends to be more localized to be more localized Winter MVFR/IFR is Winter MVFR/IFR is typically more widespread typically more widespread Bottom Line: Expect relatively poor SREF performance during the warm season, with improvements during the cool season Bottom Line: Expect relatively poor SREF performance during the warm season, with improvements during the cool season ~40 km

7 The 10th annual Northeast Regional Operational Workshop, Albany, NY The Study So Far… Gather SREF CIG/VISBY data daily starting July 1, 2008 Gather SREF CIG/VISBY data daily starting July 1, 2008 Data provided specifically for the project by Binbin Zhou at NCEP Data provided specifically for the project by Binbin Zhou at NCEP Compute POD/FAR/CSI/BIAS statistics for July- September at KBGM Compute POD/FAR/CSI/BIAS statistics for July- September at KBGM MVFR and IFR (due to small sample size) MVFR and IFR (due to small sample size) Investigate using different probabilities to base forecast on Investigate using different probabilities to base forecast on 50%, 30%, 20%, 10% 50%, 30%, 20%, 10%

8 The 10th annual Northeast Regional Operational Workshop, Albany, NY The Study So Far…continued Compare SREF results to WFO Binghamton, NY and GFS MOS forecasts Compare SREF results to WFO Binghamton, NY and GFS MOS forecasts Use stats-on-demand web interface to obtain this data Use stats-on-demand web interface to obtain this data

9 The 10th annual Northeast Regional Operational Workshop, Albany, NYResults Very little MVFR/IFR at KBGM in July-September Very little MVFR/IFR at KBGM in July-September IFR or MVFR only ~10% of the time IFR or MVFR only ~10% of the time So, we’re aiming at a very small target! So, we’re aiming at a very small target!

10 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results – MVFR/IFR CIGS SREF Mean SREF50%SREF30%SREF20%SREF10%BGMGFSMOS POD0.610.480.750.860.990.720.60 FAR0.660.460.550.660.740.340.37 CSI0.270.340.390.320.260.530.44 BIAS1.800.891.672.573.841.080.96

11 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results – MVFR/IFR CIGS WFO BGM/GFS MOS more skillful than the SREF mean or any SREF probability threshold WFO BGM/GFS MOS more skillful than the SREF mean or any SREF probability threshold 30% probability threshold shows best skill 30% probability threshold shows best skill Large false alarm ratios with nearly all SREF forecasts Large false alarm ratios with nearly all SREF forecasts Large positive biases for SREF mean and nearly all probability thresholds Large positive biases for SREF mean and nearly all probability thresholds IE over forecasting MVFR/IFR CIGS IE over forecasting MVFR/IFR CIGS

12 The 10th annual Northeast Regional Operational Workshop, Albany, NY Comparing Apples with Oranges? These results compare 9-21 hr SREF forecasts with 0-6 hour WFO BGM forecasts and 6-12 hr GFS forecasts These results compare 9-21 hr SREF forecasts with 0-6 hour WFO BGM forecasts and 6-12 hr GFS forecasts Due to later availability of SREF data (9Z SREFS not available for use until 18Z TAFs) Due to later availability of SREF data (9Z SREFS not available for use until 18Z TAFs) How well does a 9-24 hr GFS MOS (or BGM) forecast perform? How well does a 9-24 hr GFS MOS (or BGM) forecast perform? 21 hr not available using stats-on-demand 21 hr not available using stats-on-demand

13 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results – MVFR/IFR CIGS SREF Mean SREF50%SREF30%SREF20%SREF10%BGM9-24hrGFSMOS9-24hr POD0.610.480.750.860.990.630.64 FAR0.660.460.550.660.740.330.27 CSI0.270.340.390.320.260.480.52 BIAS1.800.891.672.573.840.930.88

14 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results – MVFR/IFR CIGS WFO BGM / GFS MOS performance does not decrease substantially by changing the comparison time window WFO BGM / GFS MOS performance does not decrease substantially by changing the comparison time window

15 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results – MVFR/IFR VSBYS SREF Mean SREF30%SREF20%SREF10%BGMGFSMOSBGM9-24hrGFSMOS9-24hr POD0.040.140.310.580.660.650.520.54 FAR0.000.460.590.640.530.580.600.63 CSI0.040.120.210.280.380.340.290.28 BIAS0.040.250.741.621.421.531.301.46

16 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results MVFR/IFR VSBYS SREF Mean as well as 30 and 20% thresholds fail to identify enough cases to be useful SREF Mean as well as 30 and 20% thresholds fail to identify enough cases to be useful 10% threshold shows greatest skill and is comparable to GFS MOS forecasts! 10% threshold shows greatest skill and is comparable to GFS MOS forecasts! There is a significant positive bias at this threshold There is a significant positive bias at this threshold

17 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results – IFR CIGS SREF Mean SREF30%SREF20%SREF10%BGMGFSMOSBGM9-24hrGFSMOS9-24hr POD0.290.480.580.830.510.530.250.38 FAR0.420.420.620.730.240.480.400.48 CSI0.240.360.300.260.440.360.220.28 BIAS0.500.831.503.030.671.000.420.73

18 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results – IFR CIGS SREF Mean poor at identifying IFR CIGS SREF Mean poor at identifying IFR CIGS CSI scores for SREF probability fields are an improvement on WFO BGM/GFS MOS CSI scores for SREF probability fields are an improvement on WFO BGM/GFS MOS Bias scores indicate underforecasting at a 30% threshold but large overforecasting for 20,10% thresholds Bias scores indicate underforecasting at a 30% threshold but large overforecasting for 20,10% thresholds WFO BGM/ GFS MOS tend to underforecast IFR CIGS WFO BGM/ GFS MOS tend to underforecast IFR CIGS

19 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results – IFR VSBYS SREF Mean SREF30%SREF20%SREF10%BGMGFSMOSBGM9-24hrGFSMOS9-24hr POD0.040.150.260.670.450.290.280.29 FAR00.780.880.870.460.700.620.76 CSI0.040.100.090.120.330.180.190.15 BIAS0.040.672.225.250.850.970.731.22

20 The 10th annual Northeast Regional Operational Workshop, Albany, NY Results – IFR VSBYS SREF can only readily identify IFR VSBY situations except at the 10% threshold SREF can only readily identify IFR VSBY situations except at the 10% threshold Tremendous biases indicate, however that these forecasts are not useful Tremendous biases indicate, however that these forecasts are not useful

21 The 10th annual Northeast Regional Operational Workshop, Albany, NYSummary SREF performance occasionally comparable to GFS MOS  potentially useful guidance SREF performance occasionally comparable to GFS MOS  potentially useful guidance Promising for “~direct” model output Promising for “~direct” model output Hampered by later arrival time at WFO Hampered by later arrival time at WFO MEAN fields show little/no skill MEAN fields show little/no skill Different probability thresholds show best skill for different variables/categories Different probability thresholds show best skill for different variables/categories CIGS: CIGS: SREFS frequently over forecast MVFR/IFR CIGS SREFS frequently over forecast MVFR/IFR CIGS SREFS perform surprisingly well with IFR CIGS SREFS perform surprisingly well with IFR CIGS Best performing probability thresholds are 20-30% balancing BIAS with CSI Best performing probability thresholds are 20-30% balancing BIAS with CSI

22 The 10th annual Northeast Regional Operational Workshop, Albany, NY Summary, continued VSBYS: VSBYS: SREFS have trouble identifying VSBY restrictions SREFS have trouble identifying VSBY restrictions 10% probability threshold necessary to get any signal, but this may be useful for MVFR/IFR (not IFR alone) 10% probability threshold necessary to get any signal, but this may be useful for MVFR/IFR (not IFR alone)

23 The 10th annual Northeast Regional Operational Workshop, Albany, NY Future Plans Continue computing statistics through the upcoming cool season Continue computing statistics through the upcoming cool season Expect improved results given more widespread (i.e. resolvable) restrictions Expect improved results given more widespread (i.e. resolvable) restrictions Expand to other WFO BGM TAF sites Expand to other WFO BGM TAF sites Work with NOAA/NWS/NCEP in improving calculations of CIG/VSBY Work with NOAA/NWS/NCEP in improving calculations of CIG/VSBY

24 The 10th annual Northeast Regional Operational Workshop, Albany, NY Acknowledgements Binbin Zhou – NOAA/NWS/NCEP Binbin Zhou – NOAA/NWS/NCEP For providing access to SREF data in near real-time For providing access to SREF data in near real-time

Download ppt "The 10th annual Northeast Regional Operational Workshop, Albany, NY Verification of SREF Aviation Forecasts at Binghamton, NY Justin Arnott NOAA / NWS."

Similar presentations

Slide 1ECMWF forecast User Meeting -- Reading, June 2006 Verification of weather parameters Anna Ghelli, ECMWF.

Slide 1ECMWF forecast User Meeting -- Reading, June 2006 Verification of weather parameters Anna Ghelli, ECMWF.
Slide 1ECMWF forecast products users meeting – Reading, June 2005 Verification of weather parameters Anna Ghelli, ECMWF.

Slide 1ECMWF forecast products users meeting – Reading, June 2005 Verification of weather parameters Anna Ghelli, ECMWF.
14 May 2001QPF Verification Workshop Verification of Probability Forecasts at Points WMO QPF Verification Workshop Prague, Czech Republic 14-16 May 2001.

14 May 2001QPF Verification Workshop Verification of Probability Forecasts at Points WMO QPF Verification Workshop Prague, Czech Republic May 2001.
A Brief Guide to MDL's SREF Winter Guidance (SWinG) Version 1.0 January 2013.

A Brief Guide to MDL's SREF Winter Guidance (SWinG) Version 1.0 January 2013.
Upcoming Changes in Winter Weather Operations at the Weather Prediction Center (WPC) Great Lakes Operational Meteorological Workshop Dan Petersen, Wallace.

Upcoming Changes in Winter Weather Operations at the Weather Prediction Center (WPC) Great Lakes Operational Meteorological Workshop Dan Petersen, Wallace.
“Where America’s Climate, Weather and Ocean Services Begin” NCEP CONDUIT UPDATE Brent A Gordon NCEP Central Operations January 31, 2006.

“Where America’s Climate, Weather and Ocean Services Begin” NCEP CONDUIT UPDATE Brent A Gordon NCEP Central Operations January 31, 2006.
Matthew Vaughan, Brian Tang, and Lance Bosart Department of Atmospheric and Environmental Sciences University at Albany/SUNY Albany, NY 12222 NROW XV Nano-scale.

Matthew Vaughan, Brian Tang, and Lance Bosart Department of Atmospheric and Environmental Sciences University at Albany/SUNY Albany, NY NROW XV Nano-scale.
Louisville, KY August 4, 2009 Flash Flood Frank Pereira NOAA/NWS/NCEP/Hydrometeorological Prediction Center.

Louisville, KY August 4, 2009 Flash Flood Frank Pereira NOAA/NWS/NCEP/Hydrometeorological Prediction Center.
NWS TAF Verification Brandi Richardson NWS Shreveport, LA.

NWS TAF Verification Brandi Richardson NWS Shreveport, LA.
Lead Time Aviation Verification Onset and Cessation of Ceiling and Visibility Flight Category Conditions (IFR, MVFR, VFR) at FAA Core Airports NWS Aviation.

Lead Time Aviation Verification Onset and Cessation of Ceiling and Visibility Flight Category Conditions (IFR, MVFR, VFR) at FAA Core Airports NWS Aviation.
NOAA/NWS Change to WRF 13 June 2006. What’s Happening? WRF replaces the eta as the NAM –NAM is the North American Mesoscale “timeslot” or “Model Run”

NOAA/NWS Change to WRF 13 June What’s Happening? WRF replaces the eta as the NAM –NAM is the North American Mesoscale “timeslot” or “Model Run”
Improving Excessive Rainfall Forecasts at HPC by using the “Neighborhood - Spatial Density“ Approach to High Res Models Michael Eckert, David Novak, and.

Improving Excessive Rainfall Forecasts at HPC by using the “Neighborhood - Spatial Density“ Approach to High Res Models Michael Eckert, David Novak, and.
Paul Fajman NOAA/NWS/MDL September 7, 2011.  NDFD ugly string  NDFD Forecasts and encoding  Observations  Assumptions  Output, Scores and Display.

Paul Fajman NOAA/NWS/MDL September 7,  NDFD ugly string  NDFD Forecasts and encoding  Observations  Assumptions  Output, Scores and Display.
A tale of two severe weather surprises – The isolated event of 16 July 2010 and the severe weather outbreak of 17 July 2010 Neil A. Stuart NOAA/NWS Albany,

A tale of two severe weather surprises – The isolated event of 16 July 2010 and the severe weather outbreak of 17 July 2010 Neil A. Stuart NOAA/NWS Albany,
Recent performance statistics for AMPS real-time forecasts Kevin W. Manning – National Center for Atmospheric Research NCAR Earth System Laboratory Mesoscale.

Recent performance statistics for AMPS real-time forecasts Kevin W. Manning – National Center for Atmospheric Research NCAR Earth System Laboratory Mesoscale.
Application of Numerical Model Verification and Ensemble Techniques to Improve Operational Weather Forecasting. Northeast Regional Operational Workshop.

Application of Numerical Model Verification and Ensemble Techniques to Improve Operational Weather Forecasting. Northeast Regional Operational Workshop.
Reliability Trends of the Global Forecast System Model Output Statistical Guidance in the Northeastern U.S. A Statistical Analysis with Operational Forecasting.

Reliability Trends of the Global Forecast System Model Output Statistical Guidance in the Northeastern U.S. A Statistical Analysis with Operational Forecasting.
MOS Developed by and Run at the NWS Meteorological Development Lab (MDL) Full range of products available at:

MOS Developed by and Run at the NWS Meteorological Development Lab (MDL) Full range of products available at:
The Collaborative Effort Between Stony Brook University and the National Weather Service Part 3 – Integration of Mesoscale Models in Operational Weather.

The Collaborative Effort Between Stony Brook University and the National Weather Service Part 3 – Integration of Mesoscale Models in Operational Weather.
National Centers for Environmental Prediction (NCEP) Hydrometeorlogical Prediction Center (HPC) Forecast Operations Branch Winter Weather Desk Dan Petersen.

National Centers for Environmental Prediction (NCEP) Hydrometeorlogical Prediction Center (HPC) Forecast Operations Branch Winter Weather Desk Dan Petersen.

Similar presentations

Ads by Google