CFI GROUP WORLDWIDE ANN ARBOR ATLANTA BUENOS AIRES KUALA LUMPUR LONDON MADRID MELBOURNE MILAN PARIS PORTO ALEGRE SEOUL SHANGHAI STOCKHOLM National Weather Service Uncertainty Study October 4, 2007

2 © CFI Group 2007 Today’s Discussion Project Goals Results Overview

3 © CFI Group 2007 Project Background Goals 1. Obtain and describe uncertainty guidance requirements from NWS operations, especially guidance related to high impact events (e.g., heavy snow, tornadoes, damaging winds, floods, and hurricanes). 2. Identify specific uncertainty guidance (e.g., ensembles) used to generate products today, and describe issues with these guidance sources. 3.Identify and describe barriers to using uncertainty information in forecast preparation. 4.Identify and describe barriers to communicating uncertainty information in forecast products. 5. Identify current single-value forecast processes, products or services that would benefit from providing forecast uncertainty information either internally or externally.

4 © CFI Group 2007 Project Background Sample comprised of the following: –WFOs (Weather Forecast Offices) The Meteorologist in Charge (MIC) Warning Coordination Meteorologist (WCM) Science Operations Officer (SOO) –RFC (River Forecast Centers ) The Hydrologist in Charge (HIC) Development and Operations Hydrologist (DOH) Data collection was open 8/21/2007 through 9/19/2007 Web invitations were sent, as well as reminder invitations, to 399 individuals and 237 responses were received for a 59% response rate

5 © CFI Group 2007 Results Overview

6 © CFI Group 2007 Employment at NWS Majority of respondents indicate that they are employed in the NWS at a WFO Given the low sample sizes at the RFC and National Center, comparisons among these groups should be done so with caution

7 © CFI Group 2007 Knowledge Level Opportunity exists to get forecasters more knowledgeable about ensemble systems and data and uncertainty information

8 © CFI Group 2007 Ensemble Datasets Ensemble MOS, NCEP GFS Ensemble (MREF), and NCEP Short Range Ensemble Forecast (SREF) are the ensemble datasets most often used in forecast preparation. –NWS should endeavor to provide additional information and/or training around these ensemble datasets

9 © CFI Group 2007 Ensembles The more individual members of an ensemble provided, the more resources necessary Stay as close to 10 as is reasonable and focus extra resources in other areas, such as training

10 © CFI Group 2007 Priority Item Respondents would most like to see the following issue addressed first: data not available in AWIPS/GFE/N-AWIPS –Consistent across the regions with enough sample to rely on (Central, Eastern, Southern, Western)

11 © CFI Group 2007 AWIPS *Multiple responses allowed 2 Answered only if Q8 response is "AWIPS" Central, Eastern, Southern and Western respondents indicate that Full NCEP GFS ensemble and Full NCEP SREF ensemble are the uncertainty guidance datasets that are most needed by forecasters in AWIPS/GFE The NWS should focus on ensuring that these are provided with the most current and accurate information possible

12 © CFI Group 2007 Format of Visualizing Uncertainty In all regions, spaghetti diagram and mean and spread plot were consistently the top two formats NWS should give priority focus for these visual means

13 © CFI Group 2007 Training and Knowledge The majority indicates that their office has not developed local training for uncertainty guidance, which calls for a need from the national level The only rated question asked on the survey was to rate their forecaster’s knowledge on areas as they apply to preparing forecasts –The lowest ratings were received for ensemble system design and perturbations and statistics –The rest also scored low, including decision support, weather risk management, and user requirements –As a way of prioritizing which to address first, the NWS should begin with the lowest scoring

14 © CFI Group 2007 Qualitative Interview Issues Other items identified in the qualitative interviewing not brought to light in the multi, single select, or rated questions include: –As transfer of multiple models is consistently used across WFO’s, the NWS should ensure that the technological bandwidth is robust enough to support product enhancements –Make uncertainty products available on systems that local offices use routinely to produce forecasts (NA Web System) to prevent the need for local offices to do additional work on the production system –Provide a more statistical representation of the solution space (e.g., mean, median and mode) to account for skewed distributions

15 © CFI Group 2007 Verbatim Comments Above all we need to listen to our customers to determine their needs for uncertainty information, and work with them to develop products and services that meet their needs. Too often, we assume to know our customer needs and independently develop products and services that we think are the best solution, only to find out when the product or service is introduced, that it is either no longer needed or falls way short of the mark. Improve the graphical display of data. We generally have a less sophisticated audience, so we have to figure out ways to convey uncertainty without requiring people to have an in-depth understanding of statistics and probabilities. The NWS needs to provide training (face-to-face) on how to use uncertainty information (ensemble forecasting). This has been severely lacking over the years. We need to move faster in implementation. This organization is paralyzed when it comes to making a change (take for example the recent elimination of TWEB forecasts and the ten year process it took to finally get rid of them). We need to evolve and evolve fast or we risk being irrelevant.