1 What’s New in Verification? A Verification Briefing for the SAWS IV Workshop October 26, 2011 Chuck Kluepfel National Weather Service Headquarters Silver Spring, Maryland x132 Starter Videos

2 Part 1 Lead Time Measurements

3 History NOAA (GSD) was contracted to measure the lead time with which NWS TAFs predict the onset and cessation of IFR conditions (2-category, IFR vs. VFR). This software package has been available thru the NWS Performance Management Web site for 2 years. Next 6 months: Joint NWS/FAA TRWG (Traffic Flow Management Weather Requirements Working Group) is directing the NWS Performance Branch to measure TAF performance, using more specific Decision Support requirements.

4 GSD Software Allows for tight and loose accuracy standards. Verifies IFR onsets similar to the way NWS verifies warnings. LT = IFR onset time - TAF issuance time. Uses the TAF with the longest LT.

5 Example of an IFR Onset at 1800

6

7

8

9 TRWG Decision Support Software Requirements have just been defined. Code writing should begin soon. Very specific requirements set by TRWG. Measure NWS skill in forecasting the Onset and Cessation of events critical to operations.

10 TRWG Decision Support Software Measure the Onset and Cessation of IFR (Phase 1) VFR (Phase 1) MVFR (Phase 1)

11 TRWG Decision Support Software Measure the Onset and Cessation of IFR (Phase 1) VFR (Phase 1) MVFR (Phase 1) Wind shifts (Phase 2) Wind speed changes (Phase 2)

12 TRWG Decision Support Software Measure the Onset and Cessation of IFR (Phase 1) VFR (Phase 1) MVFR (Phase 1) Wind shifts (Phase 2) Wind speed changes (Phase 2) Thunderstorms (Phase 3) Freezing precipitation (Phase 3)

13 TRWG Decision Support Software The Onset of these conditions: IFR (Onset - always deterioration) VFR (Onset - always improvement or clearing)

14 TRWG Decision Support Software Measure the Onset and Cessation of the Following Events IFR VFR MVFR (from IFR – Onset represents clearing) MVFR (from VFR – Onset represents deterioration)

15 TRWG Decision Support Software

16 TRWG Decision Support Software

17 TRWG Decision Support Software

18 TRWG Decision Support Software

19 TRWG Decision Support Software

20 TRWG Decision Support Software Methodology For each hit, determine whether the forecast timing error met the TRWG requirements for scoring a POD hit: For 1-hour LT: ± 15 minutes For 2-hour LT: ± 15 minutes For 4-hour LT: ± 30 minutes For 6-hour LT: ± 45 minutes For 8-hour LT: ± 60 minutes

21 TRWG Decision Support Software Methodology

22 TRWG Decision Support Software Methodology

23 TRWG Decision Support Software Methodology

24 TRWG Decision Support Case Study 7 Days (5.5 active) at one terminal

25 TRWG Decision Support

26 TRWG Decision Support

27 Part 2 CWSU Verification

28 CWSU Verification

29 Part 3 Traditional Stats on Demand Results

30 Traditional Stats for Modified Southwest United States: Colorado New Mexico Utah Arizona Nevada California Minus these WFOs: San Diego, Los Angeles, San Francisco

31

32 Modified SW US – Fiscal Year 2011 Scheduled 3-6 hr IFR and Below GFS LAMP (569K - 264K) K ÷ (511K – 482K) ~ 10.5 The 3-6 hr GFS LAMP false alarmed over 10 times for every additional hit it got over the forecasters! POD 0.56 FAR 0.53 CSI 0.35 Forecast YesNo ObsObs Yes511 K402 K No569 K 13.1 Million Prevailing POD 0.53 FAR 0.35 CSI 0.41 Forecast YesNo ObsObs Yes482 K430 K No264 K 13.4 Million

33 WFOs MTR, LOX, SGX FY 2011 Scheduled 3-6 hr IFR and Below GFS LAMP POD 0.54 FAR 0.43 CSI 0.39 Forecast YesNo ObsObs Yes213 K182 K No158 K 2.7 Million Prevailing POD 0.64 FAR 0.31 CSI 0.50 Forecast YesNo ObsObs Yes251 K144 K No113 K 2.7 Million

34 WFOs PSR, TWC, EPZ FY 2011 Scheduled 3-6 hr IFR and Below GFS LAMP POD 0.18 FAR 0.73 CSI 0.12 Forecast YesNo ObsObs Yes No Million Prevailing POD 0.18 FAR 0.48 CSI 0.15 Forecast YesNo ObsObs Yes No Million