1 How Are We Doing? A Verification Briefing for the SAWS III Workshop April 23, 2010 Chuck Kluepfel National Weather Service Headquarters Silver Spring, Maryland x132 Prepared October 2008

2 Part 1 Traditional Statistics

3 The Basics: POD and FAR You can drive up your POD (also called hit rate) by over-forecasting IFR and below conditions. This practice simultaneously drives up the FAR. The CSI provides a mathematical way of correcting an inflated POD by using the FAR. The 2-category Heidke Skill Score has a similar affect, and it passes tests for equitability (statistical balance). Heidke also considers the “not forecast / not observed” situations (in an appropriately balanced manner), which are ignored by the CSI.

4 Prevailing vs. Operational Impact Forecast (OIF) Which should we use? OIF considers TEMPO groups. GPRA system uses OIF. MOS / LAMP – Do not produce TEMPOs When comparing to guidance, I used prevailing.

5 Modified SW US, March 2009 to Feb 2010 Flight Category: IFR and Below

6 Traditional Stats for Modified Southwest United States: Colorado New Mexico Utah Arizona Nevada California Minus these WFOs: San Diego, Los Angeles, San Francisco

7

8 Modified SW US, March 2009 to Feb 2010 Flight Category: IFR and Below

9

10 Modified SW United States TAF Performance vs. Projection March 2009 to February 2010

11 Modified SW US - March 2009 to Feb 2010 Scheduled 3-6 hr IFR and Below GFS LAMP (77K - 36K) K ÷ (77K – 63K) ~ 2.9 The 3-6 hr GFS LAMP false alarmed almost 3 times for every additional hit it got over the forecasters! POD 0.55 FAR 0.50 CSI 0.35 Forecast YesNo ObsObs Yes77 K64 K No77 K 3.4 Million Prevailing POD 0.45 FAR 0.36 CSI 0.36 Forecast YesNo ObsObs Yes63 K77 K No36 K 3.4 Million

12 Modified SW US - March 2009 to Feb 2010 Scheduled 3-6 hr IFR and Below NAM MOS (117K – 36K) ÷ (75K - 63 K) ~ hr NAM MOS false alarmed over 6 times for every additional hit it got over the forecasters! POD 0.54 FAR 0.60 CSI 0.30 Forecast YesNo ObsObs Yes76 K64 K No117 K 3.3 Million Prevailing POD 0.45 FAR 0.36 CSI 0.36 Forecast YesNo ObsObs Yes63 K77 K No36 K 3.4 Million

13 WFOs San Diego, Los Angeles, San Francisco TAF Performance vs. Projection IFR and Below March 2009 to February 2010

14 WFOs El Paso, Tucson, Phoenix (Low Desert Southwest) IFR and Below March 2009 to February 2010

15 Part 2 Lead-Time Software

16

17 Part 3 The Future

18 The Future Output to CSV files (just posted) oStarting with Flight Category and Sig Wx Data oCeiling / Visibility (next) oWinds (last) Plots of POD / FAR / CSI Sort Elements by Sig Wx Type

OBSERVED (A) FORECAST (F) (X) (Y) (W) (Z) POD FOM POFD PON FOHFAR DFR FOCN (Ma,Mf) A(F): Regression of Observations upon the forecast F(A): Regression of Forecast upon observations Ma: Average of Observations (x+y)/N Mf: Average of Forecasts (x+z)/N Improving the Current System: Geometric Interpretation

20 Observed ForecastYesNoTotal Yes606 No066 Total6612 Observed ForecastYesNoTotal Yes066 No606 Total6612 Basic Interpretation: Extreme Cases PERFECT FORECAST RESIGN FROM THE NWS

21 Observed ForecastYesNoTotal Yes336 No336 Total6612 RANDOM CHANCE Basic Interpretation: Random Chance

22 Observed ForecastYesNoTotal Yes500 No Total Observed ForecastYesNoTotal Yes No0100 Total Under-forecast Over-forecast Assesses Bias in one glance!! Basic Interpretation: False Alarms vs. Misses

23 Finis

24 Traditional Stats Entire National Weather Service

25 Nation, March 2009 to Feb 2010 Flight Category: IFR and Below

26 Nation, March 2009 to Feb 2010 Flight Category: IFR and Below

27 Nation, March 2009 to Feb 2010 Flight Category: IFR and Below

28 Nation Performance vs. Projection March 2009 to February 2010