Dan Shoemaker Aviation Curmudgeon, NWS FWD From a 2005 study done with: Rick Curtis, Chief Meteorologist, SWA Paul Witsaman, Southern Region RAM

Stats on Demand (SOD) verifies to five minute intervals. I can’t forecast to five minute intervals! Airlines can’t land in a five minute window. SOD verification numbers seem low (CSI’s <50%) Is this the best way to measure performance? How might an airline customer measure performance?

Pick a day when widespread weather was a factor. Examine all SWA flights that landed in Southern Region (869 flights, 24 airports). Use an airline perspective – Look at “alternate fuel required” (2000/3) performance. Did each flight’s planning TAF accurately reflect the landing conditions…was the extra fuel needed?

Weather impacts included: SN/PL/FZRA in ABQ AMA BHM BNA DAL LBB LIT MAF OKC TUL FG in AMA CRP HOU IAH LBB LIT +RA/SHRA/TSRA in BHM BNA HOU IAH JAN JAX MSY MVFR/IFR ceiling/visibility at numerous other stations.

For each flight: Examine the TAF valid 2 hours prior to each flight’s take-off. This is assumed to be the TAF the dispatcher used to compute the fuel load. Were conditions below 2000/3 forecast at each flight’s landing time (includes prevailing and TEMPO)?

For each flight: Examine all observations, include specials, within the hour bracketing each flight’s landing time. If conditions below 2000/3 occurred at any time within the landing hour…assume alternate fuel was required (the TAF was “reasonable” since the weather occurred near landing time).

Verify the TAF against the observations for each flight creating the 2 x 2 contingency table. TAF forecasted conditions below 2000/3? (Y/N) Conditions below 2000/3 occurred at some time during landing hour? (Y/N)

Obs Alt Req’d YESNO TAF Alt Req’d YES320 (Hits)90 (False Alarms) NO39 (Misses)420 (Correct Negatives) POD FAR CSI Accuracy.891 Hits/(Hits + Misses).220 FA/(FA + Hits).713 Hits/(Hits + Misses + FA).852 (Hits + Correct Neg)/Total

Obs Alt Req’d YESNO TAF Alt Req’d YES320 (Hits)90 (False Alarms) NO39 (Misses)420 Correct Negatives) POD.891 FAR.220 CSI.713 Accuracy.852 Hits:Extra fuel costs were required. Correct Neg:Extra fuel costs were saved. False Alarms:NWS cost airlines money. Misses:Diversions possible*.

SOD CSI for 2000/3 – unavailable (early 2005). SOD CSI for 1000/3 -- available. I examined 3 comparable winter events from 2009 that produced similar 1000/3 CSI’s (and had 2000/3 CSI available). Averaged 2000/3 CSI’s used as an approximate value for the TAFs used in this study. Yes, this is an “apples to oranges” comparison. But…ball park numbers still provide insight.

2005 study CSI:.713 SOD ballpark CSI:.425

Verifying five minute intervals makes verification scores look low. Real-world skill and value to customers are likely higher. Concentrate on SOD scores vs guidance -- change focus to examine local improvement over models. (do we add value compared to an automated product?) Use SOD results to look for weak areas. Negative forecasting will show up in the bias scores.

Intended for Aviation Program Leaders/ Forecasters Real change has to be made locally-where the TAFs are written.

Biggest obstacle to change—office inertia. You have to change your office culture. …but APL’s have responsibility with no authority. We can’t “make” anybody do anything. What can you do?

Decide to write them. You/MIC are the advocates. Your customers want them. Decide on which airports. Widespread GA in North Texas - we write all our sites. Obtain union cooperation. I took a poll of all 12 forecasters, and got 12 “yes” votes. I did have to do some “lobbying”.

Let your customers/back-up sites know. We issued a PNS; added a notice to Aviation AFDs; added a headline on web page; notified sister offices. Decide on local procedures. Make some AWIPS changes (alarms, etc).

Implement 3 hour TAF amendments. Emphasize PPTAF best practices. PPTAF minimizes bad TAF practices that hurt performance/verification. If all forecasters use PPTAF practices, TAFs will be more consistent between forecasters/offices.

Use the AVNFPS conditional climatology. For operations and research. Encourage “optimistic” forecasts. FWD’s PROB30’s “hurt” the TAF 99% of the time. Coordinate/communicate with your CWSU.

Make sure your TAF writers know how they affect their customers. “ TEMPO SM TSRA BKN035CB ”

Use “force of personality” to get individual forecasters to improve. Group statements ( s to all, memos) don’t work. Provide individual feedback. A number of FWD forecasters will tell you “I didn’t want to get a ‘Shoe’ talk, so I didn’t…” Newton’s first law: Aviation programs at rest remain so. Change the comfort zone.

Provide formal feedback; office and individual. You can’t change what you don’t measure. FWD - annual IFR/MVFR/alt reqd stats. Performance vs SR and LAMP guidance. 2000/3 is more important than total IFR. Individual annual IFR/alt reqd performance vs office average and LAMP. Results are kept anonymous.

0-12 hours Scheduled TAFS all sites FWD Category FWD CSISR CSI % Occurrence* VLIFR LIFR Slice IFR Slice MVFR Slice LIFR & Below IFR & Below MVFR & Below VFR

First, do no harm. When in doubt, leave it out. or -- optimism beats pessimism. Airplanes will NOT fall out of the sky if you write optimistic TAFs. Forecast the probable weather, not the worst possible weather.

The TAF is not a portrait of the atmosphere, it is a stick figure. As long as it is anatomically correct, it’s a good representation.

“ Constant abrasion produces the pearl…it’s a disease of the oyster.” Lenny Bruce

You cannot wait for improvements to come from above — so make them locally. If you make TAF improvement a priority — TAFs will improve. They need to improve. Most local improvements you can make involve effort, but zero cost. You don’t have to do what I do, but DO SOMETHING!

Thanks for your time.