Verification of WAFS Global Icing Products Jennifer Mahoney 1 and Sean Madine 1,2 1 NOAA/Earth System Research Laboratory 2 Cooperative Institute for Research in the Atmosphere This work was completed on behalf of the U.S. Federal Aviation Administration Presented at: WAFS Workshop on the Use and Visualisation of Gridded SIGWX Forecasts Paris, September

Acknowledgements This work was completed on behalf of the U.S Federal Aviation Administration Project Team Project Lead: Sean Madine Project Analyst: Chungu Lu Programming Support: Mike Kay Program Chief: Jennifer Mahoney

Outline of Discussion Motivation, Background, and Challenges Approach and Assessment Strategy – Creation and validation of CloudSat Icing Product (CLIP) Forecast Analyses – Comparison of WAFC U.K. and WAFC U.S. Global Icing – Comparison of WAFC Global Icing to Significant Weather (SigWx) Product Summary

Motivation Icing forecasts are used in strategic planning – One example: Planning for amount of fuel carried on board aircraft to manage possible icing encounters. The WAFC Washington (U.S.) and the WAFC London (U.K.) have developed Global Icing Products, updated frequently, that may improve icing information that is provided to the aviation community.

Background Project Goal: Evaluate the quality of the WAFC Global Icing Forecasts; with resource and time constraints: – Altitudes that typically contain icing (i.e., mid-levels) – Scoped evaluation to Nov 2008 – Jan 2009; winter in northern hemisphere – Domains: Global, North Pacific, North Atlantic; with a North Atlantic focus Evaluation criteria : 1) accuracy of the products should be at least as good as the human generated Significant Weather (SigWx) forecasts produced today, and 2) WAFC U.S. and U.K icing products need to be compatible

Challenges Obtaining a representative observation dataset – Voice pilot reports (PIREPS) for icing are sparse in time and space – PIREPs poorly sample at flight levels relevant to operational decisions (enroute flight levels are typically above the occurrence of icing) – PIREPs disproportionately sample icing conditions Exploring new techniques for creating a proxy observation of icing in the atmosphere

Assessment Approach Compare the quality of the WAFC U.K icing product to the WAFC U.S. icing products – Measure the quality – Qualitative examination of compatibility (i.e., incompatible means ambiguity of the forecast in the planning process) Compare the quality of the WAFC Global Icing Products to the Significant Weather Product – Measure the quality

Assessment Strategy Build an icing verification proxy based on satellite products and use for evaluating the WAFC icing forecasts over global data-sparse domains Development of CloudSat Icing Product (CLIP) – Tuned and validated CLIP with respect to the Current Icing Potential (CIP), current U.S. operational icing analysis – Study period: November 2007 through January 2008 (Note: Same season, 1 year earlier) Use CLIP to assess the quality of the WAFC global icing products and SigWx products over global domains – Study period: November 2008 through January 2009

CloudSat Icing Product (CLIP) Strengths CloudSat cloud profiling radar very effective at diagnosing existence of clouds, particularly at levels relevant to our study – CLIP highly accurate at detecting no cloud and therefore ‘no icing’ Information for CLIP allows assessment of forecast risk (miss) and forecast efficiency (false alarms) – PIREPs don’t allow the false alarm measure because of the imbalance of yes/no reporting CLIP’S radar input provides – An objective measure of cloud, at high resolution – Continuous reporting along orbiting track – An improvement upon the subjective nature and sparseness of using PIREPs

Comparison of the WAFC Global Icing Products

Analysis of WAFC Global Icing Products Many stratifications examined – Geographic domain Global, N. Atlantic, N. Pacific – Flight level All, FL100, FL140 – Cloud type (as diagnosed by CloudSat) All, Convective, Non-convective – Algorithm threshold 0.1, 0.3, and 0.5 – Lead time All leads 6, 12, …, 36-h individually

Analysis of WAFC Global Icing Products Main thread of investigation – North Atlantic domain Domain of interest during assessment winter time period and included significant amount of air traffic density – Altitudes 10, ,000 ft Operationally significant altitudes and typically contain ice – Non-convective clouds Products definition specifically includes non-convective icing – Max icing attribute Most compatible WAFC forecasts

Domains Global, North Atlantic, North Pacific; 3 CloudSat orbits, each orbit ~1.5h,

WAFS Maximum Icing 12 January 2009 UK/FL100 UK/FL140US/FL140 US/FL100

Vertical Cross Sections Along CloudSat Path Atlantic Domain WAFS-UK WAFS-US Cloud Classification CLIP 12 January 2009

WAFS Maximum Icing 23 December 2008 UK/FL100 US/FL100 UK/FL140US/FL140

Vertical Cross Sections Along CloudSat Path Atlantic Domain 23 December 2008

Comparison of Performance: Flight Level (Statistics for 12- and 18-hr leads, Atlantic domain, Non-convective clouds, Max icing attribute) ROC Thresholds: 0.1, 0.3, and 0.5 Product False Alarm Rate (1-PODn) POD

Atlantic domain, 0.1 Threshold, FL100/140, 12- and 18-hr leads, Max icing intensity, Non-convective clouds CLIP Yes Icing No Icing No Cloud No Icing Warm Cloud No Icing Cold Cloud UK (fcst) Yes No US (fcst) Comparison WAFC Icing Forecasts in Operational Context Yes Icing No Icing No Cloud No Icing Warm Cloud No Icing Cold Cloud Yes No Blue – Agreement Yellow – False alarms Red - Misses

Comparison Summary The WAFC U.K and WAFC U.S. forecasts provide roughly the same forecast efficiency, but the WAFC U.K. provides lower operational risk FARPODBias WAFC U.K64%65%1.8 WAFC U.s66%47%1.4

Comparison of the WAFC Global Icing Products to the SigWx Product

WAFS Mid Level SIGWX Product Thunderstorms/CB Tropical Cyclones Mod/Sev Turb (CAT or IC) Mod/Sev Icing Tropopause Heights Jetstreams (80kt and above depicted) Jet Depth Volcanic Eruptions Widespread Sand/Dust storms Release of Radioactive Materials NOTE: Because of choice to use N. Atlantic domain, only U.S SigWx Product was evaluated

Analysis - Main Thread Atlantic domain Altitudes 10, ,000 ft Non-convective clouds Max icing attribute

Comparison of Performance: SigWx Chart (FL100/140, SigWx issuances/lead, Atlantic domain) ROC Thresholds: 0.1, 0.3, and 0.5 False Alarm Rate (1-PODn) POD Product NOTE: Because of choice to use N. Atlantic domain, only U.S SigWx product was evaluated

Comparison Summary The WAFC U.K and WAFC U.S. forecasts provide roughly the same forecast efficiency, but the WAFC U.K. provides lower operational risk FARPODBias WAFC U.K64%65%1.8 WAFC U.S.66%47%1.4 SigWx Product 69%31%1.2

Summary The U.K. icing product slightly outperforms the U.S. icing product in an operational; For similar FAR values, the U.K. has higher POD Qualitatively, the WAFC U.K. and U.S products are incompatible in volume extent and intensity of potential Both U.K. and U.S. products perform better than the SigWx Product in the operational context; For similar FAR, both WAFC products have higher POD values Based on this study and understanding of the algorithms, the automated WAFC product could be significantly improved with respect to volume coverage and calibration of potential

Appendix False alarm ratio = yn/yy+yn False alarm rate = 1-PODn = yn/nn+yn PODn = nn/nn+yn POD = yy/yy+ny 2x2 Contingency Table Observation ForecastYN YYYYN NNYNN