WAFS Workshop on the Use and Visualization of Gridded SIGWX Forecasts September 2009

Overview  Introduction  Compatibility of forecasts of the two WAFCs  Quality issues: seasonal aspects  Quality issues: CB diurnal variation  Quality issues: CB associated with TC  Verification issues  Calibration issues  User survey  Conclusions

Introduction  WAFSOPSG/4: WAFC Provider States, in coordination with WAFSOPSG members from IATA, IFALPA and WMO, to undertake systematic comparisons of trial gridded forecasts of icing, turbulence and CB clouds of the two models, highlighting characteristics of areas with different values.  The results should lead to the alignment of algorithms used by the two WAFCs.

Introduction  To facilitate comparison of the trial gridded forecasts, HK, China set up website ( grid-fc.weather.gov.hk) to display gridded values (without smoothing/interpolation). grid-fc.weather.gov.hk  The observations were reported in CNS/MET SG/13 meeting in July 2009 and were agreed to be referred to the WAFSOPSG/5 meeting. (Earlier observations reported in CNS/MET SG/12 meeting in 2008)CNS/MET SG/12

Compatibility of forecasts of the two WAFCs T+24 h Max CAT Potential at FL340 based on 12 UTC 2 July 2009 ) UKUSDiff  Difference map shows absolute difference in the two forecasts (little agreement between the two forecasts if difference map appears to be the union of them).

Compatibility of forecasts of the two WAFCs  CB horizontal extent CB horizontal extent - CB coverage of UK more extensive - Forecast values of US reaching 50-75% (OCNL following Annex 3 spec) over tropics, larger than those given in the UK forecast  ICAO height at CB top ICAO height at CB top - UK still gives more extensive CB areas - UK over-forecast CB for vast areas in the Pacific Ocean  Mean in-cloud turbulence potential (FL180) Mean in-cloud turbulence potential - UK and US forecast values with similar range - UK forecast shows small values almost everywhere

Compatibility of forecasts of the two WAFCs  Max icing potential (FL180) Max icing potential - Significant differences in the spatial coverage over the tropical and higher latitude regions - WAFS Science Coordination Meeting 2009: “percentage overlap of all (icing) forecasts between UK and US ~ 20%”  Max CAT potential (FL340) Max CAT potential - UK gives much less extensive CAT areas (except over certain spots)

Compatibility of forecasts of the two WAFCs  Difference maps in most cases appear to be the union of the forecasts from the two centres, indicating little agreement

Latest CB forecasts of UK CB top height valid 12 UTC 14 Sep 2009 CB horiz extent valid 12 UTC 14 Sep 2009 MTSAT-1R infrared satellite imagery, with deep convection indicated in red, at 08 UTC on 14 Sep 2009

Quality issues: seasonal aspects mean in-cloud turbulence at FL 180 For both winter & summer: large values are forecast by UK for high latitudes in both hemispheres. They are not seen in the US forecasts. T+24 h Gridded Forecasts (based on 00 UTC 6 Jan 2009) T+24 h Gridded Forecasts (based on 00 UTC 2 Jul 2009) UK US

Quality issues: seasonal aspects max icing potential at FL 180 At high latitudes in winter: large values (> 85%) forecast by UK, but much smaller values (mostly < 30%) forecast by US Scientific paper Scientific paper on icing climatology: non-occurrence of icing at FL180 in the wintertime north of 25  N UKUS T+24 h Gridded Forecasts (based on 00 UTC 2 Jul 2009) T+24 h Gridded Forecasts (based on 00 UTC 6 Jan 2009)

Quality issues: seasonal aspects max CAT potential at FL340 – 3 days in Jan large values are forecast by UK for a number of isolated locations - stationary “hot spots” associated with mountainous regions

Quality issues: seasonal aspects  Some common issues can be identified when data for different seasons are monitored (e.g. in-cloud turb, icing)  Some issues (e.g. terrain-related) may show up at different locations in different seasons  The above observations are not exhaustive  It is imperative for the WAFCs to: conduct verification for all seasons examine model climatological biases

Quality issues: seasonal aspects Climatology of differences between two WAFCs (May – Aug 2009) Max icingMax CAT Mean in-cloud turb CB horiz extent CB top height ? ?

Quality issues: CB diurnal variation ICAO height at CB top (UK) T+24h gridded forecasts Based on 00 UTC 17 August 2008 T+24h gridded forecasts Based on 06 UTC 17 August 2008 Solar heating

Again the UK forecasts are giving much more extensive CB areas compared to the US forecasts T+24h gridded forecasts Based on 00 UTC 17 August 2008 T+24h gridded forecasts Based on 06 UTC 17 August 2008 Quality issues: CB d iurnal variation ICAO height at CB top (US) Solar heating

MTSAT-1R infrared satellite imagery, with deep convection indicated in red, at 23:30 UTC on 17 August 2008 MTSAT-1R infrared satellite imagery, with deep convection indicated in red, at 05:30 UTC on 18 August 2008 Actual weather as shown by the satellite images Weather was generally fine over southern China throughout the period Both UK and US over- forecast CB occurrence over this region at 06 UTC (14 H local time), probably due to difficulties of the models in handling CB development induced by daytime solar heating Quality issues: CB d iurnal variation

 2009 WAFS Science Coordination Meeting acknowledged that further work would need to be undertaken by the WAFCs for improving the CB gridded forecasts, including to “look at diurnal variations of forecasts (manual and automatic), particularly in tropics”

Quality issues: CB associated with TC Satellite imagery clearly shows deep convection associated with tropical cyclone Linfa Rather difficult to identify the area associated with TC without referring to the satellite imagery Based on 06 UTC 19 June 2009 UKUS Need TC (and other SIGWX) information in future visualization models

Verification issues  APANPIRG called for systematic SIGWX verification in 2006  WAFSOPSG/4: Evaluation of the SIGWX forecasts proposed by APANPIRG would be sufficiently addressed in the systematic comparison of WAFS SIGWX forecasts with the gridded forecasts

2009 Science Coordination Meeting Forecast verifiedPeriod Parameters verified Data used as “ground truth” for verification Verification metrics considered Criteria for counting hit and false alarm Icing Nov 2008 – Jan 2009 Maximum & mean icing potential at FL100 & FL140 CloudSat Icing Potential (CLIP) product (derived from CloudSat cloud classification data and numerical model temperature forecast) POD + & FAR ^ Icing potential thresholds – 0.1, 0.3 & 0.5 CLIP – trace icing CAT Nov 2008 – Jan 2009 CAT Global Aircraft Data Set (GADS) based on British Airways fleet of Boeing aircraft POD & FAR CAT potential thresholds – numerous GADS – DEVG # >= 2 m/s In-cloud Turbulence Not verified CB Nov 2008 & Jan 2009 UK - convective rainfall rate* US - ?? UK Sferics dataPOD & FAR Rainfall rate thresholds – 16 levels 0 to kg/m 2 /s Sferics data - ?? + POD = probability of detection, or hit rate = hit / (hit + miss). ^ FAR = false alarm rate = false alarm /(false alarm + correct rejection). For rare events, correct rejection is very No indication whether max or mean turb potential was considered, and whether or not forecasts for all flight levels were verified. # DEVG = Derived equivalent vertical gust * Not a product of the CB gridded forecasts

Verification issues  In-cloud turbulence – systematic verification yet to be done.  Icing – verification based on another icing product (CLIP) as ground truth, with “trace icing” criterion Kay et al (2009) on CLIP: “It is important to note that our algorithm only states where icing conditions have the potential to exist rather than stating exactly where within the clouds exactly the icing does or does not exist.... Therefore our algorithm should be considered as a biased representation of icing conditions in the atmosphere. In the absence of a true measure of icing conditions the amount of bias cannot be known”. The choice of the ground truth for the verification, as well as the verification method itself, must be agreed by WAFSOPSG. “Trace icing”  calibration issue.

Verification issues  CAT – verification based on DEVG >= 2 m/s from aircraft data, with numerous thresholds, rather than ICAO adopted turbulence metrics, i.e. MOD/SEV in PIREP or EDR thresholds. Numerous thresholds  calibration issue.  CB – verification based on sferics data, with numerous thresholds, rather than observations of CB (e.g. from satellite). Also lack of information on how CB forecast parameters are derived, and use of smoothing algorithm by UK.

Calibration issues  Need to be mindful about users’ perception and expectation based on their experience with the existing SIGWX forecasts  Thresholds for verifying and visualizing gridded forecasts need to be determined in consultation with users

User survey  HKO conducted a user survey on the WAFS trial gridded forecasts in Jul to Aug 2009, 15 returns from IFALPA, pilots, airline management and dispatchers:15 returns 80% consider gridded products are NOT suitable for operational use in flight planning and flight documentation; 93% considered gridded products CANNOT replace the SWH and SWM charts.  Specific user comments are summarized below.

User comments CommentsUser  Why is there so much difference from the London and Washington charts if they are from the same time period? IFALPA Rep  Have lots of deficiencies: too general; discontinuities; disagreement between centres; loss of useful information; too long forecast time periods etc.  “Potential” appears to have no scientific definition or validity and of little or no assistance to pilot, whereas PROB gives the indication of the likelihood of the turbulence/icing/CAT being encountered.  They must reflect the situation and not be so far forecast that the time period does not apply or that the forecast is so general as to be meaningless. IFALPA Pilot

User comments CommentsUser  Not suitable for operational use without considerable improvement. The improvement suggested by the HKO would help a lot, particularly if the horizontal "real time" strips can be developed. These would be very useful and very practical for the pilots.  The gridded products could not replace the high- level/medium level at this stage as the information is too general and some of the previous information to be omitted. IFALPA Pilot (cont’d)

User comments CommentsUser  The scale is too large and the areas of hazard are too wide. If pilot was presented with so much weather, he/she would be unable to fly around it.  Different aircraft types have different rides through turbulence depending on wing loading.  Icing is only a problem at cruise levels in large tropical convective clouds. Pilots  The technological advance and gridded forecasts could not take the place of the forecaster’s expertise of the MET situation, particularly for SIGWX like CB. Member of WAFS OPSG

User comments Comments User  Large difference between WAFC London and Washington.  Two WAFCs should align to produce an integrated WAFS.  Very large scale, too big an area, insufficient details for short-haul flights.  It is a rough forecast; user cannot obtain detailed information just by a glance; not precise and detailed in comparison with JMA data.  We do prefer using High-level SIGWX and Medium-level SIGWX charts (the current one). It's simple, easy to understand and contains lots of information. Airlines

Conclusions  Suitability of WAFS gridded forecasts of icing, turbulence and CB for operational use : Issues with compatibility, quality, verification, and calibration need to be satisfactorily addressed before they could be considered suitable for operational use. Systematic verification: WAFS output performance indicators (including methodology, thresholds, choice of ground truth) for verifying the gridded forecasts should be systematically developed and agreed. Calibration: the thresholds for verifying and visualizing the gridded forecast should be calibrated for compatibility with users’ expectation/perception. Pilots have problems with the use of “potential” for icing and turbulence.

Conclusions  Suitability of WAFS gridded forecasts of icing, turbulence and CB to replace the current SIGWX forecasts (SWH and SWM): Not at this stage (not for operational use) Visualization issues (including thresholds to be adopted, forecast irregularities e.g. CB top/base heights) need to be resolved High “at-a-glance” visualization models should be agreed by users SIGWX features other than icing, turbulence and CB required by users should be included, especially if they are safety-related (e.g. TC, VA)

Thank you

Supplementary information

Observations reported in CNS/MET SG/12, 2008 CB horizontal extent CB coverage is more extensive with generally larger forecast values in the UK forecast compared with that of US. Furthermore, small values of CB horizontal extent are given almost everywhere in the UK forecast valid 00 UTC 23 Jun (based on 00 UTC 22 June 2008) UKUS Back

Observations reported in CNS/MET SG/12, 2008 ICAO height at CB top UK forecast generally gives more extensive but lower heights of CB top (generally below 37kft) compared with that of US (above 43 kft) valid 00 UTC 23 Jun (based on 00 UTC 22 June 2008) UKUS Back

Observations reported in CNS/MET SG/12, 2008 Mean in-cloud turbulence potential UK forecast generally gives wider range and higher turbulence potential values compared with that of US. On the other hand, small values of mean in-cloud turbulence potential are given almost everywhere in the US forecast valid 00 UTC 23 Jun at FL180 (based on 00 UTC 22 June 2008) UK US Back

Observations reported in CNS/MET SG/12, 2008 Maximum icing potential At FL180: UK forecast giving generally higher icing potential over the higher latitude regions; US forecast giving generally higher icing potential over the tropical region and the UK forecast giving generally higher icing potential over the higher latitude regions valid 00 UTC 23 Jun at FL180 (based on 00 UTC 22 June 2008) UK US Back

Observations reported in CNS/MET SG/12, 2008 Maximum CAT potential At FL340: UK forecast generally gives much less extensive CAT areas compared with that of US valid 00 UTC 23 Jun at FL340 (based on 00 UTC 22 June 2008) UKUS Back

Bernstein, B.C. and Christine LeBot, 2009: An Inferred Climatology of Icing Conditions Aloft, Including Supercooled Large Drops Part II: Europe, Asia and the Globe, Journal of Applied Meteorology and Climatology, American Meteorological Society Back

CB horizontal extent at FL340 T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009) UKUSDiff Back

ICAO height at CB top T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009) UKUSDiff Back

ICAO height at CB top T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009) Back UKUS MTSAT-1R IR sat pic With deep convection indicated in red At 11:30 UTC 3 July 2009

Back

Mean in-cloud turbulence potential at FL 180 T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009) UKUSDiff Back

Maximum icing potential at FL 180 T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009) UKUSDiff Back

Maximum CAT potential at FL 340 T+24 h Gridded Forecasts (based on 12 UTC 2 July 2009) UKUSDiff Back

List of Respondents to WAFS Survey IFALPA Rep (Captain Miguel Marin, Chairman of ATS Committee) IFALPA Pilot (Captain Brian Greeves, HKALPA) Pilots: The Guild of Air Pilots and Air Navigators (GAPAN) (Captain Brent Hawkins, Vice-Chairman, Hong Kong Region) Metrojet (Captain Peter Garraty) Airlines: Cathay Pacific Airways (Neil Phillips, Manager Line Operations) China Airlines (Lo Wai Lun Kevin, Dispatcher) Dragonair (Andy Jepps, FTU A320; Jim Ashby, Deputy Manager Line Operations) El Al Israel Airlines Limited (c/o Hammond Law, Team Manager) Fedex (Andy Yeung, Manager - Operations) Hong Kong Airport Services Limited (Gary Hor, Flight Dispatch Manager; Jackson Lam, Quality Assurance Manager) Japan Airlines (Stephen Chan, Manager of Flight Operation) Jardine Airport Services Limited (Hammond Law, Team Manager) South African Airways (Ringo Chui, Airport Manager & Vice-chairman of Airline Operators Committee) Back