BoM / CAWCR. Text Generation in the Next-Gen Forecast System (GFE) J Bally & T Leeuwenburg

Background & Drivers.... Next-Gen Forecast System Better use of NWP models Systematic forecast process Temporal and spatial detail Can verify everything Efficiency gains Many new services: grids, graphics and text all from the same weather database

Nowcast: TIFS (objects) On-the-fly, shallow, slot filling

Text Generation… introduction Most sophisticated meteorological text generation system ??? Large jump from “slot filling” systems (TIFS, TC, Scribe etc) Text as a network of nodes Goal directed multi-pass processing 64,000 lines of python - > 15 p-yr development

Text Generation : example goals Try for <= three weather sub-phrases (2 for wind etc.) Describe the weather trends, rather than a sequence Describe changes in weather only if the impact differs substantially Try for elegant sentence structure; split out unusual weather types if they are not part of the trend Must-goals (guarantees) vs should-goals ……….etc etc

Text Generation… multi-pass processing

Text Generation.... overview Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

Information Representation: Scalars, Vectors, Weather…… PoP Sky Weather Temp / Wind

Information Representation: Hazards Hazards

Text Generation.... Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

Data Gathering.... Grid sampling Use Statistics for scalars and vectors Element th percentile wind speed, 90 th percentile wind speedWind Phrase 25 th and 75 th percentile wind directions centred on average dirWind Phrase 90 th percentile, 10 th percentileSea Height 90 th percentile, 10 th percentileSwell Height 25 th and 75 th percentile swell directions centred on average dir Swell Direction What about weather and hazards? How to summarise a bit of patchy rain, isolated severe thunderstorms and raised dust? Lets concentrate on the weather

Data Gathering.... Grid sampling- eg 3 hr time slices } Isolated Thunderstorms NoWx Sct SH - Wide SH m Patchy RA m Sct TS n Isolated Showers } KeyNumber of Points *Percentage Wide SH m10, 00010% Sct SH -34, 53335% Patchy RA m7, 6448% Sct TS n10, 00010% No Weather45, 00045% Reported coverage = Σ (internal coverage * grid point count) total points

Data Gathering.... Grid sampling NoWx Sct SH - Wide SH m Patchy RA m Sct TS n Reported coverage = Σ (internal coverage * grid point count) total points Reported Intensity = Σ (intensity contribution* grid point count) total affected grid points Similar calculation to collapse similar weather types…Sh/Dz/Ra

Data Gathering.... Grid sampling NoWx Sct SH - Wide SH m Patchy RA m Sct TS n Filtering the Weather List Wx TypesCoverage Threshold SN, SNSH, SL, SLSH2.5% of total area TS, FG, MI5% of total area FR5% of the area below 500m All other types15% of the total area

Text Generation.... Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

Information Processing.... Embedded Local Effect > Winds: Easterly 10 to 20 knots decreasing to 10 to 15 knots around midday then increasing to 15 to 20 knots during the afternoon, locally up to 30 knots in the east. Seas: Below 0.5 metres increasing to 0.5 to 1 metres by early evening, locally up to 1.5 metres in the east. Forecast-Split Local Effect > In the east: Winds: Easterly 10 to 20 knots increasing to 20 to 30 knots during the afternoon. Seas: 0.5 to 1 metres, increasing up to 1.5 metres by early evening. Elsewhere: Easterly 10 to 20 knots decreasing to 10 to 15 knots around midday then increasing to 15 to 20 knots during the afternoon. Seas: Below 0.5 metres increasing to 0.5 to 1 metres by early evening.

Information Processing.... Check for Local Effects …. Scalar Metrics ElementStatScale Value / Embedded Consideration Value At 0.5 Wind DIRAVG135 deg90 deg Wind SPEEDMAX15 kt10 kt Sea HEIGHTMAX1.5m1.0 m Swell DIRAVG135 deg90 deg Swell HEIGHTMAX1.5m1.0 m

Information Processing.... Check for Local Effects NameWind Speed Wind DirSeaSwell Height Swell DirAvg East-West Far West Far East Inshore Offshore

Text Generation… multi-pass processing

Information Processing.... Pre-Process Weather  Arrange statistics in time order;  Combine where appropriate, maintaining ranges;  Separate co-reportable types 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx SH + + DU SH + + TS SHm NoWx 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx ( SH+, SHm) TS DU--- Subphrases after preProcessWx Subphrases before preProcessWx

Information Processing.... Simplify Weather  Collapse Ranges 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWxSHm RA+ SH- Subphrases after preProcessWx Subphrases before preProcessWx 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx(SHm, RAm) (RAm, RA+) SH-

Information Processing.... Merge Weather …. telling little white lies 0-3am3-6am6-9am9-noonnoon-3pm3-6pm6-9pm9-night NoWxSH + TSSH After mergeOverlap: 0-3am3-6am6-9am9-noonnoon-3pm3-6pm6-9pm9-night NoWxSH + TS before mergeGap: 0-3am3-6am6-9am9-noonnoon-3pm3-6pm6-9pm9-night NoWxIsol SH-NoWxSct SH -AreasRAm Subphrases after mergeGap: 0-3am3-6am6-9am9-noonnoon-3pm3-6pm6-9pm9-night NoWxIsol SH-Sct SH -AreasRAm

Recall … multi-pass processing

Information Processing.... Have we tried every processing step enough? Have we achieved our goals for level of detail? Can Adjust Detail by…..  Looking for more local effects?.. Split forecast?  More aggressive sub-phrase combining  Coarser sampling strategy Start again !

Text Generation.... Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

Mapping to words.... Process Trends….  Recognise and Summarise trends 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx SH - SHmRAm 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx SH- developing >..skip.. > increasing to Ram Subphrases after ProcessTrends Subphrases before ProcessTrends

Mapping to words.... Connectors  Increasing / Decreasing  Becoming / Tending  Developing / Clearing  Winds W toNW’y at 15 to 25 knots tending W to SW’ly then increasing to 30 knots.  Isolated showers developing during the morning then increasing to heavy widespread rain…..

Mapping to words.... Time reporting  Transition (change verbs)  Over-time (nouns)  Mixed (trend verbs)  Winds W to NW’y at 15 to 25 knots tending W to SW’ly around noon then increasing to 30 knots.  Morning Fog. Isolated showers developing during the afternoon then increasing to widespread rain…

Text Generation.... Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

Post Processing.... Post-Process Phrases - string replacements to cover limitations - “band-aid”… eg  Early frost. Early fog. >> Early frost and Fog. Remove repeated words eg  W to NW’y winds becoming NW’ly

Example District Forecast... inc local effects

Products all forecasts are in XML...

QC.. with some help from our testing infrastructure...

Change Management.... Importance of specifications Agreed? policies Big change in the role of forecasters Forecaster edits for style and/or substance Change management

The End Text Generation in the GFE