1. BoM / CAWCR. Text Generation in the Next-Gen Forecast System (GFE) J Bally & T Leeuwenburg

2. Background & Drivers.... Next-Gen Forecast System Better use of NWP models Systematic forecast process Temporal and spatial detail Can verify everything Efficiency gains Many new services: grids, graphics and text all from the same weather database

3. Nowcast: TIFS (objects) On-the-fly, shallow, slot filling

4. Text Generation… introduction Most sophisticated meteorological text generation system ??? Large jump from “slot filling” systems (TIFS, TC, Scribe etc) Text as a network of nodes Goal directed multi-pass processing 64,000 lines of python - > 15 p-yr development

5. Text Generation : example goals Try for <= three weather sub-phrases (2 for wind etc.) Describe the weather trends, rather than a sequence Describe changes in weather only if the impact differs substantially Try for elegant sentence structure; split out unusual weather types if they are not part of the trend Must-goals (guarantees) vs should-goals ……….etc etc

6. Text Generation… multi-pass processing

9. Text Generation.... overview Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

10. Information Representation: Scalars, Vectors, Weather…… PoP Sky Weather Temp / Wind

11. Information Representation: Hazards Hazards

12. Text Generation.... Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

13. Data Gathering.... Grid sampling Use Statistics for scalars and vectors Element... 30 th percentile wind speed, 90 th percentile wind speedWind Phrase 25 th and 75 th percentile wind directions centred on average dirWind Phrase 90 th percentile, 10 th percentileSea Height 90 th percentile, 10 th percentileSwell Height 25 th and 75 th percentile swell directions centred on average dir Swell Direction What about weather and hazards? How to summarise a bit of patchy rain, isolated severe thunderstorms and raised dust? Lets concentrate on the weather..........

14. Data Gathering.... Grid sampling- eg 3 hr time slices } Isolated Thunderstorms NoWx Sct SH - Wide SH m Patchy RA m Sct TS n Isolated Showers } KeyNumber of Points *Percentage Wide SH m10, 00010% Sct SH -34, 53335% Patchy RA m7, 6448% Sct TS n10, 00010% No Weather45, 00045% Reported coverage = Σ (internal coverage * grid point count) total points

15. Data Gathering.... Grid sampling NoWx Sct SH - Wide SH m Patchy RA m Sct TS n Reported coverage = Σ (internal coverage * grid point count) total points Reported Intensity = Σ (intensity contribution* grid point count) total affected grid points Similar calculation to collapse similar weather types…Sh/Dz/Ra

16. Data Gathering.... Grid sampling NoWx Sct SH - Wide SH m Patchy RA m Sct TS n Filtering the Weather List Wx TypesCoverage Threshold SN, SNSH, SL, SLSH2.5% of total area TS, FG, MI5% of total area FR5% of the area below 500m All other types15% of the total area

17. Text Generation.... Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

18. Information Processing.... Embedded Local Effect > Winds: Easterly 10 to 20 knots decreasing to 10 to 15 knots around midday then increasing to 15 to 20 knots during the afternoon, locally up to 30 knots in the east. Seas: Below 0.5 metres increasing to 0.5 to 1 metres by early evening, locally up to 1.5 metres in the east. Forecast-Split Local Effect > In the east: Winds: Easterly 10 to 20 knots increasing to 20 to 30 knots during the afternoon. Seas: 0.5 to 1 metres, increasing up to 1.5 metres by early evening. Elsewhere: Easterly 10 to 20 knots decreasing to 10 to 15 knots around midday then increasing to 15 to 20 knots during the afternoon. Seas: Below 0.5 metres increasing to 0.5 to 1 metres by early evening.

19. Information Processing.... Check for Local Effects …. Scalar Metrics ElementStatScale Value / Embedded Consideration Value At 0.5 Wind DIRAVG135 deg90 deg Wind SPEEDMAX15 kt10 kt Sea HEIGHTMAX1.5m1.0 m Swell DIRAVG135 deg90 deg Swell HEIGHTMAX1.5m1.0 m

20. Information Processing.... Check for Local Effects NameWind Speed Wind DirSeaSwell Height Swell DirAvg East-West000220.8 Far West112221.6 Far East000000 Inshore3.000000.6 Offshore4.000000.8

21. Text Generation… multi-pass processing

22. Information Processing.... Pre-Process Weather......  Arrange statistics in time order;  Combine where appropriate, maintaining ranges;  Separate co-reportable types 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx SH + + DU SH + + TS SHm NoWx 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx -----------------------( SH+, SHm)----------------------- ----------TS----------- ---DU--- Subphrases after preProcessWx Subphrases before preProcessWx

23. Information Processing.... Simplify Weather......  Collapse Ranges 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWxSHm RA+ SH- Subphrases after preProcessWx Subphrases before preProcessWx 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx(SHm, RAm) (RAm, RA+) SH-

24. Information Processing.... Merge Weather …. telling little white lies 0-3am3-6am6-9am9-noonnoon-3pm3-6pm6-9pm9-night NoWxSH + TSSH After mergeOverlap: 0-3am3-6am6-9am9-noonnoon-3pm3-6pm6-9pm9-night NoWxSH + TS before mergeGap: 0-3am3-6am6-9am9-noonnoon-3pm3-6pm6-9pm9-night NoWxIsol SH-NoWxSct SH -AreasRAm Subphrases after mergeGap: 0-3am3-6am6-9am9-noonnoon-3pm3-6pm6-9pm9-night NoWxIsol SH-Sct SH -AreasRAm

25. Recall … multi-pass processing

26. Information Processing.... Have we tried every processing step enough? Have we achieved our goals for level of detail? Can Adjust Detail by…..  Looking for more local effects?.. Split forecast?  More aggressive sub-phrase combining  Coarser sampling strategy Start again !

27. Text Generation.... Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

28. Mapping to words.... Process Trends….  Recognise and Summarise trends 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx SH - SHmRAm 0-3am3-6am6-9am9-noonnoon- 3pm 3-6pm6-9pm9-night NoWx SH- developing >..skip.. > increasing to Ram Subphrases after ProcessTrends Subphrases before ProcessTrends

29. Mapping to words.... Connectors  Increasing / Decreasing  Becoming / Tending  Developing / Clearing  Winds W toNW’y at 15 to 25 knots tending W to SW’ly then increasing to 30 knots.  Isolated showers developing during the morning then increasing to heavy widespread rain…..

30. Mapping to words.... Time reporting  Transition (change verbs)  Over-time (nouns)  Mixed (trend verbs)  Winds W to NW’y at 15 to 25 knots tending W to SW’ly around noon then increasing to 30 knots.  Morning Fog. Isolated showers developing during the afternoon then increasing to widespread rain…

31. Text Generation.... Information representation Data Gathering Information Processing and Document Planning Mapping to Words ( Surface Realisation ) Post Processing

32. Post Processing.... Post-Process Phrases - string replacements to cover limitations - “band-aid”… eg  Early frost. Early fog. >> Early frost and Fog. Remove repeated words eg  W to NW’y winds becoming NW’ly

33. Example District Forecast... inc local effects

34. Products all forecasts are in XML...

35. QC.. with some help from our testing infrastructure...

36. Change Management.... Importance of specifications Agreed? policies Big change in the role of forecasters Forecaster edits for style and/or substance Change management

37. The End Text Generation in the GFE