1. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Forecast text editing rates and analysis Jaime Clark and Bianca Gibson Brianna Laugher Tennessee Leeuwenburg December 2012 www.cawcr.gov.au

2. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Graphical Forecast Editor

3. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Text formatters

4. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Text formatters - editing

5. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GuidanceGrid Editing & Smart Tools Text Formatting Forecast Production GFE Forecasting

6. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Text Formatting Forecast Production GFE Forecasting Grids edited text original gfemetrics

7. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Metrics

8. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Metrics

9. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Metrics

10. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Metrics http://tardis.bom.gov.au/gfemetrics Web front-end to “FDB” database created/maintained by ITB Initially developed by Brianna, further developed by Anthony Briggs STATS/SUMMARY Can view graphs/tables of total edits: by time, by state, by formatter VIEW Can view any edits made to any GFE forecasts Can query edits by state, date, formatter, period, issuance, locale Can search for edited/unedited/either containing particular words TOOLS Can view graphs/tables of most used tools: by time, by state

11. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Metric? We initially hoped that improvements to the formatters would be reflected in lower edit rates... this appears not to be the case... but it's hard to tell...

12. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Dimensions of edits Is the generated text consistent with grid contents? “Wrong” results (genuine error) vs stylistic changes (subjective) “Possible shower” -> “Mostly dry” Is the edited text consistent with Services policy? Policy may need resolving with Services “Fine”, “moderating”, “freshening” Is the info required to produce the edited text present in the GFE? Formatter missing rules - “Humid”, “Hot”, sea breeze GFE missing grids – climatology, uncertainty! Other info in forecaster brain (local expertise) Can potentially fix Won't fix Can't fix (for everyone) Should fix ASAP

13. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Limitations  Formatters are mainly just a group of phrases – Sky, Wx, Wind, Temp, Seas, Swell. Effort is directed at improving phrases – but there's no way to check edit rates of phrases  Hard to get a sense of any general trend when a query returns hundreds of results  Little/no time to devote to exploring regional differences

14. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Monash Talented Students program Science-Industry Student Research Placement Program “short-term, research-based industry placements” Rationale (i) facilitating industry experience for upper undergraduate students; (ii) building collaborations in work integrated learning between Monash and industry partners; (iii) furthering science and its broader communication.

15. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology About Bianca Second year in undergraduate Computer Science

16. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Analysing edits by formatter Per forecast period Per state Percentages

17. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Marine edits by forecast period - percentage

18. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Marine edits by forecast period - totals

19. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Precis edits by forecast period - percentages

20. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Precis edits by forecast period - totals

21. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Town, metro and district edits by forecast period - percentage

22. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Town, metro and district edits by forecast period - totals

23. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology NSW edits by formatter - percentage

24. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology NSW edits by formatter - totals

25. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology SA edits by formatter - totals

26. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology SA edits by formatter - percentage

27. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology TAS edits by formatter - totals

28. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology TAS edits by formatter - percentage

29. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology VIC edits by formatter - totals

30. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology VIC edits by formatter - percentage

31. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology WA edits by formatter - totals

32. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology WA edits by formatter - percentage

33. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons Hypothesis: Bigger towns (capital cities) get more attention than smaller towns Does “the gap” represent editing that would be done if forecasters had infinite time? Or does it represent stylistic editing that could be done without?

34. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - VIC

35. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - NSW

36. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - TAS

37. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - SA

38. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - WA

39. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparisons Hypothesis: initial edits very low, increase as forecasters learn to use the GFE and have more time for text editing “Month 0” is the month the GFE went live in that region VIC: 5/2009 NSW: 9/2010 TAS: 6/2011 SA: 11/2011 WA: 10/2012 Each state has a different version of the GFE and different weather conditions The dot and percentage on the far right represents edits in 11/2012

40. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison – all formatters (incl. warning)

41. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - Precis

42. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - Town

43. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - District

44. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - Metropolitan

45. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - marine

46. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology About Jaime  4th year in double undergraduate degree:  Science  Major - Pure Mathematics  Minor - Developmental Psychology  Minor – Computational Mathematics  Computer Science  Major - Intelligent Systems  Major – Software Engineering  5 year course

47. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Edits per formatter by phrase type edited How do we do this? Formatter Unedited forecastEdited forecast Break into sentences Identify the sentence type Examining key known words within the sentence to determine type Unedited categorized sentencesEdited categorized sentences Compare categories: Identify if a change is present

48. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - Town 01-NOV-2012 --- 30-NOV-2012 :: TOWN_FORECAST

49. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - District 01-NOV-2012 --- 30-NOV-2012 :: DISTRICT_FORECAST

50. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - Metropolitan 01-NOV-2012 --- 30-NOV-2012 :: METROPOLITAN_FORECAST

51. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - Coastal 01-NOV-2012 --- 30-NOV-2012 :: COASTAL_FORECAST

52. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited – Local Waters 01-NOV-2012 --- 30-NOV-2012 :: LOCAL_WATERS_FORECAST

53. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - Marine 01-NOV-2012 --- 30-NOV-2012 :: MARINE_FORECAST

54. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Precis: Investigating additions/removals How do we do this? Formatter Unedited forecastEdited forecast Identify if the given key is present Unedited keys identifiedEdited keys identified Compare keys: Identify if a key has been added/removed

55. Ratio to un-edited Text The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology

56. Morning – strongly removed 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

57. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Morning – strongly removed 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

58. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Afternoon – strongly added 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

59. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Afternoon – strongly added 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

60. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Late – strongly added 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

61. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Late – strongly added 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

62. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Developing – strongly removed 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

63. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Developing – strongly removed 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

64. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Clearing – strongly removed 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

65. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Clearing – strongly removed 01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT

66. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Towns - Haze/Humid? 01-FEB-2012 --- 29-FEB-2012 :: PRECIS_TEXT The forecaster trick to get humid into the forecast: – Add haze into the grids. – Generate forecasts. – Remove haze from the grids. – Complete a search and replace all: haze to humid.

67. The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Bianca Gibson: brgib1@student.monash.edubrgib1@student.monash.edu Jaime Clark: jkcla3@student.monash.edujkcla3@student.monash.edu Brianna Laugher: b.laugher@bom.gov.au, x4690b.laugher@bom.gov.au Tennessee Leeuwenburg: t.leeuwenburg@bom.gov.au, x4310t.leeuwenburg@bom.gov.au Web: www.cawcr.gov.au Thank you www.cawcr.gov.au