The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Forecast text editing rates and analysis Jaime Clark and Bianca Gibson Brianna Laugher Tennessee Leeuwenburg December

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Graphical Forecast Editor

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Text formatters

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Text formatters - editing

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GuidanceGrid Editing & Smart Tools Text Formatting Forecast Production GFE Forecasting

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Text Formatting Forecast Production GFE Forecasting Grids edited text original gfemetrics

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Metrics

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Metrics

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Metrics

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology GFE Metrics Web front-end to “FDB” database created/maintained by ITB Initially developed by Brianna, further developed by Anthony Briggs STATS/SUMMARY Can view graphs/tables of total edits: by time, by state, by formatter VIEW Can view any edits made to any GFE forecasts Can query edits by state, date, formatter, period, issuance, locale Can search for edited/unedited/either containing particular words TOOLS Can view graphs/tables of most used tools: by time, by state

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Metric? We initially hoped that improvements to the formatters would be reflected in lower edit rates... this appears not to be the case... but it's hard to tell...

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Dimensions of edits Is the generated text consistent with grid contents? “Wrong” results (genuine error) vs stylistic changes (subjective) “Possible shower” -> “Mostly dry” Is the edited text consistent with Services policy? Policy may need resolving with Services “Fine”, “moderating”, “freshening” Is the info required to produce the edited text present in the GFE? Formatter missing rules - “Humid”, “Hot”, sea breeze GFE missing grids – climatology, uncertainty! Other info in forecaster brain (local expertise) Can potentially fix Won't fix Can't fix (for everyone) Should fix ASAP

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Limitations  Formatters are mainly just a group of phrases – Sky, Wx, Wind, Temp, Seas, Swell. Effort is directed at improving phrases – but there's no way to check edit rates of phrases  Hard to get a sense of any general trend when a query returns hundreds of results  Little/no time to devote to exploring regional differences

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Monash Talented Students program Science-Industry Student Research Placement Program “short-term, research-based industry placements” Rationale (i) facilitating industry experience for upper undergraduate students; (ii) building collaborations in work integrated learning between Monash and industry partners; (iii) furthering science and its broader communication.

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology About Bianca Second year in undergraduate Computer Science

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Analysing edits by formatter Per forecast period Per state Percentages

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Marine edits by forecast period - percentage

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Marine edits by forecast period - totals

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Precis edits by forecast period - percentages

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Precis edits by forecast period - totals

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Town, metro and district edits by forecast period - percentage

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Town, metro and district edits by forecast period - totals

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology NSW edits by formatter - percentage

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology NSW edits by formatter - totals

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology SA edits by formatter - totals

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology SA edits by formatter - percentage

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology TAS edits by formatter - totals

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology TAS edits by formatter - percentage

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology VIC edits by formatter - totals

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology VIC edits by formatter - percentage

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology WA edits by formatter - totals

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology WA edits by formatter - percentage

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons Hypothesis: Bigger towns (capital cities) get more attention than smaller towns Does “the gap” represent editing that would be done if forecasters had infinite time? Or does it represent stylistic editing that could be done without?

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - VIC

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - NSW

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - TAS

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - SA

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Two-Town comparisons - WA

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparisons Hypothesis: initial edits very low, increase as forecasters learn to use the GFE and have more time for text editing “Month 0” is the month the GFE went live in that region VIC: 5/2009 NSW: 9/2010 TAS: 6/2011 SA: 11/2011 WA: 10/2012 Each state has a different version of the GFE and different weather conditions The dot and percentage on the far right represents edits in 11/2012

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison – all formatters (incl. warning)

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - Precis

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - Town

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - District

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - Metropolitan

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Go-Live comparison - marine

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology About Jaime  4th year in double undergraduate degree:  Science  Major - Pure Mathematics  Minor - Developmental Psychology  Minor – Computational Mathematics  Computer Science  Major - Intelligent Systems  Major – Software Engineering  5 year course

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Edits per formatter by phrase type edited How do we do this? Formatter Unedited forecastEdited forecast Break into sentences Identify the sentence type Examining key known words within the sentence to determine type Unedited categorized sentencesEdited categorized sentences Compare categories: Identify if a change is present

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - Town 01-NOV NOV-2012 :: TOWN_FORECAST

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - District 01-NOV NOV-2012 :: DISTRICT_FORECAST

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - Metropolitan 01-NOV NOV-2012 :: METROPOLITAN_FORECAST

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - Coastal 01-NOV NOV-2012 :: COASTAL_FORECAST

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited – Local Waters 01-NOV NOV-2012 :: LOCAL_WATERS_FORECAST

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Phrase type edited - Marine 01-NOV NOV-2012 :: MARINE_FORECAST

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Precis: Investigating additions/removals How do we do this? Formatter Unedited forecastEdited forecast Identify if the given key is present Unedited keys identifiedEdited keys identified Compare keys: Identify if a key has been added/removed

Ratio to un-edited Text The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology

Morning – strongly removed 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Morning – strongly removed 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Afternoon – strongly added 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Afternoon – strongly added 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Late – strongly added 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Late – strongly added 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Developing – strongly removed 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Developing – strongly removed 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Clearing – strongly removed 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Clearing – strongly removed 01-JAN NOV-2012 :: PRECIS_TEXT

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Towns - Haze/Humid? 01-FEB FEB-2012 :: PRECIS_TEXT The forecaster trick to get humid into the forecast: – Add haze into the grids. – Generate forecasts. – Remove haze from the grids. – Complete a search and replace all: haze to humid.

The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology Bianca Gibson: Jaime Clark: Brianna Laugher: Tennessee Leeuwenburg: Web: Thank you