PROGRESS ON A KNOWLEDGE-BASED INTERNET WEATHER FORECASTING SYSTEM 100-day evaluation trial from 25 September, 2002 to 8 January, Harvey Stern (Bureau of Meteorology, Australia)

Location Diagram (1)

Location Diagram (2)

Example (NWP MSL Prognosis)

Example (The System’s Output)

Background An early version of the knowledge-based system was presented to the 18th IIPS Conference (Stern, 2002). The system was developed for the small (227,000 sq km) southeast Australian State of Victoria. It was described as being capable of generating forecasts for public, aviation, marine and media interests, in languages other than English, and for more than 200 localities in Victoria. The large volume of output proposed here (forecasts for than 200 localities) would be difficult (if not impossible) to generate manually However, it would be straightforward (to generate) utilising automated systems.

“Bank” of forecaster experience A major benefit of the knowledge-based system is that it incorporates an extensive "bank" of forecaster experience. Ramage (1993) has proposed an "iterative" approach to "locking in" improvements in forecasting methodology. The automated nature of the system lends itself to Ramage's approach. The system's skill increases as new knowledge is incorporated into its operation. Hence, progress is gradually made towards the realisation of Ramage's dream.

Not “yet another” instrument of forecast guidance The system is not seen as "yet another" instrument of forecast guidance. Rather, its development is seen as a logical step along the path of having the computer replicate the best aspects of the manual side of the forecast process, by systematically "locking in" new knowledge. As Brooks (1995), wrote: "technology, which initially allowed humans to make routine weather forecasts, will soon close that avenue of human endeavour... (and thereby permit) concentration on severe events". The main human-interaction is in utilising forecast verification data (after the event) to iteratively incorporate new additional forecaster knowledge into its algorithm.

A Forecasting Research Tool A simple illustration is now presented of how the system for generating Internet Forecasts might be employed as a research tool to enhance understanding. The first Figure following depicts the MSL pressure (MSLP) distribution for Synoptic Type 41 (strong cyclonic SSW flow).

A Forecasting Research Tool (cont.) The Figure plots the annual march of PoP forecasts for strong cyclonic SSW flow, with 850 hPa temperatures of 0 deg C, & with 700 hPa RHs of 30% & 70%. One might ask why there is such a marked difference between winter & summer responses to the same situation.

A Forecasting Research Tool (cont.) An illustration is now presented of how one may proceed from the enhanced understanding achieved via the preceding analysis, to increase the potential accuracy of the forecasts generated. A study of Synoptic Type 41 cases suggests that the strength of the SSW flow between SE Australia and over waters to the west of that region might be significant. Regression analysis confirms that adding cyclonicity to the other predictors, cyclonicity proves to be the most significant.

A Forecasting Research Tool (cont.) The Figure plots the annual march of PoP forecasts for strong cyclonic SSW flow with RH=30% and the new equation. It reveals just how important cyclonicity is in assessing the likelihood of precipitation.

November 2001 trial The system's forecast performance (during November 2001) was evaluated for the city of Melbourne. The evaluation showed that, although superiority over climatology was achieved, the forecasts (on most measures) proved to be inferior to the official forecasts. The results of the November 2001 trial were analysed, potential improvements in the forecasting process (employed by the system) were identified, and those improvements were "locked in". The system (so modified) was then subjected to another evaluation for the city of Melbourne, this time over 100 days (between late September 2002 and early January 2003).

Overall Accuracy (in 2001)

Modifications The 2001 system operated by producing its predictions from a restricted number of discrete "forecast sets". The set that was chosen (by the system) was largely determined by the particular synoptic pattern suggested by the selected NWP model. The 2002/2003 modification utilises regression analysis to allow predictions to be selected from a continuous array of possible forecasts. The particular form of regression analysis employed is parameter enveloping. Parameter enveloping allows definition of how the various predictors impact upon, or envelope, the influence (on a predictand) of other predictors.

An illustration of parameter enveloping

Overall Accuracy (2002/2003)

RMS Error of Min Temp Forecasts

RMS Error of Max Temp Forecasts

% Correct Rain/No Rain

RMS Error of QPFs

Need for Further Testing The 100-day verification trial suggests that substantial progress, albeit uneven, towards achieving computer replication of the manual forecast process, has been made. Specifically, the impact of the "locked in" improvements upon the skill displayed by the system has been considerable, particularly for Day-1 forecasts. Nevertheless, the performance of the system over the 100- day trial is so impressive, that one must entertain the possibility that it was simply a "fluke". For this reason, further testing needs to be undertaken.

Implications It was considered that, should further testing confirm the 100- day trial results, the implications would be profound. We could see ourselves at the "dawn" of the operational implementation of Ramage's approach to weather forecasting which, through computer replication of the manual process, would allow for: Systematic incorporation of new procedures that may lead to a quantum leap in the accuracy of the forecast products (in a related piece of work, Ryan et al. (2003) have developed a system that archives the subjective inputs to the forecast process and makes them available for statistical analysis); The opportunity to greatly increase the number and variety of such products.

Further Testing So far, further testing is, indeed, confirming the 100-day trial results. A modest post-trial exercise, involving the derivation of forecasts for Day-1 only for the (now) 130 days ended February 1, 2003, yields: A CSS of 61.4 for the Internet Forecasts; and, A CSS of 60.3 for the official forecasts.