Weather Forecast Verification Training - Recent Advancements - 0. Introduction Weather Forecast Verification Training - Recent Advancements - 1 training modules 2 Pertti Nurmi Meteorological Research & Development Finnish Meteorological Institute Laurie Wilson Meteorological Service of Canada Abstract: RECENT ADVANCEMENTS IN WEATHER FORECAST VERIFICATION TRAINING Pertti Nurmi 1) and Laurence Wilson 2) pertti.nurmi@fmi.fi 1) Finnish Meteorological Institute, Helsinki, Finland 2) Environment Canada, Montreal, Canada The WWRP/WGNE Joint Working Group on Verification (JWGV) is a body under WMO established for fostering and promoting of weather forecast verification activities. The members of the JWGV are a dozen verification experts from several national meteorological institutes and universities from several countries and continents. One important objective of the JWGV is to organize various forms of training of forecast verification. The JWGV has arranged three extensive forecast verification workshops during the past few years (Boulder, 2002; Montreal, 2004; Reading, 2007). The most recent one, the 3rd International Workshop on Verification Methods, held at ECMWF, included comprehensive tutorial sessions. These sessions provided a pedagogical introduction and a review of new verification techniques of deterministic and probabilistic (including ensemble) forecasts. Hands-on laboratory sessions were an integral part of the tutorials. The students were able to apply state-of-the-art verification methods on real case studies and, in some cases, used their own datasets which were sent to the course organizers in advance. The outcome of the tutorials is a comprehensive set of training material which can be effectively used for self-learning purposes as well as for arranging internal training sessions within the home institutes of the students. Another recent forecast verification training activity (by two members of the JWGV) is the development and realization of a set of EUMETCAL training modules. Produced during 2006 and 2007, the EUMETCAL training package covers verification methods for (i) continuous, (ii) categorical, and (iii) probabilistic forecasts. All of the modules provide information on the basic and most commonly used verification techniques, and include hands-on tools either for trainers or users of the modules to illustrate and perform practical verification experiments with the data samples provided. The CAL modules are aimed at assisting the understanding of the behavior of various verification methods and to choose and apply relevant techniques for given forecast situations. After completion of the full set of CAL modules, the users would have sufficient knowledge to be able to choose and apply suitable verification methods for a given type of weather forecast and/or weather parameter. The paper provides an overview of the most important outcomes of the 2007 3rd International Workshop on Verification Methods, as well as an account of the key elements of the EUMETCAL Forecast Verification Learning Modules.

Verification (training) History, 1884 Hypothesis Tornados in USA Verification (training) History, 1884 Tornados in USA ”The Finlay case” 2680 + 30 2800 = 96,8 % 2750 + 0 2800 = 98,2 % Never forecast a Tornado a Tornado ! Hypothesis CALMet - Beijing 5 July 2007

Goals of Verification Training: Goals of Verification Training: Understand the basic properties and relationships of (common) verification measures Learn to extract useful information from (graphical) verification results Increase interest in forecast verification and the methods Apply verification in everyday fc practice at every fc office Generally HEAVILY neglected topic in training activities ! CALMet - Beijing 5 July 2007

International Activities ”WMO WWRP/WGNE JWGV” (* Harold Brooks International Activities ”WMO WWRP/WGNE JWGV” (* Harold Brooks (NSSL, USA; not shown) Barbara Brown (NCAR, USA; Chair) Barbara Casati (MSC, Canada) Ulrich Damrath (DWD, Germany) Beth Ebert (BMRC, Australia) Anna Ghelli (ECMWF) Pertti Nurmi (FMI, Finland) David Stephenson (U. Reading, UK; not shown) Clive Wilson (UKMO, UK) Laurie Wilson *) WMO ~ World Meteorological Organization WWRP ~ World Weather Research Program WGNE ~ Working Group on Numerical Experimentation JWGV ~ Joint Working Group on Verification CALMet - Beijing 5 July 2007

1 2 International Activities ”WMO WWRP/WGNE JWGV” Goals of JWGV: International Activities ”WMO WWRP/WGNE JWGV” Goals of JWGV: Cooperation between forecasters, users and verifiers Development and application of verification methodologies Promote awareness of the importance of verification  via training Recent activities of JWGV: Research and training workshops on verification methods  Recommendations for verification methodologies Test-bed activities Beijing 2008 Olympics Real-Time Forecast Verification (RTFV) THORPEX and TIGGE EUMETCAL verification training modules  1 2 CALMet - Beijing 5 July 2007

JWGV website Google search: “Forecast > verification” JWGV website Google search: “Forecast > verification” gives this as the first prompt ! www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html CALMet - Beijing 5 July 2007

CALMet - Beijing 5 July 2007

1 www.ecmwf.int/ newsevents/ meetings/ workshops/ 2007/ jwgv/ lecture_notes/ index.html CALMet - Beijing 5 July 2007

1 31 students, One / country TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) Argentina, Australia , Brasil, Canada, Checz Republic, China, Croatia, Denmark, Ethiopia, Finland, France, Germany, Hong Kong, India, Italy, Kenya, Lithuania, Nigeria, Norway, Romania, Senegal, Slovakia, South Africa, South Korea, Sweden, Switzerland, Ukraine, USA CALMet - Beijing 5 July 2007

1 Components: Tutorial ~ 2 days Scientific workshop and presentations ~ 3 days i. Tutorial contents: Use/interpretation of “standard” verification methods Application of new methodologies (e.g. spatial forecast verification) Hands-on laboratory exercises ”R” software (verification components) training ii. Workshop contents: Invited keynotes & contributed presentations User-oriented forecast verification Verification of extreme/ high impact weather, EPS, uncertainty … TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 Tutorial contents: Introduction to R (Matt Pocernich) Basic Verification Concepts (Barb Brown) Verification of Continuous Predictands (Barbara Casati) Verification of Categorical Predictands (Pertti Nurmi) Verification of Probabilistic Predictands and Ensemble Forecasts (Beth Ebert) Confidence Intervals and Hypothesis Tests (Simon Mason) Projects (Laurie Wilson) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (i) www.r-project.org TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (ii) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (ii) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (iii) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (iii) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (iii) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (iv) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (iv) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (v) E.Ebert@bom.gov.au TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (v) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (v) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (v) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 (vi) TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

1 Projects (vii) Laurie Wilson Environment Canada Project No. 4: Wind speed forecasts (Dataset: Moreau/ Canada) Choose the verification problem… Look at the data… Plot forecast-observation scatterplots... Choose verification measures... Calculate the scatter index... Project No. 5 Climatological analysis (Dataset: Korecha/ Ethiopia) Produce box and whisker plots… Calculate the annual statistics… Do the same for each month separately... Project No. 6: Precipitation forecasts (Dataset: Suaya/ Argentina) Look at the data… Prepare histogram plots… Pose the relevant verification questions… Carry out the verification and produce graphs to display your results… Projects Laurie Wilson Environment Canada Lawrence.Wilson@ec.gc.ca Project No. 1: Verification of ensemble-based PoP forecasts (Dataset: Tong/ Hong Kong) Calculate the predictand… Verify the POP forecasts... Prepare and plot a reliability table... Compute the BS… Project No. 2: Temperature verification (Dataset: Diaconu/ Romania) Choose a verification problem… Look at the data… Produce and examine scatter plots…Choose verification scores… Project 3: Cloudiness forecasts in 3 categories (Dataset: Diaconu/ Romania) Choose a verification issue... Look at the data… Produce contingency tables... Choose verification measures… deterministic CALMet - Beijing 5 July 2007

“Probably the best training 1 “Probably the best training course I ever had. Thank you to all of you! “ Overall evaluation by students: > 2.5 / 3 TOM: FYI, the 1st international workshop was not sponsored by WMO. It was the Aug 2002 workshop at NCAR, which we organized with support from the FAA and from NCAR. The JWGV did not exist at that point, but our proposal to form the JWGV were formulated then. The 2nd workshop was 1 year ago (Sept. 2004) in Montreal. It was supported by WWRP and WGNE (though most funds came from WWRP) CALMet - Beijing 5 July 2007

2 CAL training modules What’s included in the modules: Introduction Goals and general guidelines Continuous variables Categorical events Probability forecasts Glossary + References + Practical exercises ~ Quiz ! finished spring 2006 finished winter 2007 finished summer 2007 CALMet - Beijing 5 July 2007

2 www.eumetcal.org CALMet - Beijing 5 July 2007

Demonstration on Sat 10:10-11:00 2 www.eumetcal.org Demonstration on Sat 10:10-11:00  WELCOME TO TEST  CALMet - Beijing 5 July 2007

2 Demonstration on Sat 10:10-11:00  WELCOME TO TEST  CAL modules ! www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm Demonstration on Sat 10:10-11:00  WELCOME TO TEST  CAL modules ! … produced by Pertti Nurmi, Laurie Wilson & Sigbritt Näsman, 2006-07

2 … open browser ! ”Help button” ”Glossary button” www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm 2 ”Help button” ”Glossary button” ”Communication tool” ”Return to Index” … open browser !

References: Jolliffe and Stephenson, Eds. (2003): Forecast Verification: A Practitioner’s Guide, Wiley & Sons. Nurmi (2003): Recommendations on the verification of local weather forecasts. ECMWF Tech. Mem. 430. Wilks (2006): Statistical Methods in the Atmospheric Sciences, Chapter 7 (Forecast Verification). Academic Press. WMO Joint Working Group on Verification Website: www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html ECMWF Verification Methods Training Workshop Lectures: www.ecmwf.int/newsevents/meetings/workshops/2007/jwgv/lecture_notes/index.html EUMETCAL Forecast Verification Learning Modules: www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm 1 2 CALMet - Beijing 5 July 2007

… except (proper) forecast verification ! Finishing motto There is nothing as difficult as forecasting the weather (correctly) ... … except (proper) forecast verification ! - Allan Murphy, RIP Training is one key issue ! CALMet - Beijing 5 July 2007

Thank You Pertti Nurmi Meteorological Research & Development Finnish Meteorological Institute