PP CARMA Common Area with Rfdbk/MEC Application Amalia IRIZA-BURCA (NMA), Flora GOFA (HNMS) 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 1

PP CARMA Goal replace the existing VERSUS verification software environment with the MEC-Rfdbk software developed by DWD, as a Common Verification Software (CVS) to perform part of the verification activities in the consortium main use of the new CVS - production of the Common Plot (CP) verification EPS, spatial and other verification - with MEC-Rfdbk (not the purpose of this project) or with other available tools (VERSUS, VAST, etc.) in each service addresses the need to perform traditional point verification both for the surface and the upper air centralized transfer and visualization of CP statistics on the COSMO wed server (following NWP Test suite example) Link to Support Activities - NWP Test Suite and Common Plot verification (SPRT) Start 12.2018 – End 02.2020 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 2

The Model Equivalent Calculator (MEC) PP CARMA The Model Equivalent Calculator (MEC) applies the observation operators from the DA scheme to model forecasts (COSMO, ICON) stores the results in verification files (NetCDF feedback file format) fof-files (feedback file format as well) written by COSMO (nudging mode or fg run for KENDA) use the original observational data (CDFIN, BUFR2NetCDF) => verification files for each observation type (TEMP, PILOT, SYNOP, etc) Characteristics and requirements produces feedback files namelist based Installation Sources: Fortran 2003/2008 and C (Makefile for gfortran provided) Dependencies: NetCDF, CGRIBEX (MPI Hamburg), GRIP-API (ECMWF), (MPI recommended), Fortran compiler, C compiler Sufficient memory to hold one model state (1 ensemble state) IO specifications model in Grib2 format – COSMO or ICON-LAM parameters - PS, T, U, V, P, Q (mandatory, all model levels); T2M, TD2M, CLC, CLCT, CLCL, CLCM, CLCH, CLC, H_SNOW, TOT_PREC, VMAX_10, TMIN_2M, TMAX_2M observations (CDFIN: BUFR converted by bufrx2netcdf to NetCDF) Output: ver-files, NetCDF feedback files including past forecasts 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 3

Rfdbk package PP CARMA Characteristics and requirements R-based code of functions developed to exploit the information contained in feedback files load FF content (partially, parallel), calculate basic verification scores (deterministic & EPS) perform some convenience functions like data adjustment, re-labelling, binning etc Characteristics and requirements R interface for COSMO feedback files main purpose of is to load feedback file content with R additional functionalities useful for verification implemented as well namelist based verification scipts using Rfdbk do the verification Installation Sources: R language Dependencies: netcdf library and R (additional R packages: RNetCDF, data.table, parallel, stringr, survival, grid, verification, reshape2, pcaPP) IO specifications feedback files obtained previously with MEC - one file for each validity date and observation type station list file Output: score files for each validity date and observation type Visualization of results on the COSMO web-server using Shiny R libraries 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 4

Rfdbk package PP CARMA Presentation during GM2017 by F. Fundel (DWD) Rfdbk development is in the following stage: Verification done for a number of observation types (SYNOP, TEMP, SATOB, GPSRO, SCATT, AIREP, PILOT Methods continuous and categorical EPS ensemble and probabilistic Aggregation on e.g. sub-domains and height bins or levels or periods Verification of „hindcast“ mode is possible Cross model verification (e.g. COSMO vs. ICON) is possible 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 5

PP CARMA Design of the MEC-Rfdbk System 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 6

Prerequisites PP CARMA DWD can provide support (through the PST) only for installation of MEC-Rfdbk in the “as-is” versions (both software and documentation), i.e. as it is implemented now at DWD. Future developments and applications of MEC-Rfdbk by the COSMO members beyond common plots after the completion of the project are desirable but implementation and support, e.g. following the concept of COSMO-software, would require a re-assessment of resources, responsibilities and technical as well as organisational constraints – Not part of PP CARMA If different software/library versions are used from the various services, then they hold the responsibility to adapt the installation files 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 7

Task 1. 1st Level Support Implementation and Training PP CARMA Task 1. 1st Level Support Implementation and Training (Start 12.2018–End 04.2019) First level support implementation and training of the Project Support Team (PST); they will, in turn, ensure and support the implementation of the system for all the other partners. Documentation review and preparation (MEC Rfdbk), analysis of resources required Preparation of a complete example set of data and scripts to be used during the testing period. Training provided by DWD experts for first level support to the PST Implementation of the MEC-Rfdbk system @NMA and @HNMS Adaptation of scripts to produce the necessary statistical information for Setup of web interface with the use of Shiny R routines on COSMO server Deliverables: Installation of the MEC-Rfdbk system @NMA and @HMNS and first tests of the implementation. Scripts for semi-automatic use of the system, available to project participants through common WG5 repository. Data for a test period experiment Completion of test period experiment and production of statistics by PST Web interface to host Common Area plots for all countries Contributors: A. Iriza-Burca (NMA) – 0.17FTE, F. Gofa (HNMS) – 0.12FTE, B. Maco (NMA) – 0.155FTE, F. Fundel (DWD) – 0.13FTE, R. Potthast (DWD), E. Bauernschubert (DWD) – 0.12FTE, D. Boucouvala (HNMS) – 0.07FTE, T. Andreadis (HNMS) – 0.015FTE Total Task 1: 0.780 FTEs 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 8

Task 2. Second Level Implementation and support PP CARMA Task 2. Second Level Implementation and support (Start 05.2019–End 10.2019) Implementation of the MEC-Rfdbk system by all the member countries, with support from PST DWD support will be provided only when necessary and always through the PST Remote training by PST for users from each center, mailing list for problems solving, videoconferences Implementation of MEC-Rfdbk system in each center with support of PST Deliverables: MEC-Rfdbk system installed by all project participants with support from PST. Contributors: A. Iriza-Burca (NMA) – 0.07FTE, F. Gofa (HNMS) – 0.03FTE, B. Maco (NMA) – 0.07FTE, D. Boucouvala (HNMS) – 0.03 FTE NMSs responsible person P. Kaufmann (MCH) – 0.15FTE, F. Batignani (CoMET) – 0.15FTE, J.Linkowska (IMGW-PIB) – 0.15FTE, D. Boucouvala (HNMS) - 0.015FTE, A.Kirsanov – 0.15FTE (RHM), ?? (IMS) – 0.15FTE, M.S.Tesini (ArpaE) – 0.15FTE, N.Vela (Arpa-PT) – 0.15FTE Total Task 2: 0.90 FTEs 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 9

Task 3. Cross-validation of implementation (Start 09.2019–End 12.2019) PP CARMA Task 3. Cross-validation of implementation (Start 09.2019–End 12.2019) Testing of the verification system implementation and training through practical use Performance of a complete seasonal test with all the necessary output for CP reports Transfer statistical output to COSMO web server and visualization of results Comparison of test output with VERSUS or any other “home” verification system. Deliverables: Evaluation of MEC-Rfdbk system for CP production Comparison with results from other verification software Contributors: F. Fundel (DWD) – 0.10FTE, P. Kaufmann (MCH), A. Pauling – 0.13FTE, F. Batignani (CoMET) – 0.13FTE, J.Linkowska (IMGW-PIB) – 0.13FTE, T. Andreadis (HNMS) – 0.005FTE, D. Boucouvala (HNMS) - 0.13FTE, M. Bogdan (NMA) – 0.015FTE, A. Iriza-Burca (NMA) – 0.13FTE, B. Maco (NMA) – 0.005FTE, A.Kirsanov (RHM) – 0.13FTE, ?? (IMS) – 0.13FTE, M.S.Tesini (ArpaE) – 0.13FTE, N.Vela (Arpa-PT) – 0.10FTE, Total Task 3: 1.25 FTEs 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 10

Task 4. Implementation of CP activity content PP CARMA Task 4. Implementation of CP activity content (Start 10.2019–End 12.2019 ) Implementation and testing of the MEC-Rfdbk system for Common Plot activities Development of procedures for seasonal CP activities by all services MEC output files for seasonal CP Transfer of MEC output by each service to PST Run of Rfdbk scripts by PST on an NMA local machine Visualization of results and evaluation of statistical data by CP activity Deliverables: Implementation of Common Plot seasonal output with the MEC-Rfdbk system Contributors: F. Fundel (DWD) – 0.04FTE, P. Kaufmann (MCH) – 0.04FTE, F. Batignani (CoMET) – 0.04FTE, J.Linkowska (IMGW-PIB) – 0.04FTE, T. Andreadis (HNMS) – 0.01FTE, D. Boucouvala (HNMS) - 0.04FTE, A. Iriza-Burca (NMA) – 0.09FTE, M. Bogdan (NMA) – 0.05FTE, B. Maco (NMA) – 0.01FTE, A.Kirsanov (RHM) – 0.04FTE, ?? (IMS) – 0.04FTE, M.S.Tesini (ArpaE) – 0.04 Total Task 4: 0.44 FTEs 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 11

Task 5. Elaboration of guidelines for CARMA (MEC-Rfdbk) system use PP CARMA Task 5. Elaboration of guidelines for CARMA (MEC-Rfdbk) system use (Start 12.2019–End 02.2020) Preparation of documentation on the use of the MEC-Rfdbk system (incl. CP activities specifications) Deliverables: Guidelines on the implementation, use and application of the MEC-Rfdbk system for CP production Contributors: A. Iriza-Burca (NMA) – 0.05FTE, F. Gofa (HNMS) – 0.03FTE Total Task 5: 0.08 FTEs Task 0. Administrative Tasks (Start 10.2018 – End 12.2019 ) Project coordination, meetings, preparation of plans/reports, workshops, regular web conference organization. Amalia Iriza-Burca (NMA) – 0.2FTE Total Task 0: 0.2 FTEs Estimated resources (in FTE-years) needed in total: 3.450 FTEs (mostly in 2019) 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 12

PP CARMA Risks MEC - depending on implementation and resource requirements (Task 1.1) - delays in the installation COSMO Shiny server – might only be accessible to restricted users, due to web security issues Solution: 2 people from the PST team (T. Andreadis - HNMS, B. Maco - NMA) will ensure the uploading of results on the web server. Use of VERSUS - Task 3 involves a cross validation of the verification results against VERSUS (or another “home” verification system). Risk: VERSUS system may cause some delays in the activities of this task. Solution: subtask will be optional for participation. Model data for MEC - Task 4 requires the use of model data generated by each center. Risk: If necessity model (re)runs for MEC, task can be delayed (resources available at the national centers) Rfdbk package limitations - Spatial methods not currently included, does not allow for conditional verification applications if the required information about the observations is not in the FF, not linked with an RDB. Updates – if DWD will not provide new MEC/Rfdbk versions Either the system will be maintained “as is” at implementation or each member responsible for later development. 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 13

Evaluation of new version 5.05 (vs. v5.03) PP CARMA Already implemented for NWP Meteorological Test Suite @ ECMWF "Testbed for the Evaluation of COSMO Model Versions" (2018 – 2020) MEC-Rfdbk for the NWP Test Suite@ECMWF Evaluation of new version 5.05 (vs. v5.03) forecast mode, DP 7km and 2.8km, VERSUS and MEC-Rfdbk forecast mode, SP 7km and 2.8km, MEC-Rfdbk hindcast mode, MEC-Rfdbk Numerical Weather Prediction Meteorological Test Suite: COSMO 5.05 vs. 5.03 Document - A. Montani, F. Fundel, M. Bogdan, R.C. Dumitrache, F. Gofa, A. Iriza-Burca, F. Batignani soon to be published on the COSMO web site 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 14

PP CARMA 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 15

Questions? Thank you for your attention! 20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 16