1. PP CARMA Common Area with Rfdbk/MEC Application
Amalia IRIZA-BURCA (NMA), Flora GOFA (HNMS)
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 1

2. PP CARMA
Goal
replace the existing VERSUS verification software environment with the MEC-Rfdbk software developed by DWD, as a Common Verification Software (CVS) to perform part of the verification activities in the consortium
main use of the new CVS - production of the Common Plot (CP) verification
EPS, spatial and other verification - with MEC-Rfdbk (not the purpose of this project) or with other available tools (VERSUS, VAST, etc.) in each service
addresses the need to perform traditional point verification both for the surface and the upper air
centralized transfer and visualization of CP statistics on the COSMO wed server (following NWP Test suite example)
Link to Support Activities - NWP Test Suite and Common Plot verification (SPRT)
Start – End
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 2

3. The Model Equivalent Calculator (MEC)
PP CARMA
The Model Equivalent Calculator (MEC)
applies the observation operators from the DA scheme to model forecasts (COSMO, ICON)
stores the results in verification files (NetCDF feedback file format)
fof-files (feedback file format as well) written by COSMO (nudging mode or fg run for KENDA)
use the original observational data (CDFIN, BUFR2NetCDF) => verification files for each observation type (TEMP, PILOT, SYNOP, etc)
Characteristics and requirements
produces feedback files
namelist based
Installation
Sources: Fortran 2003/2008 and C (Makefile for gfortran provided)
Dependencies: NetCDF, CGRIBEX (MPI Hamburg), GRIP-API (ECMWF), (MPI recommended), Fortran compiler, C compiler
Sufficient memory to hold one model state (1 ensemble state)
IO specifications
model in Grib2 format – COSMO or ICON-LAM
parameters - PS, T, U, V, P, Q (mandatory, all model levels); T2M, TD2M, CLC, CLCT, CLCL, CLCM, CLCH, CLC, H_SNOW, TOT_PREC, VMAX_10, TMIN_2M, TMAX_2M
observations (CDFIN: BUFR converted by bufrx2netcdf to NetCDF)
Output: ver-files, NetCDF feedback files including past forecasts
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 3

4. Rfdbk package PP CARMA Characteristics and requirements
R-based code of functions developed to exploit the information contained in feedback files
load FF content (partially, parallel), calculate basic verification scores (deterministic & EPS)
perform some convenience functions like data adjustment, re-labelling, binning etc
Characteristics and requirements
R interface for COSMO feedback files
main purpose of is to load feedback file content with R
additional functionalities useful for verification implemented as well
namelist based verification scipts using Rfdbk do the verification
Installation
Sources: R language
Dependencies: netcdf library and R (additional R packages: RNetCDF, data.table, parallel, stringr, survival, grid, verification, reshape2, pcaPP)
IO specifications
feedback files obtained previously with MEC - one file for each validity date and observation type
station list file
Output: score files for each validity date and observation type
Visualization of results on the COSMO web-server using Shiny R libraries
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 4

5. Rfdbk package PP CARMA Presentation during GM2017 by F. Fundel (DWD)
Rfdbk development is in the following stage:
Verification done for a number of observation types (SYNOP, TEMP, SATOB, GPSRO, SCATT, AIREP, PILOT
Methods continuous and categorical
EPS ensemble and probabilistic
Aggregation on e.g. sub-domains and height bins or levels or periods
Verification of „hindcast“ mode is possible
Cross model verification (e.g. COSMO vs. ICON) is possible
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 5

6. PP CARMA Design of the MEC-Rfdbk System
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 6

7. Prerequisites PP CARMA
DWD can provide support (through the PST) only for installation of MEC-Rfdbk in the “as-is” versions (both software and documentation), i.e. as it is implemented now at DWD.
Future developments and applications of MEC-Rfdbk by the COSMO members beyond common plots after the completion of the project are desirable but implementation and support, e.g. following the concept of COSMO-software, would require a re-assessment of resources, responsibilities and technical as well as organisational constraints – Not part of PP CARMA
If different software/library versions are used from the various services, then they hold the responsibility to adapt the installation files
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 7

8. Task 1. 1st Level Support Implementation and Training
PP CARMA
Task 1. 1st Level Support Implementation and Training
(Start –End )
First level support implementation and training of the Project Support Team (PST);
they will, in turn, ensure and support the implementation of the system for all the other partners.
Documentation review and preparation (MEC Rfdbk), analysis of resources required
Preparation of a complete example set of data and scripts to be used during the testing period.
Training provided by DWD experts for first level support to the PST
Implementation of the MEC-Rfdbk
Adaptation of scripts to produce the necessary statistical information for
Setup of web interface with the use of Shiny R routines on COSMO server
Deliverables:
Installation of the MEC-Rfdbk and first tests of the implementation.
Scripts for semi-automatic use of the system, available to project participants through common WG5 repository.
Data for a test period experiment
Completion of test period experiment and production of statistics by PST
Web interface to host Common Area plots for all countries
Contributors:
A. Iriza-Burca (NMA) – 0.17FTE, F. Gofa (HNMS) – 0.12FTE, B. Maco (NMA) – 0.155FTE, F. Fundel (DWD) – 0.13FTE, R. Potthast (DWD), E. Bauernschubert (DWD) – 0.12FTE, D. Boucouvala (HNMS) – 0.07FTE, T. Andreadis (HNMS) – 0.015FTE
Total Task 1: FTEs
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 8

9. Task 2. Second Level Implementation and support
PP CARMA
Task 2. Second Level Implementation and support
(Start –End )
Implementation of the MEC-Rfdbk system by all the member countries, with support from PST
DWD support will be provided only when necessary and always through the PST
Remote training by PST for users from each center, mailing list for problems solving, videoconferences
Implementation of MEC-Rfdbk system in each center with support of PST
Deliverables:
MEC-Rfdbk system installed by all project participants with support from PST.
Contributors:
A. Iriza-Burca (NMA) – 0.07FTE, F. Gofa (HNMS) – 0.03FTE, B. Maco (NMA) – 0.07FTE, D. Boucouvala (HNMS) – 0.03 FTE
NMSs responsible person
P. Kaufmann (MCH) – 0.15FTE, F. Batignani (CoMET) – 0.15FTE, J.Linkowska (IMGW-PIB) – 0.15FTE, D. Boucouvala (HNMS) FTE, A.Kirsanov – 0.15FTE (RHM), ?? (IMS) – 0.15FTE, M.S.Tesini (ArpaE) – 0.15FTE, N.Vela (Arpa-PT) – 0.15FTE
Total Task 2: FTEs
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 9

10. Task 3. Cross-validation of implementation (Start 09.2019–End 12.2019)
PP CARMA
Task 3. Cross-validation of implementation
(Start –End )
Testing of the verification system implementation and training through practical use
Performance of a complete seasonal test with all the necessary output for CP reports
Transfer statistical output to COSMO web server and visualization of results
Comparison of test output with VERSUS or any other “home” verification system.
Deliverables:
Evaluation of MEC-Rfdbk system for CP production
Comparison with results from other verification software
Contributors:
F. Fundel (DWD) – 0.10FTE, P. Kaufmann (MCH), A. Pauling – 0.13FTE, F. Batignani (CoMET) – 0.13FTE, J.Linkowska (IMGW-PIB) – 0.13FTE, T. Andreadis (HNMS) – 0.005FTE, D. Boucouvala (HNMS) FTE, M. Bogdan (NMA) – 0.015FTE, A. Iriza-Burca (NMA) – 0.13FTE, B. Maco (NMA) – 0.005FTE, A.Kirsanov (RHM) – 0.13FTE, ?? (IMS) – 0.13FTE, M.S.Tesini (ArpaE) – 0.13FTE, N.Vela (Arpa-PT) – 0.10FTE,
Total Task 3: FTEs
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 10

11. Task 4. Implementation of CP activity content
PP CARMA
Task 4. Implementation of CP activity content
(Start –End )
Implementation and testing of the MEC-Rfdbk system for Common Plot activities
Development of procedures for seasonal CP activities by all services
MEC output files for seasonal CP
Transfer of MEC output by each service to PST
Run of Rfdbk scripts by PST on an NMA local machine
Visualization of results and evaluation of statistical data by CP activity
Deliverables:
Implementation of Common Plot seasonal output with the MEC-Rfdbk system
Contributors:
F. Fundel (DWD) – 0.04FTE, P. Kaufmann (MCH) – 0.04FTE, F. Batignani (CoMET) – 0.04FTE, J.Linkowska (IMGW-PIB) – 0.04FTE, T. Andreadis (HNMS) – 0.01FTE, D. Boucouvala (HNMS) FTE, A. Iriza-Burca (NMA) – 0.09FTE, M. Bogdan (NMA) – 0.05FTE, B. Maco (NMA) – 0.01FTE, A.Kirsanov (RHM) – 0.04FTE, ?? (IMS) – 0.04FTE, M.S.Tesini (ArpaE) – 0.04
Total Task 4: FTEs
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 11

12. Task 5. Elaboration of guidelines for CARMA (MEC-Rfdbk) system use
PP CARMA
Task 5. Elaboration of guidelines for CARMA (MEC-Rfdbk) system use
(Start –End )
Preparation of documentation on the use of the MEC-Rfdbk system (incl. CP activities specifications)
Deliverables:
Guidelines on the implementation, use and application of the MEC-Rfdbk system for CP production
Contributors:
A. Iriza-Burca (NMA) – 0.05FTE, F. Gofa (HNMS) – 0.03FTE
Total Task 5: 0.08 FTEs
Task 0. Administrative Tasks
(Start – End )
Project coordination, meetings, preparation of plans/reports, workshops, regular web conference organization.
Amalia Iriza-Burca (NMA) – 0.2FTE
Total Task 0: 0.2 FTEs
Estimated resources (in FTE-years) needed in total: FTEs (mostly in 2019)
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 12

13. PP CARMA
Risks
MEC - depending on implementation and resource requirements (Task 1.1) - delays in the installation
COSMO Shiny server – might only be accessible to restricted users, due to web security issues
Solution: 2 people from the PST team (T. Andreadis - HNMS, B. Maco - NMA) will ensure the uploading of results on the web server.
Use of VERSUS - Task 3 involves a cross validation of the verification results against VERSUS (or another “home” verification system).
Risk: VERSUS system may cause some delays in the activities of this task.
Solution: subtask will be optional for participation.
Model data for MEC - Task 4 requires the use of model data generated by each center.
Risk: If necessity model (re)runs for MEC, task can be delayed (resources available at the national centers)
Rfdbk package limitations - Spatial methods not currently included, does not allow for conditional verification applications if the required information about the observations is not in the FF, not linked with an RDB.
Updates – if DWD will not provide new MEC/Rfdbk versions
Either the system will be maintained “as is” at implementation or each member responsible for later development.
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 13

14. Evaluation of new version 5.05 (vs. v5.03)
PP CARMA
Already implemented for NWP Meteorological Test ECMWF
"Testbed for the Evaluation of COSMO Model Versions" (2018 – 2020)
MEC-Rfdbk for the NWP Test
Evaluation of new version 5.05 (vs. v5.03)
forecast mode, DP 7km and 2.8km, VERSUS and MEC-Rfdbk
forecast mode, SP 7km and 2.8km, MEC-Rfdbk
hindcast mode, MEC-Rfdbk
Numerical Weather Prediction Meteorological Test Suite: COSMO 5.05 vs Document - A. Montani, F. Fundel, M. Bogdan, R.C. Dumitrache, F. Gofa, A. Iriza-Burca, F. Batignani
soon to be published on the COSMO web site
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 14

15. PP CARMA
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 15

16. Questions? Thank you for your attention!
20th COSMO General Meeting – Sankt Petersburg, 3 – 7 September 2018 16