WMO 4.1 A critique of the existing data models for weather radar data exchange as presented in item 3.1 docs CBS/OPAG-IOS Workshop on Radar Data Exchange Exeter, UK, April 2013 Daniel Michelson, SMHI, Sweden

WMO Data exchange model has four components: 1.File format – container for storing data in physical files. 2.Data/information model – the way in which information is organized/represented, either in computer memory or files, independently of file format. 3.Envelope – extra header announcing/describing the contents of the information being exchanged. 4.Protocol – mechanisms for communication over a network.

test footer 3 Content-Disposition: form-data; name=" " Content-Type: multipart/form-data; charset=UTF-8 Content-Transfer-Encoding: 8bit --s7Wj4EEwDrDPwOdPM30KMQzaEpfnyl Content-Disposition: form-data; name=" "; filename="pas1.h5" Content-Type: multipart/form-data Content-Transfer-Encoding: binary  HDF ÿ (… binary payload ODIM_H5 file …) --s7Wj4EEwDrDPwOdPM30KMQzaEpfnyl-- E.g. an early BALTRAD HTML exchange message

WMO Different categories of file formats 1.Transmission – to facilitate data transfer from site to central facility. Can be optimized for network load balancing, e.g. ray-by-ray. 2.Production – contains all data and metadata required to derive higher-order outputs (e.g. QC:ed data, products) from input data. 3.Exchange – representation of data or product for sharing but not necessarily further processing, e.g. a radar composite or vertical profile. 4.Archive – e.g. to represent data from many different observation systems for storage.

WMO Basic data exchange concepts  “Passive” Simple availability, e.g. “pull” Sender/recipient are aware of each other but transact indirectly, e.g. “push” (GTS) Directory polling or file-system event monitoring  “Active” Sender/recipient are aware of each other and interact/transact directly E.g. BALTRAD

WMO GTS for radar data exchange? Advantages +Already exists +Active networking Disadvantage -Hierarchical organization causes delays For example: Exeter  Offenbach  Norrköping  Riga faster?

WMO BALTRAD for radar data exchange? Positives +Already exists +Active networking +WIS integration already demonstrated (2010) Challenges -Still being developed -Requires dedicated software -Not yet officially recognized as part of WIS

File format: GRIB +WMO standard +Already in use with national composite products in RA IV. +Demonstrated use for exchange. (GRIB1) -No single official, centrally maintained, software. -No mature information model representation.

File format: TITAN +Already in use in two regions (I and III) +Storm tracks also available, e.g. XML +Open software, centrally maintained +Efficient for transmission, production, exchange -Documentation -No DQ preparedness -Prepared for dual-pol moments? -Built-in compression? -Unknown outside meteorology

File format: BUFR +WMO standard. +Suitable for point and profile data. +Very flexible, “can encode anything”. +Demonstrated use in Region VI, ODIM_BUFR -“Unwieldy” for non-trivial data such as radar. -Many mutually incompatible software and data model implementations. -No single official, centrally maintained, software. -Bottleneck to academia. -Not a marketable skill. -Largely unknown outside meteorology.

File format: HDF5 +Central, open, modern software developed/maintained by HDF Group. +Bundled for various OSes. +Established in Earth Sciences. +ODIM_H5 prepared for dual-pol and DQ. +ODIM_H5 is “policed”. +Efficient for production and exchange. +ODIM_H5 compatibility between regions V and VI -Steep learning curve. -Currently unsuitable as transmission format. -ODIM_H5 not an orthodox information model. -Ambiguities in ODIM_H5 risk incompatibilities.

File format: netCDF, version 4 +Central, open, modern software developed/maintained by NCAR. +Bundled for various OSes. +Established in Earth Sciences through “CF Conventions”. +GIS-compliant. +“CF Radial” prepared for dual pol? -Requires HDF5, added abstraction layer. -No mature CF Convention yet for operational radar, although “CF Radial” for R&D. -Unprepared for DQ?

WMO ODIM in OPERA, February 2013 Input data Output products Green = ODIM_H5 Brown = ODIM_BUFR Yellow = Both

WMO Summary  Terminology needs clarifying. “Data model”, “exchange”, “file format”, etc.  Several file formats in use.  No orthodox information model as such in use. Not necessarily a disadvantage.  Tight connections between file formats and information models.  Good solutions exist already.

Thank you for your attention Daniel Michelson Swedish Meteorological and Hydrological Institute Norrköping, Sweden