New ways in Big Data Management for NWP Dr. Dieter Schröder, Dr. Jürgen Seib and Dr. Jochen Dibbern Deutscher Wetterdienst Frankfurter Straße 135 D-63067 Offenbach, Germany

Data Volume – How Big is Big Gigabyte - 210*3 Terabyte - 210*4 Petabyte - 210*5 Exabyte - 210*6 Zettabyte - 210*7 Yottabyte - 210*8 Brontobyte* - 210*9 Gegobyte* - 210*10 *This terminology is still subject to change.

Why is a new management for NWP data needed? Increase of remote sensing data Higher resolution of NWP models Probabilistic vs. Deterministic forecasts Multi-variable data analysis in now-casting systems

How fast can we read a Petabyte? Read speed of a storage disc: 100 Megabyte per second (MBps) Bytes Second Hour Month MB 1.048.576 0,01 GB 1.073.741.824 10,24 TB 1.099.511.627.776 10.485,76 2,91 PB 1.125.899.906.842.624 10.737.418,24 2982,62 > 4

File-based Management File-Distribution-System Application Grib-Management-System Main memory Disc storage /tmp Input store Grib files Grib field store

Grib-Management-System Database-oriented Management Database Application Grib-Management-System DB-Mirror Application DB-Mirror Application DB-Mirror Application Main memory Disc storage Grib files Grib field store

Grib Data Management System Type of Grib data store: File-based Database-oriented Smallest access unit: Grib file Grib field Grib value at grid point

Goals Database-oriented management of Grib data such that The smallest access unit will be the Grib value at a grid point The size of the database will not be bigger than the Grib store of a file-based Grib-Management-System The insert of Grib data into the database is not slower than the insert into a file-based Grib-Management-System Requests for spatial-temporal analytics can be formulated with SQL The database is also able to store vector data (polygons, lines, points, etc.) in order to have a common store for all types of meteorological data

Snowflake model grid point forecast step runtime forecast value ensemble member level

GRIB_DATA table Coordinate values Grib values Grid point Runtime Forecast step Level Ensemble member Value 1 2016-11-02 2 123 4 86 3 99 255 2016-11-03 155 5 33 6 145 7 16 12 Coordinate values have to be stored for each Grib value More storage needed for the coordinates than for the Grib values

Revised relational data model Grid Id integer Point geometry ….  Queries will be slow with classic row store database systems

Row storage Tables Tablespace select column1 from tab

Column storage Tables Tablespace select column1 from tab

PoC Hardware Environment Fujitsu Server PRIMERGY RX900 S2 with 8 sockets / 80 cores / 160 threads Processor: Intel® Xeon® processor E7-8860 @ 2.27GHz Memory: 2 TB of RAM 3298.5 GB of SAN storage from a NetApp filer over Emulex Corporation Saturn: LightPulse Fibre Channel Host Adapter (rev 03) @ 8Gbit

PoC Dataset All forecast data of the ensemble prediction system COSMO-DE-EPS for one day Forecast range: 27 h / 45 h Forecast runs at: 00, 03, 06, 09, 12, 15, 18, 21 UTC Ensemble members: 20 Multi-level parameters: 9 on 50 vertical levels Single-level parameters: 101 Mesh size: 2.8 km (421 * 461 grid points) 4498 Grib2 files per day where each file contains 551 grib fields Size: 928 GB

Storage overview in SAP Hana 719 GB

Sample query 1 Get the predicted minimum, maximum and average values of the 2m temperature within a given area.

Sample query 2 Get the probability at each grid point that it will be warmer than a given threshold value. The calculation should be based on the results of two forecast runs. select point_id, 100 * count(value) / 40 from t_2m where (runtime = ? or runtime = ?) and forecasttime = ? and value > ? group by point_id order by point_id

Summary Analysis of Grib fields with a database system is feasible The size of the database will not be bigger than the Grib store of a file-based Grib-Management-System Time for database import of Grib data needs further optimisation Requests for spatial-temporal analytics of both vector and raster data can be formulated with SQL Relocation of meteorological analysis functionality into the database Full advantage of database features, e.g. replication, persistence, query optimisation, parallelisation, concurrent access, geo-spatial extensions, etc.