Data Warehouse Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss The MeteoSwiss Data Warehouse Real-time access to data and metadata – basis for the WIS- implementation in Switzerland WMO RA VI Workshop on the implementation of WIS Sofia, 1-3 November 2011 Stephan Vogt, Meteorological Data Coordination, MeteoSwiss

2 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Agenda Situation in 1999 Towards the MeteoSwiss Data Warehouse Metadata in the DWH Ongoing improvements Considerations on the WIS-implementation

3 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Situation in 1999 – and the vision Year 2000 Problem: maximum credible data accident to be expected?

4 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Objectives of the MeteoSwiss Data Warehouse Implementation of a central, integrated data platform for the collection, storage and processing of meteorological data and its metadata. Storage of metadata in one central metadata repository Subdivision in a processing and an application area Creating a basis for a customized access to data respecting official standards for internal and external clients

5 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Phenology & Pollen 2002 Surface data (rain gauge measurements, AWS data, observations) 2004 Aviation messages 2005 Lightning data 2006 International surface data (SYNOP) 2007 National sounding measurements 2008 Homogenized Data 2009 National & International Upper Air Data 2010 Gridded Datasets Start of the MeteoSwiss Data Warehouse Project Model Data 2011 Metadata

6 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Conceptual Architecture Data sources Staging area Data storage Database for point data analysis Calc, Check, Data Transformation & Integration Data delivery and products (on-/offline) Weather Forecast Data for the Public Internet Durable data archive Database for point data processing (incl. raw data repository) Light- ning data Radar Clima Analysis Tools Model ValidationResearch ClimatologyClimate Information Service Data analysis Autom. Observ. systems Upper Air data Aviation data Geodatabase (raster data,...) Data analysis & Interfaces (API, DB Tools, Web access, GIS, R …) VERA CM SAF NWP T L Partner Observ. systems QC Kontextd aten Meta Data Obser- vations Metadata Management Tools

7 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Metadata in the MeteoSwiss DWH Spatial Reference Instrument Lifecycle Data Import Contract DataParameter Quality Control data Calculation and Aggregation Homogenization Other Context data …stored in more than 200 tables in the DWH

8 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Enhanced Data Clean Data customers Raw Data level 2 data level 1 data level 3 data level 5 data level 4 data quality control II calculation of derived quantities data collection quality control I measurement simulation metadata Raw data repository Data repository for external data applications for data distribution data source quality control III errorr correction extreme value messages immediately days years external data source applications for data processing immediately durable data archive MeteoSwiss data archives durable data archives outside MeteoSwiss MeteoSwiss data sources data sources outside MeteoSwiss data transmission and distribution Legend Level 1 measurement data Level 2 raw data Level 3 cleaned data Level 4 corrected data Level 5 homogenized data homogenization Instrument Lifecycle Spatial reference Data import Aggregation calculation Aggregation calculation Contract data Station history Homo- genization Quality control Parameters

9 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Development cycle & metadata management Development Deployment Production Failover Production Master Metadaten Repository

10 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss How are the metadata accessed? DWH-Webtool with user guidance for all important use cases MS ACCESS for all other cases RAUMBEZUG (Data Owner) SMN CONTRACT- DATA (KD/TM) TAMSI LISTEN (FW) PARAMETER (Data Owner oder MD) INSTRUMENT LIFE CYCLE (TM) APPLICATION (MD, IM, IT) IMPORT (Data Owner) STAGE (KD) CLIMAP BENUTZER (KD, MD) AGGREGIERUNG (Data Owner) THOMAS (KD) POSTAL CODE (MD) PLAUSIBILITAETS- TESTS (Data Owner) SENSOR TROUBLE (TM) APPLIKATIONS- STEUERUNG (AO) FeuOz (DA) Ozoncalc (DA)

11 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Look and Feel of DWH-Webtools

12 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Advantages of the Data Warehouse measurement data metadata control data quality tests  Allows an efficient and economic maintenance  company-wide data integration platform for the long term  independant from technical realisation specified in a uniform way, centrally administrated and configurable

13 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Lessons learned Do‘sDont‘s Listen to the data; conduct broad-based reviews Listen too much to single data users Follow a spiral development model, gain experience in operation, grow the infrastructure accordingly Do everything in one step  waterfall procedure Follow the golden rules of database design Fix technology too early Understand the business process Don‘t let consultants talk you into a solution Provide one attractive application for data analysis from the very beginning Segregated architectures

14 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Do things the same way all over the organisation  data governance Data architecture documents Gremium with representatives from all MeteoSwiss departements

15 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Not everything is solved... RENAISSANCE = Renewal and ENhAncement of the servIceS baSed oN the mEteoswiss data warehouse Subprojects: Data supply of COSMO from DWH Generic delivery mechanism for data marts Station history Consolidation of production tools

16 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Generic delivery mechanism for data marts Existing delivery mechanisms have been generalized Towards event-based triggering of processes Generic delivery mechanism includes Measurement data Metadata  based on XML standard  Primary purpose: Data delivery to GIN (Swiss joint Information Platform for Natural Hazards)

17 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Lots of tools for lots of products…

18 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss …and what about WIS? All remarks so far refer to our status as NC MeteoSwiss has been asked to develop GAWSIS as a DCPC for WIS Today GAWSIS is not yet integrated into the DWH Two options for the next steps: solution on the basis of the current GAWSIS  Provide the data with an OAI-Service  ready for collection by GISC Integration of GAWSIS into the DWH, make use of the planned Web Service-infrastructure

19 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Vision regarding WIS

20 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Vision regarding WIS

21 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Questions? Thanks for your attention!

22 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Implementation ProductionFailover Production (hot standby) HW Sun SPARC Enterprise M5000 CPU: 2 SPARC 64 VII 4-core RAM: 32GB Disk: 3 TB increase: 320 GB/Jahr Sun SPARC Enterprise M5000 CPU: 1 SPARC 64 VII 4-core RAM: 16GB Disk: 250 GB OS Solaris10 RDBMS Oracle10g Oracle10g (cold standby) ETL Informatica PowerCenter 8.6.1Informatica PowerCenter (cold standby) Stored data 1755 – today + 10 days Last 60 days Max downtime Max 15‘ per interrupt on Production system (8 days operation possible with Failover Production possible)

23 MeteoSwiss Data Warehouse | Real-time access to data and metadata – basis for the WIS-implementation in Switzerland Stephan Vogt Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Some figures more than 70 MeteoSwiss collaborators with ~17000 working days over the last 10 years External costs: 18.6 Mio € 7 Billion data records have been migrated from other data sources. 10 Mio data records added every data (increasing)