Explaining the statistical data warehouse (S-DWH)

Slides:



Advertisements
Similar presentations
The governance of metadata management in the S-DWH
Advertisements

ESSnet on Data Warehousing - WP2 Overview Amsterdam September 2013.
Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.
Best practice case: Comparing the implementations of the Irish CDM and the Dutch DSC ESSnet on microdata linking and data warehousing in statistical production.
United Nations Economic Commission for Europe Statistical Division Exploring the relationship between DDI, SDMX and the Generic Statistical Business Process.
Chapter 10: Analyzing Systems Using Data Dictionaries Instructor: Paul K Chen.
Pieter Vlag ESSnet DWH: business register. Outline Central role of the  statistical units,  population frame, which includes number of enterprises,
International Seminar on Modernizing Official Statistics:
Metadata for the S-DWH ‒ an overview Lars-Göran Lundell Statistics Sweden.
S-DWH Architecture (Recap):
United Nations Economic Commission for Europe Statistical Division Applying the GSBPM to Business Register Management Steven Vale UNECE
TOWARDS INTEROPERABLE STATISTICAL BUSINESS REGISTERS Harrie van der Ven Project manager ESSnet EGR January 2014 Valencia.
WP.5 - DDI-SDMX Integration E.S.S. cross-cutting project on Information Models and Standards Marco Pellegrino, Denis Grofils Eurostat METIS Work Session6-8.
Using ISO/IEC to Help with Metadata Management Problems Graeme Oakley Australian Bureau of Statistics.
M ETADATA OF NATIONAL STATISTICAL OFFICES B ELARUS, R USSIA AND K AZAKHSTAN Miroslava Brchanova, Moscow, October, 2014.
SDMX and DDI Working Together Technical Workshop 5-7 June 2013
Metadata for a Statistical Data Warehouse Lars-Göran Lundell Statistics Sweden Luxembourg 22 September 2011.
Development of metadata in the National Statistical Institute of Spain Work Session on Statistical Metadata Genève, 6-8 May-2013 Ana Isabel Sánchez-Luengo.
Deliverable 2.6: Selective Editing Hannah Finselbach 1 and Orietta Luzi 2 1 ONS, UK 2 ISTAT, Italy.
Metadata Models in Survey Computing Some Results of MetaNet – WG 2 METIS 2004, Geneva W. Grossmann University of Vienna.
ESS-net DWH ESSnet DWH - Metadata in the S-DWH Harry Goossens – Statistics Netherlands Head Data Service Centre / ESSnet Coordinator
1 Conclusions from Sessions 4, 5, 6 Rapporteurs: Donatella Fazio, Istat Maria Grazia Calza, Istat Arianna Carciotto, Istat ESSnet Workshop 2012 Cavour.
ESSnet on Datawarehousing - the business register Pieter Vlag – Statistics Netherlands.
Modernisation in Istat Nadia Mignolli Italian National Institute of Statistics (Istat) Department for Integration, Quality, Research and Production Networks.
Case Study Statistics Netherlands Max Booleman Statistics Netherlands METIS, 2010.
Jenny Linnerud, 27/10/2011, Cologne1 ESSnet CORE Common Reference Environment ESSnet workshop in Cologne 27th and 28th of October 2011.
ESSnet on microdata linking and data warehousing in statistical production: Metadata Quality in the Statistical Data Warehouse.
DWH Aggregate Statistics Aggregate Statistics Microdata Dataset Business register Storage, combination OutputsInput data 1.The magic data pixie model.
ESSnet Workshop Cologne 2011 ESSnet on measuring global value chains.
ESSnet ON MICRO DATA LINKING AND DATA WAREHOUSING IN STATISTICAL PRODUCTION RESULTS OF STOCKTAKING, CONCLUSIONS OF FIRST YEAR * Pieter Vlag Senior Statistical.
United Nations Economic Commission for Europe Statistical Division High-Level Group Achievements and Plans Steven Vale UNECE
Statistical Metadata Strategy and GSIM Implementation in Canada Statistics Canada.
1 1 Developing a framework for standardisation High-Level Seminar on Streamlining Statistical production Zlatibor, Serbia 6-7 July 2011 Rune Gløersen IT.
Work packages SGA II ESSnet on microdata linking and data warehousing in statistical production Harry Goossens – Statistics Netherlands Head Data Service.
Michelle Simard, Thérèse Lalor Statistics Canada CSPA Project Manager UNECE Work Session on Statistical Data Confidentiality Helsinki, October 2015 Confidentialized.
Eurostat SDMX and Global Standardisation Marco Pellegrino Eurostat, Statistical Office of the European Union Bangkok,
SDMX IT Tools Introduction
2.An overview of SDMX (What is SDMX? Part I) 1 Edward Cook Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015.
Modernization of official statistics Eric Hermouet Statistics Division, ESCAP
United Nations Oslo City Group on Energy Statistics OG7, Helsinki, Finland October 2012 ESCM Chapter 8: Data Quality and Meta Data 1.
ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production Harry Goossens – Statistics Netherlands Head Data Service Centre.
Metadata Framework for a Statistical Data Warehouse
1 Statistical business registers as a prerequisite for integrated economic statistics. By Olav Ljones Deputy Director General Statistics Norway
Generic Statistical Information Model (GSIM) Jenny Linnerud
11 Centre of knowledge and expertise Data Warehousing ESSnet (DWH ESSnet)
7b. SDMX practical use case: Census Hub
ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.
Harry Goossens Centre of Competence on Data Warehousing.
Best practice case Finland / Estonia 22th. of September 2011 Maia Ennok.
Describe a layered S-DWH Technology Architecture Information Systems Architecture Business Architecture.
Statistical process model Workshop in Ukraine October 2015 Karin Blix Quality coordinator
REPORTING SDG INDICATORS USING NATIONAL REPORTING PLATFORMS
Towards connecting geospatial information and statistical standards in statistical production: two cases from Statistics Finland Workshop on Integrating.
Contents Introducing the GSBPM Links to other standards
S-DWH layered architecture – Statiscs Finland
at Statistics Netherlands
Tomaž Špeh, Rudi Seljak Statistical Office of the Republic of Slovenia
Metadata in the modernization of statistical production at Statistics Canada Carmen Greenough June 2, 2014.
ESSnet on Data Warehousing 4th Workshop Maia Ennok 20th. of March 2013
Methodology Working Group Luxemburg
Applying the Generic Statistical Business Process Model to Business Register Maintenance Steven Vale UNECE
2. An overview of SDMX (What is SDMX? Part I)
Modernization of Statistical data processes
SDMX in the S-DWH Layered Architecture
Metadata The metadata contains
GSBPM and Data Life Cycle
Contents Introducing the GSBPM Links to other standards
Business architecture
ESTP course on Statistical Metadata – Introductory course
Presentation transcript:

Harry Goossens - ESSnet Coordinator ESSnet on microdata linking and data warehousing in statistical production The statistical data warehouse: a central datahub, integrating new datasources and statistical output Harry Goossens - ESSnet Coordinator Head Data Service Centre at Statistics Netherlands hct.goossens@cbs.nl UNECE - Seminar on New Frontiers for Data Collection Geneva, 31 October - 2 November 2012 ESS-net DWH

Explaining the statistical data warehouse (S-DWH) Content Background ESS-net Challenges Explaining the statistical data warehouse (S-DWH) Elements of the S-DWH Business architecture GSBPM mapping Meta data ESS-net DWH

ESSnet on microdata linking and data warehousing in statistical production ESS-net DWH

Statistics Netherlands (CBS) Co-partners: ESSnet Partnership ESS-net coordinator: Statistics Netherlands (CBS) Co-partners: Estonia, Italy, Lithuania, Portugal, Sweden, UK Starting date: 4 October 2010 SGA 1: first year, till 3 October 2011 SGA 2: last 2 years, till 3 October 2013 ESS-net DWH

General Objectives ESSnet DWH Provide assistance in: the development and implementation of a maximum efficient statistical process for business and trade statistics, independent of any (technical) specific architecture Results in daily statistical practice: increase the efficiency of data processing in statistical production systems maximize the reuse of already collected data a 'data warehouse' approach to statistics ESS-net DWH

Rapidly changing demand for information: The Challenges Decrease of costs & administrative burden versus increase of efficiency & flexibility Rapidly changing demand for information: growing need for more information on more topics decreasing lifecycle of policymakers, quicker delivery Disclosure of all new data sources coming from global use of modern technology Make optimal use of all available data sources (existing & new) ESS-net DWH

The Statistical Data Warehouse A central data hub to connect and integrate all available data sources, supporting statistical production AND data collection processes by providing: a detailed and correct overview/insight of all available data sources a framework for adequate data governance, including metadata management, confidentiality aspects and data authorisation flexible data storage and data exchange between processes access to registers sampling frames (BR, etc); A central ‘statistical data store’ for managing all available data of interest, regardles of its source, enabling the NSI to produce necessary information (= statistics !) and to (re)use available data to create new data / new outputs. ESS-net DWH

Data extracts Selected Dataset sample Data extracts Working data Rules for generating samples etc. Data extracts Selected sample Dataset Data extracts Selected sample Dataset Working data Aggregate Statistics Staging area Admin data source Dataset Aggregate Statistics Microdata Admin data source Backbones (BR eg.) BB snapshots Data extracts Rules for updating BB Input reference frame Input data Storage, combination Outputs ESS-net DWH

rules for updating the sources for the DWH Explaining the S-DWH A system or set of integrated systems, designed to handle the processing of statistical data in the production of statistics, comprimising: technical facilities for storing and processing data, receiving data in and producing outputs in a flexible way rules for updating the sources for the DWH definitions necessary to achieve those samples / sources The S-DWH is a concept that provides an architectural model of the statistical data flow, from data collection to statistical output ESS-net DWH

The S-DWH Business Architecture Conceptualisation of how to build up a S-DWH A common model for the total statistical process and data flow Provide optimal organisation of all structured data, enabling re-use, creation of new data etc. 4 Layers, covering all statistical activities Sources Integration Interpretation & Analysis Data Access / Output ESS-net DWH

The layered architecture of the S-DWH, with focus on the data sources used in each layer ESS-net DWH 10 10

Mapping the S-DWH on the GSBPM Use the GSBPM as common language to identify and locate the various phases on the 4 S-DWH layers ESS-net DWH

Metadata is vital in the governance, satisfying 2 essential needs: Managing the S-DWH The S-DWH is a logically coherent central data store, not necessarily one single physical unit. Metadata is vital in the governance, satisfying 2 essential needs: to guide statisticians in processing and controlling the statistical data to inform users by giving insight in the exact meaning of the statistical data The vertical metadata layer enables to search all (meta)data in the 4 layers and, if permitted, give access to the data. ESS-net DWH

Interpretation and Data Analysis Layer Meta data layer Metadata Layer Data Access Layer Interpretation and Data Analysis Layer Integration Layer Source Layer ESS-net DWH

14 Meta data - the DNA of the S-DWH Framework: General meta data definitions Meta data for the S-DWH Use of meta data models Meta data standards & norms Meta data quality & governance Categories & subsets Minimum requirements ESS-net DWH 14 14

Internal rules Guidelines S-DWH meta data requirements Subsets Standards & Norms More … Data models Authorization metadata Technical metadata Quality metadata Process metadata Statistical metadata ISO 11179 Internal rules Guidelines Mata data model S-DWH Gatekeeper ESS-net DWH 15 15

Organisational aspects Implementation of a S-DWH has huge organisational impact: It means: moving from single operations to integrated, generic processes It needs: a redesign of the statistical process It asks: new IT systems, tools, high investments It is: a new way of working Only changing systems will not do the trick, changing people is the key to success ESS-net DWH

ESSnet on data warehousing Thank you ! ESS-net DWH 17