ESS-net DWH ESSnet DWH - Metadata in the S-DWH Harry Goossens – Statistics Netherlands Head Data Service Centre / ESSnet Coordinator

ESS-net DWH 1 No NSI answers ‘YES’ on all these four questions:  Do you have a single coherent system which covers most of your data in the production of business statistics ?  Is your metadata currently integrated into your data systems ?  Is your data input for current needs integrated into your data systems ?  Are your current output requirements integrated into your data systems ?  No NSI has a finished DWH and metadata system Questionnaire stocktaking

ESS-net DWH 2 Overall daily practice:  All NSI’s find metadata (highly) important  Mostly NO metadata systems operational (yet some in development)  Most NSI’s struggle with metadata  Often capacity problem, ‘extra work’  Need for guidance on metadata Conclusion Stocktaking

ESS-net DWH 3 Metadata definitions Data & Metadata  Data are qualitative or quantitative information collected through observation  Metadata are data about / describing data. Statistical Data & Metadata  Statistical data are data from surveys and/or administrative sources, used to produce statistics  Statistical metadata are data about / describing statistical data or better: about STATISTICS

ESS-net DWH 4 Metadata for a DWH Technical metadata  Structural information How to physically find and use logical data  Process descriptions How data flows in the DWH  Authentication rules Who may do what ? Business metadata  Definitions and descriptions Help the end-user interpret and evaluate the data

ESS-net DWH 5 Metadata for statistics production Structural metadata  Act as identifiers and descriptors of the data: Identify, use, and process data matrixes and data cubes Names of databases, columns, dimensions Reference metadata  Describe the contents and the quality of the data: Include conceptual, methodological and quality metadata Algorithms, definitions, Q-indicators Source: METIS

ESS-net DWH 6 Metadata categories A metadata item is either  Structural (technical) or Reference (business) Other mutually exclusive categories:  active  passive  structured  free - form  standardised  non standardised  centralised  local

ESS-net DWH 7 The Statistical DWH Data Warehouse Statistics production Statistical Data Warehouse A central ‘statistical data store’ for managing all available data of interest, regardles of its source, enabling the NSI to: - produce necessary information (= statistics !) - (re)use available data to create new data / new outputs - execute analysis and perform reporting

ESS-net DWH 8 Metadata for a S-DWH Emphasis / focus on:  Active, Structural and Structured metadata  Reference metadata (common to all statistics production) and  Process metadata Describe expected or actual outcome of one or more processes using evaluable and operational metrics  Quality metadata Source quality, methods used, usability/restrictions  Tracing information Which surveys/registers contributed to a specific output ?

ESS-net DWH 9 Metadata standards in a S-DWH  What should be standardised ? Contents, formats, repository, software  Which level of standards should be used ? International/Eurostat, National/NSI, DWH internal  How should a standard be interpreted ? Complete adherence, compatible  How strict adherence should be required ? Mandatory, recommended  Should some components be prioritised ? Big bang, evolution

ESS-net DWH 10 Metadata Quality  The more data, the more need for metadata  The S-DWH contains lots of data, making it dependent on its metadata  Correct, high-quality metadata are vital for its use and for metadata governance: - No metadata  useless data - Bad metadata  misused data - Good metadata  useful data

ESS-net DWH 11 Metadata as a design tool  Metadata is a complex issue and central to the concept and implementation of the data warehouse. The Project needs to consider how guidance may/can be given to ensure that metadata systems allow all the gains of the S-DWH to be exploited effectively.  Metadata has a role to play in the abstract design process, independent of any specific structure. The S-DWH model has implications for the way metadata is collected, transmitted and used. If this is the case: the process design could be determined entirely within the metadata requirements and provide automatic consistency between technical architecture and metadata needs

ESS-net DWH 12 Fitting S-DWH in current metadatamodels and standards:  Building a framework which defines metadata requirements and roles in the S-DWH context  Study on the use of metadata models and standards: define the various functionalities of a metadata system to facilitate and support the operation of the S-DWH  Provide recommendations and guidelines on the governance of metadata management in the S-DWH  Keep it manageable & practical !!! SGA II: WP 1 - Metadata