1. ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production Harry Goossens – Statistics Netherlands Head Data Service Centre / ESSnet Coordinator hct.goossens@cbs.nl

2. ESS-net DWH 1 MEETS action 3.1: “make better use of data that already exist in the statistical system, including the possibility of estimates” Ultimate aim: “create fully integrated data sets for enterprise and trade statistics at micro level”  a 'data warehouse' approach to statistics General context

3. ESS-net DWH 2 ESS-net coordinator:  Statistics Netherlands (CBS) Co-partners:  Estonia, Italy, Lituania, Portugal, Sweden, UK\ Starting date:  4 October 2010  SGA 1: first year, till 3 October 2011  SGA 2: last 2 years, till 3 October 2013 Partnership (FPA)

4. ESS-net DWH 3 Broad definition for this ESSnet: A common conceptual model for all available data of interest, enabling the NSI: to (re)use this data to create new data/new outputs, to produce the necessary statistical information and to perform reporting and analysis, regardless of the data’s source. DWH Definition

5. ESS-net DWH 4 General objective ESS-net: Provide assistance in the development and implementation of a maximum efficient statistical process for business and trade statistics, independent of any (technical) specific architecture Results in daily statistical practice:  increase the efficiency of dataprocessing in statistical production systems  maximize the reuse of already collected data Objectives

6. ESS-net DWH 5  Review of current practices in integrated business data systems - realised solutions, ongoing developments - problems, desired solutions, opportunities  Examination of ways in which data can be combined to support new outputs - Role of Business Register, metadata  Provision of recommendations on how the ESS can improve data warehousing  Dissemination of results to all ESS countries (and broader) Main goals (at start)

7. ESS-net DWH 6 1.Stocktaking / Inventory of current situation in MS - several MS already introduced data warehousing; - important to make it possible to compile and compare current practices. 2.Analysis and Development - development of a functional model of the S-DWH; - model as basis to assess all issues, raised within this ESSnet, in a generic and standardized way. 3.Dissemination - sharing knowledge and expertise. Actions around 3 axes

8. ESS-net DWH 7 Activities mainly around axis 1 (stock taking)  What is already been done, experienced by MS ?  DWH is a very broad field and scope:  First strictly determine specific detailed subjects/topics to explore and study in depth. and axis 3 (disssemination)  Knowing and sharing what is going on  Finding out what the ESS needs / wants First SGA

9. ESS-net DWH 8 The Statistical Data Warehouse Aggregate Statistics Aggregate Statistics Microdata Data extracts Data extracts Data extracts Dataset Business register Selected samples Admin data source BR snapshots Storage, combination Outputs Input / Micro dataInput reference frame Staging area Working data Rules for generating samples etc Rules for updating BR

10. ESS-net DWH 9 Conclusions  Data Warehousing in statistics is ‘hot’  Metadata is found important….. but also often neglected !  S-DWH is very difficult to compare with common commercial DWH  Visiting NSIs has proven very effective for gathering information AND for sharing knowledge and expertise  Great need for knowledge & expertise First SGA

11. ESS-net DWH 10  24 months (till 10/2013 – end of MEETS)  3 Workshops planned  4 WP (+ management & coordination)  Interconnecting with other ESSnets  Keep overviews BP up-to-date  Continuation of visiting NSIs to support actively  Strong focus on dissemination Second SGA

12. ESS-net DWH 11  Building a framework which defines metadata requirements and roles in the S-DWH context  Study on the use of metadata models and standards: define the various functionalities of a metadata system to facilitate and support the operation of the S-DWH  Provide recommendations and guidelines on the governance of metadata management in the S-DWH WP 1 - Metadata

13. ESS-net DWH 12 WP 2 - Methodological topics Building the S-DWH: Methodological topics  Study and recommendations on the position and role of the Business Register (BR) in the context of a S-DWH  Examine selective editing options for the S-DWH and provide guidelines  Study and recommendations on the various ways of ‘linking data’, considering aspects of combining data (from surveys and administrative sources etc.)  Study on the impact of data confidentiality in micro data linking and data warehousing, considering confidentiality aspects of combining data from various sources etc.

14. ESS-net DWH 13 WP 3 - Technical aspects Building the S-DWH: Technical aspects  Provide a Business Architecture of the S-DWH  Produce a functional architecture of the S-DWH  Produce a modular workflow of the S-DWH  Relate the 'ideal' architectural schemes (frameworks) into an actual implementation and development strategy

15. ESS-net DWH 14 WP 4 - Dissemination  Keep the ‘stocktaking of best practices, problems encountered and desired solutions’ up-to-date  Create a glossary and keep it up-to-date  Create an overall report, containing all deliverables of this ESSnet (guidelines, recommendations, models etc.)  Create an overall handbook to set up the S-DWH  Establishing and implementing a ‘knowledge and expertise centre’ to manage the expertise and knowledge acquired in an active/dynamic manner  ensure sustainability after this ESSnet is completed

16. ESS-net DWH 15 Specific attention:  overlap, compatibility and interfacing with other ESSnets - sharing and using results  scoping, prioritising - keep the ESSnet manageable Project management