Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 The European Census Hub Changing the Business Model Adam Wroński, Marco Pellegrino, Nadezhda Vlahova, Bengt-Åke Lindblad Eurostat, “Quality, Methodology and Information Systems” Directorate

2 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 The European Census Hub: key issues Dissemination of the data from the 2011 population and housing censuses in the European Union Massive amount of data produced and controlled by Member States Comparable data, structured according to agreed “hypercubes”: harmonised concepts and definitions (Census Regulation) Easy access to detailed census data (advanced functionalities) Easy to use: tabulation of data from different sources High accessibility of data and metadata

3 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 EU Census: Implementing measures Regulation (EC) 763/2008 on population and housing censuses authorises the European Commission to adopt implementing measures on: Regulation (EC) 763/2008 –technical specifications of the topics and their breakdown (Regulation (EC) 1201/2009)Regulation (EC) 1201/2009 –programme of the statistical data and metadata to be transmitted to Eurostat (Regulation (EU) 519/2010)Regulation (EU) 519/2010 –quality reporting and technical format of data transmission (Regulation (EU) 1151/2010)Regulation (EU) 1151/2010

4 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 Article 6 of Regulation (EU) 1151/2010 –Member States shall transmit the required data conforming to the data structure definitions and related technical specifications provided by the Commission (Eurostat) –The technical format to be used for the transmission of data and metadata for the reference year 2011 shall be the Statistical Data and Metadata eXchange (SDMX) format –Member States shall store until 1 January 2025 the required data and metadata for any later transmission requested by the Commission (Eurostat) Technical format for data transmission

5 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 A system of 60 Data Structure Definitions 60 DSDs built on SDMX 2.0 (standard concepts and codes) and in use for 31 countries of the European Statistical System SDMX data message (cross-sectional format) SDMX query message Tests on-going: started with dummy data and sample hypercube, now to continue with real hypercubes Infrastructure to be maintained (and data stored) at least until 2025

6 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 Population topics Sex SEX AgeAGE Legal marital statusLMS Country/place of birth POB Country of citizenship COC Place of usual residence - one year prior to the censusROY (Size of the) LocalityLOC Household statusHST Type of private householdTPH Size of private householdSPH Family statusFST Type of family nucleusTFN Size of family nucleusSFN Topics required for all geographical levels down to Local Administrative Units (LAU = municipalities) Housing topics Occupancy status of conventional dwellingsOCS Number of occupantsNOC Useful floor space and/or Number of roomsUFS/NOR Density standardDFS/DRM Dwellings by type of buildingTOB Dwellings by period of constructionPOC Type of living quartersTLQ Which data for which geographical area?

7 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 Topics required for aggregated geographical levels NUTS 2, NUTS 1 and nation Population topics Year of arrival in the country YAE / YAT Educational attainmentEDU Location of place of workLPW Current activity status CAS OccupationOCC Industry IND Status in employmentSIE Tenure status of households TSH Housing topics Housing arrangements HAR Type of ownership (of dwellings) OWS Water supply system WSS Toilet facilities TOI Bathing facilities BAT Type of heating TOH Which data for which geographical area?

8 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 Example: DSD for Table 6 (Marital Status) IDCONCEPTCODELIST TIMETime period or rangeCL_TIME GEOGeographical areaCL_GEO SEXSexCL_SEX FSTFamily statusCL_FST LMSLegal marital statusCL_LMS CASCurrent activity statusCL_CAS POBCountry/place of birthCL_POB COCCountry of citizenshipCL_COC AGEAgeCL_AGE FREQFrequencyCL_FREQ IDATTACHMENT LEVEL CODELIST OBS_STATUSObservationCL_OBS_STATUS OBS_LEVELObservationCL_OBS_LEVEL OBS_NOTEObservation HC_NOTESeries IDNAME OBS_VALUEObservation value Dimensions Measures Attributes

9 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 The Hub concept  The Hub is based on the concept of data sharing, where a group of partners agree on providing access to their data according to standard processes, formats and technologies.  The SDMX Hub approach offers several advantages: –decoupling of NSIs' systems from the central hub via standard formats and techniques for the exchange; –NSIs free to provide more information than what is contained in the agreed hypercubes without additional effort; –Limited investment, re-usability (with the advantage of using recognized international standards). –SDMX-RI is furnished by Eurostat.

10 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 IT Architectures for Data Exchange

11 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 Data Repository (Warehousing) Architecture NSI Eurostat Pull Requestor eDAMIS Data Input SDMX Registry Intermediate storage Verification / Conversion To SDMX Received data in SDMX-ML Loader register Warehouse storage Eurobase query Dissemination XSL for SDMX-ML PULLPULL PUSHPUSH

12 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 The SDMX Hub architecture Pull mode and data sharing Hub SDMX-RI infrastructure

13 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011

14 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 Through the SDMX Hub, a data user can… Browse the Hub to define a dataset of interest, navigating via structural metadata: - Select hypercube or search by topic (filters) - Select data (level of detail, breakdowns) - Select layout (axes) View a table Save a query Export a file (CSV, Excel, SDMX-ML)

15 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 How the Hub works Eurostat Census Hub National Statistical Institute

16 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October

17 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 Where are we - ESS? Countries integrated within the Hub (19) (end 2011) Czech Republic, Germany, Ireland, Italy, Luxembourg, Poland, Portugal, Spain, Norway, Sweden, Slovenia, Bulgaria, Belgium, Latvia, Malta, Kypros, Hungary (11/2012), Finland(11/2012), France* Expressing their interest or working on it, but not yet in the Hub (6) Estonia, Greece, Lithuania, Netherlands, Switzerland, United Kingdom Asking for more information (6) Austria, Denmark, Iceland, Liechtenstein, Romania, Slovakia 1-2 June 2010: 1 st Census IT technical WG1 st Census IT technical WG February 2011: 2 nd Census IT technical WG2 nd Census IT technical WG 7-8 December 2011: 3 rd Census IT technical WG3 rd Census IT technical WG SDMX training, technical meetings

18 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 How to know more CIRCA - > Eurostat - > X-DIS Census Hub -> Census Hub public documents CIRCA - > Eurostat - > X-DIS Census Hub -> Census Hub public documents Read the experiences from the MSs that have already implemented the infrastructure In case of need, contact Eurostat unit B5 for technical advice –SDMX training available –Technical bilateral meeting can be arranged upon request

19 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 How to know more (2) The Implementation Kit SDMX Structure files –ConceptsConcepts –Codelists (excluded GEO)Codelists –Keyfamilies (one for each hypercube)Keyfamilies –Partial Geo Codelist SDMX data message example (cross-sectional), SDMX Query message example and Dummy hypercube in CSV format SDMX data message example (cross-sectional), SDMX Query message example and Dummy hypercube in CSV format MIG XML schema (XSD file) Explanatory notes Census Hub WS Implementation Guideline Census Hub WS Security Guideline Census 2011 Regulation Census 2011 Explanatory notes

20 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 Census Hub environment

21 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 Lesson learnt and benefits in participating Statistical needs first. Then, technological aspects. Capacity-building is a must –Participating organisations are gaining a good in-house experience in SDMX and its implementation A system of distributed databases is harmonised through the use of SDMX standards and content guidelines SDMX-RI can be reused for sharing data in other domains –Limited cost for installations, development costs can be reduced –Step forward towards generic solutions for statistical domains

22 Eurostat Unit B5 – Statistical Information Technologies SDMX Basics Training – October 2011 For more information: Unit B5, Section «Standardisation and Advanced IT for Statistics» Census Hub on CIRCA: