Workshop on ESA 2010 transmission programme – What and how? EUROSTAT: Julio Cabeca Fabien Jacquet Silke Stapel-Weber Daniel Suranyi Hans Wouters

SDMX-NA Workshop – Part B SDMX-NA Project overview SDMX – how do I get there? Demonstrations SDMX Converter SDMX Reference Infrastructure (SDMX-RI) SDMX-RI Mapping Assistant SDMX-RI test client (PUSH Mode) SDMX-RI web client (PUSH and PULL Mode) Questions & Answers

The SDMX Components SDMX Information Model Content Oriented Guidelines IT Infrastructure for exchange and sharing Content Oriented Guidelines SDMX Information Model 3 3

The SDMX Components Describe statistics in a standard way Objects and their relationships Data Structure Definition (DSD), Concepts, Code List Central management and standard access SDMX Registry, SDMX Web Services Cross Domain Concepts Cross Domain Code Lists Statistical Domains Metadata Common Vocabulary Push Provider generates and sends file to receiver Pull Provider opens web service to data Receiver downloads regularly Hub Special case of pull: receiver downloads on end user request 4 4

Timeline Preparation Q3/2011 - Q1/2012 Governance, Organisation, Planning Development Q4/2011 - Q2/2013 Data Flows, Concepts, Code Lists DSDs and other artefacts Consultations Q3/2012 - Q1/2013 NA and IT expert groups SDMX Sponsors & working groups Implementation Q4/2012 - Q2/2014 Pilots 1+2, content review, technical review Release Adaption of production systems Production Sept 2014 Maintenance Regular Production Item 5.4 DMES 14/3/13

State of play and next steps Content review (concepts, code lists, DSD matrix) 12/2012 - 02/2013 Follow-up on content review Development of DSD and other artefacts Development of EXCEL visualisation templates Q1/2013 Technical review phase (DSD, other artefacts, end to end tests) 04 - 07/2013 Follow-up on technical review Release of DSDs and associated artefacts Start of implementation (NSI, NB, ESTAT, other IO) Summer 2013 SDMX Global Conference Paris, France 11-13. September 2013 Item 5.4 DMES 14/3/13

Technical Review Phase – Contents Pilot package (April) DSD Guidelines DSD Matrix Pilot 1 issue log Questionnaire Samples for table 1 Excel package (May) Samples for most tables Related SDMX-ML files Tables missing (GFS): 2, 9, 11, 25, 27, 28 Excel 2 SDMX converter module, pilot version (on demand)

Introduction: data transmission task SDMX MS: NA data production environment EXCEL DB national coding and DS NA data transmission environment Manual Macros matching with ESA codes and DS ESTAT: single entry point Different production environ-ments ESA codes and DS

One goal – different possibilites Excel sheets SDMX Converter SDMX-ML file Database export Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service

Always needed: Data Structure Definitions (DSDs) DSD ID DSD Title Tables in transmission programs from 2014 MAIN Main aggregates ESA / OECD: 1, 3, 5, 20, 22 SEC Sector accounts and government finance ESA / OECD: 2, 6, 7, 8, 9,11, 26, 27, 28 ECB: MUFA, GFS EDP: 1, 2, 3, 4 OECD: 119, 7HH, 7II, PSD G20: Recommendation 15 SU Supply / use and input / output statistics ESA : 15, 16, 17, 18, 19 REG Regional statistics ESA : 10, 12, 13 PENS Pension schemes ESA: 29

SDMX-NA pilot registry & sandbox http://na.sdmxregistry.org Browse code lists Download DSDs Pilot sandbox http://na.sdmxsandbox.org Structural validation CSV, Gesmes, SDMX Data visualisation

SDMX-NA Workshop – Part B SDMX-NA Project overview SDMX – how do I get there? Demonstrations SDMX Converter SDMX Reference Infrastructure (SDMX-RI) SDMX-RI Mapping Assistant SDMX-RI test client (PUSH Mode) SDMX-RI web client (PUSH and PULL Mode) Questions & Answers

SDMX-NA Workshop – Part B SDMX-NA Project overview SDMX – how do I get there? Demonstrations SDMX Converter SDMX Reference Infrastructure (SDMX-RI) SDMX-RI Mapping Assistant SDMX-RI test client (PUSH Mode) SDMX-RI web client (PUSH and PULL Mode) Questions & Answers

SDMX Converter https://webgate. ec. europa. eu/fpfis/mwikis/sdmx/index File based conversion Open source and platform independend (Java) Different ways of using it: Graphical user interface Batch file (server or client side) Web Service interface Reusing source code in your own Java application Formats: SDMX-ML 2.0, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, XLS, XLSX (2014)

SDMX-NA Workshop – Part B SDMX-NA Project overview SDMX – how do I get there? Demonstrations SDMX Converter SDMX Reference Infrastructure (SDMX-RI) SDMX-RI Mapping Assistant SDMX-RI test client (PUSH Mode) SDMX-RI web client (PUSH and PULL Mode) Questions & Answers

SDMX-NA Workshop – Part B SDMX-NA Project overview SDMX – how do I get there? Demonstrations SDMX Converter SDMX Reference Infrastructure (SDMX-RI) SDMX-RI Mapping Assistant SDMX-RI test client (PUSH Mode) SDMX-RI web client (PUSH and PULL Mode) Questions & Answers

SDMX Reference Infrastructure https://webgate. ec. europa Data Provider Data Collector SDMX Registry Mapping Assistant DSD Web Svc Web Client Test Client Non-SDMX local database SDMX data set SDMX-RI

SDMX Reference Infrastructure https://webgate. ec. europa Data Provider Data Collector SDMX Registry Mapping Assistant DSD Web Svc Web Client Test Client Non-SDMX local database SDMX data set SDMX-RI

SDMX Reference Infrastructure https://webgate. ec. europa Data Provider Data Collector SDMX Registry Mapping Assistant DSD Web Svc Web Client Test Client Non-SDMX local database SDMX data set SDMX-RI

SDMX-NA Workshop – Part B SDMX-NA Project overview SDMX – how do I get there? Demonstrations SDMX Converter SDMX Reference Infrastructure (SDMX-RI) SDMX-RI Mapping Assistant SDMX-RI test client (PUSH Mode) SDMX-RI web client (PUSH and PULL Mode) Questions & Answers

SDMX Tools – Summary http://www.sdmxtools.org Eurostat tools  Eurostat SDMX Info Space SDMX Converter SDMX Reference Infrastructure & Mapping Assistant SDMX Registry & Data Structure Wizard Other tools availble in the community Fusion Weaver (validation, transformation, creation) Fusion Transformer (command line conversion) Fusion Registry ECB Checker/Loader (conversion and validation) FAO OpenSDMX project

SDMX-NA Workshop – Part B SDMX-NA Project overview SDMX – how do I get there? Demonstrations SDMX Converter SDMX Reference Infrastructure (SDMX-RI) SDMX-RI Mapping Assistant SDMX-RI test client (PUSH Mode) SDMX-RI web client (PUSH and PULL Mode) Questions & Answers

Questions & Answers HELP – We do not have a database! No worries. Excel templates are available Conversion will be supported through the SDMX converter. However: use the opportunity to start!

? Quesions & Answers I want to have a database. What should I do? SDMX-NA ?

Questions & Answers When do we need to start sending SDMX-ML? For which tables? When do we need to start using PULL / HUB mode? For Eurostat: start with ESA2010 (September 2014) All tables are covered Only PUSH is required ESS: PULL may be implemented voluntarily HUB is not planned in National Accounts

Questions & Answers Which versions / formats should we use? SDMX-EDI (Gesmes/TS) supported, deprecated SDMX-ML 2.0 (compact) preferred SDMX-ML 2.1 (structure specific) in preparation

Questions & Answers Where will the DSDs be managed? When will we have the final versions? What will happen to the sandbox? DSDs will be managed in the global SDMX registry Final versions will be released in September 2013 The sandbox is only available for the pilot phase Validation will be managed by receiving organisations

Questions & no Answers yet Some coding details still subject to change Usage of conditional (metadata) attributes Usage of sibling group Data flows Final number of data flows Update/Revision or Replace mode Constraints linked to data flows Will be included in final DSD Guidelines

Questions? SDMX Support Team SDMX Website Eurostat SDMX Info Space ESTAT-Support-SDMX@ec.europa.eu SDMX Website http://www.sdmx.org Eurostat SDMX Info Space https://webgate.ec.europa.eu/fpfis/mwikis/sdmx SDMX-NA pilot project page http://tinyurl.com/sdmxna SDMX Global Conference Paris, France 11-13. September 2013