Presentation is loading. Please wait.

Presentation is loading. Please wait.

7c. SDMX practical use case: National Accounts

Published byMuriel Alexia Gibson Modified over 9 years ago

Similar presentations

Presentation on theme: "7c. SDMX practical use case: National Accounts"— Presentation transcript:

1 7c. SDMX practical use case: National Accounts
Daniel Suranyi and Cristina Blanaru Eurostat SDMX Basics course, October 2015

2 Content Projects around SDMX and National Accounts
Governance, maintenance, sharing Implementation in NA Business process re-engineering Structural and content validation Interational data sharing National perspective Working with the DSD Matric Working with the SDMX-MES DSDs Architecture & tools, validation demonstration

3 National process Why? : The business case
I want to… … structure my statistical datasets … exchange data with other organisations … align my definitions internationally using standards … validate that data is correct

4 The big picture: ongoing activities
ESS.VIP Validation ESS.VIP SERV Eurostat NAPS-S Service Oriented Architecture for National Accounts SDMX-VTL Validation & Transformation Language SDMX-MES Ownership Group on SDMX for Macroeconomic Statistics TFIDC Task Force International Data Sharing

5 The big picture: ongoing activities
ESS.VIP Validation ESS.VIP SERV Eurostat NAPS-S Service Oriented Architecture for National Accounts SDMX-VTL Validation & Transformation Language SDMX-MES Ownership Group on SDMX for Macroeconomic Statistics TFIDC Task Force International Data Sharing

6 SDMX-MES Ownership Group (OG)
SDMX for Macro-Economic Statistics Mandated by SDMX Sponsors in January 2014 As a single OG for NA & GFS, BOP and FDI Maintenance: NA/GFS ( and BOP/FDI ( New domains can be added (e.g. Prices) Representation of management from content and SDMX side Oversees maintenance of SDMX artefacts Mandated technical group(s)

7 Cross Domain Concepts and Code Lists Shared Concepts and Code Lists
National Accounts SNA 2008 ESA 2010 Other NA related domains FDI statistics Other domains Balance of Payments Statistics Cross Domain Concepts and Code Lists Shared Concepts and Code Lists

8 Concepts in SDMX-MES DSDs
1 FREQ 2 ADJUSTMENT 3 REF_AREA 4 COUNTERPART_AREA 5 REF_SECTOR 6 COUNTERPART_SECTOR 7 CONSOLIDATION 8 ACCOUNTING_ENTRY 9 STO 10 INSTR_ASSET 11 MATURITY 12 ACTIVITY 13 ACTIVITY_TO 14 PRODUCT 15 PRODUCT_TO 16 EXPENDITURE 17 PENSION_FUNDTYPE 19 UNIT_MEASURE 20 CURRENCY_DENOM 21 VALUATION 22 PRICES 23 TRANSFORMATION 24 CUST_BREAKDOWN Concepts Cross Domain Shared NA specific Obs OBS_STATUS CONF_STATUS EMBARGO_DATE Ser DECIMALS TABLE_IDENTIFIER UNIT_MULT COMPILING_ORG REF_YEAR_PRICE Dset COMMENT_DSET …

9 Coding in SDMX-MES DSDs
1 FREQ 2 ADJUSTMENT 3 REF_AREA 4 COUNTERPART_AREA 5 REF_SECTOR 6 COUNTERPART_SECTOR 7 CONSOLIDATION 8 ACCOUNTING_ENTRY 9 STO 10 INSTR_ASSET 11 MATURITY 12 ACTIVITY 13 ACTIVITY_TO 14 PRODUCT 15 PRODUCT_TO 16 EXPENDITURE 17 PENSION_FUNDTYPE 19 UNIT_MEASURE 20 CURRENCY_DENOM 21 VALUATION 22 PRICES 23 TRANSFORMATION 24 CUST_BREAKDOWN Concept CL Maintenance Cross-Domain SDMX IMF/ESTAT no CD code list IMF/ESTAT code list extended Shared/Specific IMF/ESTAT Not coded - Obs OBS_STATUS CONF_STATUS EMBARGO_DATE Ser DECIMALS TABLE_IDENTIFIER UNIT_MULT COMPILING_ORG REF_YEAR_PRICE Dset COMMENT_DSET …

10 Data Structure Definitions (DSDs)
DSD ID NA_... DSD Title Dataflow defintions MAIN Main aggregates ESA / OECD: 1, 3, 5, 20, 22 SEC Sector accounts and government finance ESA / OECD: 2, 6, 7, 8, 9, 11, 26, 27, 28 ECB: MUFA, GFS EDP: 1, 2, 3, 4 OECD: 119, 7HH, 7II, PSD G20: Recommendation 15 SU Supply / use and input / output ESA : 15, 16, 17, 18, 19 REG Regional statistics ESA: 10, 12, 13 PENS Pension schemes ESA: 29

11 The big picture: ongoing activities
ESS.VIP Validation ESS.VIP SERV Eurostat NAPS-S Service Oriented Architecture for National Accounts SDMX-VTL Validation & Transformation Language SDMX-MES Ownership Group on SDMX for Macroeconomic Statistics TFIDC Task Force International Data Sharing

12 SDMX Validation Possibilities
“Technical” - Covered by SDMX today - Format Check (SDMX-ML) - Codes exist (SDMX DSD) - Codes used correctly (SDMX Dataflow & Constraint) “Statistical Domain” - Not yet covered by SDMX (VTL project) - Value check - Time series - Revisions - Validation expressions

13 What is VTL A standard language Main goals:
Considers the validation as a kind of Transformation (calculation) For defining validation and transformation rules Validation (now) Transformation (partially now, to be enriched at a later stage) Main goals: Define and preserve validation and transformation rules Exchange and share rules Apply rules in industrialized processes Apply to several standards (e.g. SDMX, DDI, GSIM) thanks to a generic information model

14 VTL features (4) VTL 1.0 allows: Persistent and temporary results
Operations on mono and multi measure data Dealing with missing data Dealing with Attributes and their propagation rules VTL 1.1 will introduce: Other operators, mainly for validation purposes Reusable rules Bug fixing, fine tuning

15 Summary: Current ESA business process
National Production Generate SDMX-ML PUSH Transmission Structural Validation Content Validation Processing & Calculation Data Sharing & Dissemination Data provider Eurostat Structural Validation

16 Possible future architecture (NAPS-S)
Shared / Replicated Services For NA and others Structural Validation Content Validation Statistical Service A Statistical Service B SDMX Registry VTL Repository Common Repository Common Repository

17 Structural Validation
Proof of concept setup Regular production All countries EDAMIS Selected TF members Eurostat Process Manager Step 1: Call Structual Validation Svc If OK: deliver to "reduced" production system If Not OK: deliver report to EDAMIS feedback channel Structural Validation Svc PoC

18 ? Structural Validation SDMX Registry Content Validation
SDMX compliance Valid SDMX-ML file Coded according to the DSD Mandatory fields present Correct data types Dataflow definition Basic logical checks Sender ID and REF_AREA Table ID is present Value "NaN" and OBS_STATUS EMBARGO_DATE and CONF_STATUS PRICES and REF_YEAR_PRICE Structural Validation SDMX Registry General plausibility and consistency (within file) Additivity of breakdowns Outliers Consistency between prices Unadjusted and adjusted series Basic content checks Missing or unexpected series Hole in series Zero values Negative values ? Content Validation VTL Repository Advanced plausability and consistency (across files) Revisions Quarterly versus Annual Same series across tables Cross-domain or source checks Balance of Payments Trade statistics Labour market statistics Data pulished by NSI or IO

19 SDMX Dataflow defintion
1 DSD covers multiple Dataflows NA_MAIN DSD: Table 1: T0101_A, T0102_A, T0103_A … Table 3: … Table 5: … … Each of those Dataflows gives a partial picture "cutting the DSD": SDMX cube region "listing each series": SDMX key set

20 SDMX Information Model - Summary
Reference DSD Reference Reference Concept Scheme Reference Code lists Data Flow Constraint

21 SDMX Cube Region Constraint
The DSD has an Activity code list with >170 items In table 101 we only report A*10 SDMX cube region constraint (linked to dataflow): List of allowed codes by concept ?

22 SDMX Key Set Constraint
Which series are allowed / forbidden?

23 The big picture: ongoing activities
ESS.VIP Validation ESS.VIP SERV Eurostat NAPS-S Service Oriented Architecture for National Accounts SDMX-VTL Validation & Transformation Language SDMX-MES Ownership Group on SDMX for Macroeconomic Statistics TFIDC Task Force International Data Sharing

24 The necessary pre-conditions
SDMX maintenance agreements Ownership and maintaining agency to be defined (normally between the international organisations) Internationally agreed data validation Data validation rules and SDMX data validation standards (beyond file structure validation) Streamlined data exchange processes Frequency and timeliness for data dissemination, data quality management etc.

25 A possible data sharing model
SDMX Web Services IMF, UN, WB, BIS, other IOs U S E R OECD Eurostat - ECB EU OECD other than EU World other than OECD

26 Technical Integration
National statistical organisations International organisations National databases (Mapping) Source SDMX Data Structure Defintions & Data Flows National data sources SDMX Implementation Challenge Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible

27 SDMX Basic Course National perspective: National Accounts
29th October 2015 Daniel SURANYI Cristina BLANARU

28 National process Why? : The business case
I want to… … structure my statistical datasets … exchange data with other organisations … align my definitions internationally using standards … validate that data is correct

29 Why? : The business case How we succeeded to…
… structure my statistical datasets using SDMX Data Structure Definitions … exchange data with other organisations using Push, Pull or Hub infrastructure … align my definitions internationally using standards with SDMX cross-domain concepts, code lists … validate that data is correct using SDMX schemas and VTL

30 Overall process Exchange Needs Concept Scheme Coding DSD Matrix
Optimising Publishing Artefacts

31 SDMX-NA DSD Matrix explaination- Overview Sheet(1)
Overview sheet - summarises all concepts used on Concept scheme and code lists used for these concepts.

32 SDMX-NA DSD Matrix explaination- Overview Sheet(2)
summarises all concepts and code lists Define the concepts (Cross domain concepts, Domain specific concepts) Adding code lists to the concepts

33 SDMX-NA DSD Matrix explaination Matrix Sheet(3)

34 SDMX-NA DSD Matrix explaination Matrix Sheet(4)
define the relationship between the datasets included in the scope of the transmission and the concepts each concept has a hyperlink pointing to the corresponding code list sheet.

35 SDMX-NA DSD Matrix explaination Matrix Sheet(5) DSD Matrix: data flows X concepts
FREQ REF_AREA STO ACTIVITY CONS PRODUCT Main Aggregates % # GDP and main indicators Sector Accounts General Government N Supply / Use A

36 SDMX-NA DSD Matrix explaination Matrix Sheet(6)
# concept fully used % concept partially used (code) one code of concept used (blank) concept not used

37 SDMX-NA DSD Matrix explaination Code list Sheets(7)
Showing the contents of each of the code lists used: CL_FREQ sheet CL_AREA sheet CL_ACCOUNT_ENTRY sheet CL_MATURITY sheet………….

38 Data Structure Definitions (DSDs)
5 DSDs covers multiple Dataflows NA_MAIN DSD NA_SEC NA_REG NA_SU NA_PENS All structures (DSDs) stored on the SDMX Global Registry+ on SDMX website:

39 Data Structure Definitions on the SDMX Global Registry

40 Data Structure Definitions on the on SDMX website
DSD ID and Name Version SDMX Download Excel templates* last update NA tables NA_MAIN Main Aggregates 1.5 2.0 / 2.1 V1.4a 12/12/2014 ESA: 1, 3, 5, 20, 22 NA_SEC Sector Accounts & Government Finance ECB: MUFA, GFS, FAM ESA-FINA: 6, 7 V1.4c ** 01/06/2015 ESA-SA: 8, 26 V1.4a 12/12/2014 ESA-GFS: 2, 9, 11, (25), 27, 28 OECD: 119, 7HH, 7II, PSD NA_REG Regional Accounts 2.0 *** V1.4a * 12/12/2014 ESA: 10, 12, 13 NA_SU Supply/Use & Input/Output Statistics V1.4b ** 19/01/2015 ESA: 15, 16, 17, 18, 19 NA_PENS Pensions V1.4b ** 09/05/2015 ESA: 29 NA_MAIN93 Main Aggregates, SNA93 N/A NA_SEC93 Sector Accounts, SNA93 IMF:GFS IMF Government Finance Statistics Manual 2014 1.0

41 Data provider view: One goal – different possibilities
Webforms Online data collection tools SDMX-ML file Database export SDMX Converter Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD Same Format Excel sheets SDMX Converter SDMX-ML file

42 Data provider view:One goal – different possibilities
Webforms Online data collection tools SDMX-ML file Excel sheets PHASE OUT FOR ESA IN 09/2016 Same DSD Same Format SDMX Converter SDMX-ML file Database export SDMX Converter SDMX-ML file Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service

43 SDMX Reference Infrastructure https://webgate. ec. europa
Data Provider Data Collector SDMX Registry Mapping Assistant DSD Web Svc Web Client Test Client Non-SDMX local database SDMX data set SDMX-RI

44 NSIs: Local Database (1)
Create a DB for National Accounts (if you don't have one in your NSI) One table per DSD (just 5 tables for the whole transmission program) Avoid the proliferation of tables A simply DB structure Load data into table Use Mapping Assistant tool of SDMX-RI package

45 NSIs: Local Database (2)
Create a link from NSI source DB to the DSD Use Test Client of the SDMX –RI package (to export data from the table to an SDMX file) Validate the table Send the table to EUROSTAT

46 Create a link from NSI source DB to the DSD Mapping Assistant: Database  SDMX (3)

47 NSI Web Client

48 Web Client pivot table

49

50 Fusion Cloud SDMX Validation Service https://data.sdmxcloud.org/
Validation against DSD Correct DSD used Correct concepts and identifiers Dimensions and mandatory attributes present Correct codes used Validation against Data Flow Correct codes for specific ESA2010 table Simple data viewer

51 Demo fusion cloud validation (1)

52 Demo fusion cloud validation (2)

53 Demo fusion cloud validation (3)

54 Summary Projects around SDMX and National Accounts
Governance, maintenance, sharing Implementation in NA Business process re-engineering Structural and content validation Interational data sharing National perspective Working with the DSD Matric Working with the SDMX-MES DSDs Architecture & tools, validation demonstration

Download ppt "7c. SDMX practical use case: National Accounts"

Similar presentations

For National Accounts (incl. GFS), BOP and FDI Meeting of the Advisory Expert Group (AEG) on National Accounts September 8-10, 2014, Washington, DC.

For National Accounts (incl. GFS), BOP and FDI Meeting of the Advisory Expert Group (AEG) on National Accounts September 8-10, 2014, Washington, DC.
Data Sharing Werner Bier Deputy Director-General Statistics European Central Bank Inter-Agency Group on Economic and Financial Statistics (IAG) G-20 Data.

Data Sharing Werner Bier Deputy Director-General Statistics European Central Bank Inter-Agency Group on Economic and Financial Statistics (IAG) G-20 Data.
Better data quality through global data and metadata sharing

Better data quality through global data and metadata sharing
SDMX – AN OECD PERSPECTIVE Paul Schreyer OECD CCSA Special Session, September 2014 Rome.

SDMX – AN OECD PERSPECTIVE Paul Schreyer OECD CCSA Special Session, September 2014 Rome.
The views expressed herein are those of the author and should not be attributed to the IMF, its Executive Board, or its management Meeting on the Management.

The views expressed herein are those of the author and should not be attributed to the IMF, its Executive Board, or its management Meeting on the Management.
Background Data validation, a critical issue for the E.S.S.

Background Data validation, a critical issue for the E.S.S.
Global SDMX Implementation Experience from on-going projects Daniel Suranyi, Eurostat Project Officer for SDMX implementation SDMX Expert Group, 13-14.

Global SDMX Implementation Experience from on-going projects Daniel Suranyi, Eurostat Project Officer for SDMX implementation SDMX Expert Group,
1 Annual National Accounts  1. Situation of OECD annual national accounts database  2. New features of the joint OECD-Eurostat questionnaire  3. COFOG2.

1 Annual National Accounts  1. Situation of OECD annual national accounts database  2. New features of the joint OECD-Eurostat questionnaire  3. COFOG2.
Implementation of SDMX for data and metadata exchange SDMX Basics Course 12 April 2013 Daniel Suranyi Eurostat B5 Management of statistical data and metadata.

Implementation of SDMX for data and metadata exchange SDMX Basics Course 12 April 2013 Daniel Suranyi Eurostat B5 Management of statistical data and metadata.
Eurostat Global Implementation of SDMX in National Accounts Committee for the Coordination of Statistical Activities 22 nd Session 4-6 September Ankara,

Eurostat Global Implementation of SDMX in National Accounts Committee for the Coordination of Statistical Activities 22 nd Session 4-6 September Ankara,
13-Jul-07 Implementation of SDMX for data and metadata exchange Balance of Payments Working Group 2-3 April 2012 Daniel Suranyi Eurostat B5 Management.

13-Jul-07 Implementation of SDMX for data and metadata exchange Balance of Payments Working Group 2-3 April 2012 Daniel Suranyi Eurostat B5 Management.
SDMX Implementation in the European Statistical System P. Everaers, A. Götzfried Eurostat.

SDMX Implementation in the European Statistical System P. Everaers, A. Götzfried Eurostat.
Basics David Barraclough OECD SDMX Coordinator

Basics David Barraclough OECD SDMX Coordinator
Eurostat 1 7a. Practical use case 1: Pesticides Use Project Blanaru Cristina Eurostat Unit B5: “Central data and metadata services” SDMX Basics course,

Eurostat 1 7a. Practical use case 1: Pesticides Use Project Blanaru Cristina Eurostat Unit B5: “Central data and metadata services” SDMX Basics course,
Model and Representations

Model and Representations
A Data Structure Definition for Eurostat's short-term business statistics Jan Planovsky, Eurostat SDMX Global Conference Bangkok, 28-30 September 2015.

A Data Structure Definition for Eurostat's short-term business statistics Jan Planovsky, Eurostat SDMX Global Conference Bangkok, September 2015.
Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”

Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”
Eurostat SDMX and Global Standardisation Marco Pellegrino Eurostat, Statistical Office of the European Union Bangkok, 28-30.

Eurostat SDMX and Global Standardisation Marco Pellegrino Eurostat, Statistical Office of the European Union Bangkok,
Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October.

Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October.
SDMX IT Tools Introduction

SDMX IT Tools Introduction

Similar presentations

Ads by Google