7. SDMX practical use case: National Accounts Blanaru Cristina Eurostat Unit B5: “Data and metadata services and standards” SDMX Basics course, 14 - 15 March 2016

Content Projects around SDMX and National Accounts Governance, maintenance, sharing National perspective Working with the DSD Matrix Working with the SDMX-MES DSDs Architecture & tools, validation demonstration

National process Why? : The business case I want to… … structure my statistical datasets … exchange data with other organisations … align my definitions internationally using standards … validate that data is correct

Why? : The business case How we succeeded to… … structure my statistical datasets using SDMX Data Structure Definitions … exchange data with other organisations using Push, Pull or Hub infrastructure … align my definitions internationally using standards with SDMX cross-domain concepts, code lists … validate that data is correct using SDMX schemas and VTL

SDMX-MES Ownership Group (OG) SDMX for Macro-Economic Statistics Mandated by SDMX Sponsors in January 2014 As a single OG for NA & GFS, BOP and FDI Maintenance: NA/GFS (https://sdmx.org/?page_id=1498) and BOP/FDI (https://sdmx.org/?page_id=1747) New domains can be added (e.g. Prices) Representation of management from content and SDMX side Oversees maintenance of SDMX artefacts Mandated technical group(s)

SDMX IMPLEMENTATION IN NATIONAL ACCOUNTS NATIONAL PERSPECTIVE

Overall process Exchange Needs Concept Scheme Coding DSD Matrix Optimising Publishing Artefacts

SDMX-NA DSD Matrix explaination- Overview Sheet(1) Overview sheet - summarises all concepts used on Concept scheme and code lists used for these concepts.

SDMX-NA DSD Matrix explaination- Overview Sheet(2) summarises all concepts and code lists Define the concepts (Cross domain concepts, Domain specific concepts) Adding code lists to the concepts

SDMX-NA DSD Matrix explaination Matrix Sheet(3)

SDMX-NA DSD Matrix explaination Matrix Sheet(4) define the relationship between the datasets included in the scope of the transmission and the concepts each concept has a hyperlink pointing to the corresponding code list sheet.

SDMX-NA DSD Matrix explaination Matrix Sheet(5) DSD Matrix: data flows X concepts FREQ REF_AREA STO ACTIVITY CONS PRODUCT Main Aggregates % # GDP and main indicators Sector Accounts General Government N Supply / Use A

SDMX-NA DSD Matrix explaination Matrix Sheet(6) # concept fully used % concept partially used (code) one code of concept used (blank) concept not used

SDMX-NA DSD Matrix explaination Code list Sheets(7) Showing the contents of each of the code lists used: CL_FREQ sheet CL_AREA sheet CL_ACCOUNT_ENTRY sheet CL_MATURITY sheet………….

Data Structure Definitions (DSDs) 5 DSDs covers multiple Dataflows NA_MAIN DSD NA_SEC NA_REG NA_SU NA_PENS All structures (DSDs) stored on the SDMX Global Registry+ on SDMX website: www.sdmx.org

SDMX Dataflow definition 1 DSD covers multiple Dataflows NA_MAIN DSD: Table 1: T0101_A, T0102_A, T0103_A … Table 3: … Table 5: … … Each of those Dataflows gives a partial picture "cutting the DSD": SDMX cube region "listing each series": SDMX key set

Data Structure Definitions (DSDs) DSD ID NA_... DSD Title Dataflow definitions MAIN Main aggregates ESA / OECD: 1, 3, 5, 20, 22 SEC Sector accounts and government finance ESA / OECD: 2, 6, 7, 8, 9, 11, 26, 27, 28 ECB: MUFA, GFS EDP: 1, 2, 3, 4 OECD: 119, 7HH, 7II, PSD G20: Recommendation 15 SU Supply / use and input / output ESA : 15, 16, 17, 18, 19 REG Regional statistics ESA: 10, 12, 13 PENS Pension schemes ESA: 29

Cross Domain Concepts and Code Lists Shared Concepts and Code Lists National Accounts SNA 2008 ESA 2010 Other NA related domains FDI statistics Other domains Balance of Payments Statistics Cross Domain Concepts and Code Lists Shared Concepts and Code Lists

SDMX Information Model - Summary Reference DSD Reference Reference Concept Scheme Reference Code lists Data Flow Constraint

Concepts in SDMX-MES DSDs 1 FREQ 2 ADJUSTMENT 3 REF_AREA 4 COUNTERPART_AREA 5 REF_SECTOR 6 COUNTERPART_SECTOR 7 CONSOLIDATION 8 ACCOUNTING_ENTRY 9 STO 10 INSTR_ASSET 11 MATURITY 12 ACTIVITY 13 ACTIVITY_TO 14 PRODUCT 15 PRODUCT_TO 16 EXPENDITURE 17 PENSION_FUNDTYPE 19 UNIT_MEASURE 20 CURRENCY_DENOM 21 VALUATION 22 PRICES 23 TRANSFORMATION 24 CUST_BREAKDOWN Concepts Cross Domain Shared NA specific Obs OBS_STATUS CONF_STATUS EMBARGO_DATE Ser DECIMALS TABLE_IDENTIFIER UNIT_MULT COMPILING_ORG REF_YEAR_PRICE Dset COMMENT_DSET …

SDMX Cube Region Constraint The DSD has an Activity code list with >170 items In table 101 we only report A*10 SDMX cube region constraint (linked to dataflow): List of allowed codes by concept ?

SDMX Key Set Constraint Which series are allowed / forbidden?

Data Structure Definitions on the SDMX Global Registry

Data Structure Definitions on the on SDMX website DSD ID and Name Version SDMX Download Excel templates* last update NA tables NA_MAIN Main Aggregates 1.5 2.0 / 2.1 V1.4a 12/12/2014 ESA: 1, 3, 5, 20, 22 NA_SEC Sector Accounts & Government Finance ECB: MUFA, GFS, FAM ESA-FINA: 6, 7 V1.4c ** 01/06/2015 ESA-SA: 8, 26 V1.4a 12/12/2014 ESA-GFS: 2, 9, 11, (25), 27, 28 OECD: 119, 7HH, 7II, PSD NA_REG Regional Accounts 2.0 *** V1.4a * 12/12/2014 ESA: 10, 12, 13 NA_SU Supply/Use & Input/Output Statistics V1.4b ** 19/01/2015 ESA: 15, 16, 17, 18, 19 NA_PENS Pensions V1.4b ** 09/05/2015 ESA: 29 NA_MAIN93 Main Aggregates, SNA93 N/A NA_SEC93 Sector Accounts, SNA93 IMF:GFS IMF Government Finance Statistics Manual 2014 1.0

Data provider view: One goal – different possibilities Webforms Online data collection tools SDMX-ML file Database export SDMX Converter Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD  Same Format Excel sheets SDMX Converter SDMX-ML file

Data provider view:One goal – different possibilities Webforms Online data collection tools SDMX-ML file Excel sheets PHASE OUT FOR ESA IN 09/2016   Same DSD  Same Format SDMX Converter SDMX-ML file Database export SDMX Converter SDMX-ML file Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service

SDMX Reference Infrastructure https://webgate. ec. europa Data Provider Data Collector SDMX Registry Mapping Assistant DSD Web Svc Web Client Test Client Non-SDMX local database SDMX data set SDMX-RI

NSIs: Local Database (1) Create a DB for National Accounts (if you don't have one in your NSI) One table per DSD (just 5 tables for the whole transmission program) Avoid the proliferation of tables A simply DB structure Load data into table Use Mapping Assistant tool of SDMX-RI package

NSIs: Local Database (2) Create a link from NSI source DB to the DSD Use Test Client of the SDMX –RI package (to export data from the table to an SDMX file) Validate the table Send the table to EUROSTAT

Create a link from NSI source DB to the DSD Mapping Assistant: Database  SDMX (3)

NSI Web Client

Web Client pivot table

Fusion Cloud SDMX Validation Service https://data.sdmxcloud.org/ Validation against DSD Correct DSD used Correct concepts and identifiers Dimensions and mandatory attributes present Correct codes used Validation against Data Flow Correct codes for specific ESA2010 table Simple data viewer

Demo fusion cloud validation (1)

Demo fusion cloud validation (2)

Demo fusion cloud validation (3)

Thank you for your attention! Questions