Exchanging Reference Metadata using SDMX Towards a Global Reference Metadata Reporting Method I’m presenting this but the Global DSDs have required a great collaboration at all levels in many different organisations – involving senior management, statisticians, technicians, project managers, methodologists, etc.

Some Essential Terms Metadata is information about data. Needed to interpret and identify the data Structural Metadata is information about the structure of the data. Examples: The “shape” of a DSD such as the dimensions The Code Lists IDs and Descriptions The list of attributes Reference Metadata is information about data that is designed for humans to understand more about the data, and can be sent separately to the data Examples: Contact information; The source of the data; Release policy; Quality aspects Attributes are information required to process and interpret the data. They need to be sent with the data, for example: The status of an observation: Break in Series; Estimated; Missing Embargo time: When the data will be available for publication Confidentiality Status: Is the data internal only or can it be disseminated? Unfortunately we often say simply “Metadata” and we may be referring to either type

The Problem Statement There is very little exchange of ref metadata Manual processes are involved that are labor-intensive and error-prone Organisations have very different reference metadata frameworks (list of types of ref metadata) IMF have a collection of metadata concepts Eurostat have SIMS OECD has something that hasn’t got a name (42 types so it could be the meaning of life!) This makes it difficult for reporters to supply ref metadata Therefore it doesn’t happen The lack of harmonisation means that there is a lack of tools

The Proposal Make exchange of ref metadata possible using SDMX Identify the SDMX cross-domain concepts that are related to ref metadata (they’re already in the SDMX Glossary) Provide a Global Reference Metadata Concept Scheme (MCS) This is an abstract artefact, therefore you will need… Also create a Global Metadata Structure Definition (MSD) This is used to define what goes into a metadata dataset/message but may not serve the purposes of all statistical domains, so… Describe how to derive custom Metadata Structure Definitions from the Global Metadata Concept Scheme when needed

Difference between a Concept Scheme and Data or Metadata Structure Definition Concept Scheme is the set of concepts (dimensions, attributes) to fully describe the concept or framework: Labour Statistics Concept Scheme has all concepts to describe Labour Statistics Balance of Payments Concept Scheme has all concepts to describe BPM A reference metadata Concept Scheme has all concepts to describe a reference metadata framework (SIMS at Eurostat, DQAF at IMF, Global for SDMX community The Concept Scheme is abstract, it cannot be used to create messages Meta/Data Structure Definition is used to create messages Derived from a Concept Scheme. Example: National Accounts has 5 DSDs that all use concepts from the National Accounts Concept Scheme It reuses concepts from the Concept Scheme for related DSD/MSDs Metadata Structure Definition will derive a subset of concepts from the Metadata Concept Scheme

Some concepts in the Global Metadata Concept Scheme Title Confidentiality Time coverage Valuation Data source Confidentiality - data treatment Data presentation - Summary description Data compilation Contact Confidentiality - policy Data presentation - Detailed description Adjustment Contact email address Professionalism Source data type Data validation Contact fax number Professionalism - code of conduct Data collection method Data revision Contact mail address Professionalism - impartiality Time period Data revision - policy Contact name Professionalism - methodology Time period - collection Data revision - practice Contact organisation Professionalism - statistical commentary Base period Data revision - studies Contact organisation unit Cost and burden Base weight Imputation Contact person function Cost and burden - efficiency management Frequency of data collection Statistical population Contact phone number Cost and burden – resources Classification system Quality management Institutional mandate Statistical concepts and definitions Quality management - quality assessment Institutional mandate – data sharing Reference area Statistical unit Quality management - quality assurance Institutional mandate - legal acts and other agreements Sector coverage Accounting conventions Quality management - quality documentation

The Global MCS and Derivations Global MCS & Global MSD Custom MSD (implementable subset of Global MCS +extensions) Custom MSD (implementable subset of Global MCS concepts)

Examples and Metadataflows

Mapping to the Global MCS Statistical Office Int. Org. Ministry/ Bureau Central Bank

Implementation Example

Implementation User Stories: Statistical agency/dissemination perspective The agency currently stores its reference metadata in a database and has an extraction method. To output ref metadata to the Global MCS Keep the existing metadata system components with no changes Create a mapping between the agency’s existing metadata concepts and the Global MCS concepts Use a proxy component to do the following: a. Take input from the existing agency’s metadata component b. Transform the existing format to SDMX c. Apply the mapping created in step 2. d. Output the resulting SDMX that matches the Global/Derived MSD Further work could improve efficiency: Change the agency’s metadata format to SDMX to avoid step 3b Align the agency’s metadata framework to the Global MSC to avoid step 3c

David Barraclough david.barraclough@oecd.org Thank you David Barraclough david.barraclough@oecd.org SDMX Official Website: https://sdmx.org/ Technical Working Group: twg@sdmx.org Statistical Working Group: swg@sdmx.org