(COmmon Reference Environment) CORE ESSnet (COmmon Reference Environment) progress report Carlo Vaccari Istat - Italy MSIS Meeting - Luxembourg May 23-25 2011 1

Outline Introduction and history CORE objectives CORE where we are Architecture implementation Information model CORE and SDMX CORE and GSIM MSIS Meeting - Luxembourg May 23-25 2011

CORA ESSnet Financed by Eurostat under 2009 Statistical Workprogramme Countries involved: it (coordinator), ch, dk, lv, nl, no, se Duration: October 2009 - October 2010 MSIS Meeting - Luxembourg May 23-25 2011

CORA Technical Architecture CORA Model: two dimensions Functional dimension Construction dimension Adoption of GSBPM 4.0 9 subprocesses of level 2 1 Specify Needs 2 Design 3 Build 4 Collect 5 Process 6 Analyse 7 Disseminate 8 Archive 9 Evaluate MSIS Meeting - Luxembourg May 23-25 2011

Construction Dimension: Layers Figures Time Series Statistic Population Unit Variable Value A domain of interest documented by statistical products Statistical series over time Integrated or simple statistical product for a given time A population at a given time A statistical unit at a given time A statistical variable at a given time A logical representation of the value of a variable MSIS Meeting - Luxembourg May 23-25 2011

CORA Model Grid Statistical processes compliant to CORA model are intended to be designed by statisticians MSIS Meeting - Luxembourg May 23-25 2011

After CORA…CORE! COmmon Reference Environment (CORE), financed by Eurostat under 2010 Statistical Workprogramme Countries involved: it (coordinator), fr, nl, no, pt, se Duration: December 2010 - January 2012 MSIS Meeting - Luxembourg May 23-25 2011

CORE Principal Outcomes Environment for the definition and execution of statistical processes Definition of a process in terms of services selected from an available repository Execution of the composed workflow MSIS Meeting - Luxembourg May 23-25 2011

CORE Outcomes: Design CORA model → CORA information model Design of CORE services and processes MSIS Meeting - Luxembourg May 23-25 2011

CORE Outcomes: Implementation Selection of available middleware solutions for process execution Realization of an environment able to permit the execution of processes: Interfaces (GUIs) for defining CORE processes for statistical users Integration APIs Repository of integration layers MSIS Meeting - Luxembourg May 23-25 2011

CORE Outcomes: Testing Realization of processes starting from services implementing some GSBPM phase Evaluation of costs related to integration Prototype implementation (to be engineered) MSIS Meeting - Luxembourg May 23-25 2011

CORE Architecture (1) GUIs to support modelling of CORE processes according to the CORA grid Modeling & control flow constructs Drag & drop facilities for process design Global schema Implementation: we are evaluating the usage of an open process editor tool (Oryx - http://bpt.hpi.uni- potsdam.de/Oryx/WebHome ) MSIS Meeting - Luxembourg May 23-25 2011 12

CORE Architecture (2) Process Runtime Controlled execution of services Implementation: integration of existing workflow solutions, currently in evaluation phase Service Runtime Integration APIs (in-out data transformation) Service execution Implementation: CSV and SQL data transformations are currently being implemented Service Repository Deployment of services MSIS Meeting - Luxembourg May 23-25 2011

CORE Information Model (1) First draft of CORE information model Del. 2.1 released: requirements for the model of the interface through which statistical services will communicate Information Model to be released, currently in discussion phase MSIS Meeting - Luxembourg May 23-25 2011

CORE Information Model (2) Design Principles (in discussion): Rectangular data sets (rows & columns) Strong typing (data, rules, parameters) Dataset kinds (eg micro/aggreg) Free-style arguments (eg scripts tool dependent) Other (service arguments and infos) MSIS Meeting - Luxembourg May 23-25 2011

CORE and SDMX Both initiatives foster standardization CORE SDMX Focus on standardization of processes and data exchanges (mainly) intra-NSI SDMX Focus on standardization of processes and data exchanges (mainly) inter-NSIs (or between NSIs and international organizations) MSIS Meeting - Luxembourg May 23-25 2011 16

CORE and SDMX - 2 CORE SDMX Focus on all phases of statistical processes Both micro and macro data considered SDMX Focus (mainly) on dissemination phase Mainly macro data considered MSIS Meeting - Luxembourg May 23-25 2011 17

Information Model Both propose an information model CORE information model Takes explicitly process dimension into account through GSBPM Data dimension SDMX information model Mainly focused on data dimension MSIS Meeting - Luxembourg May 23-25 2011 18

CORA Information Model MSIS Meeting - Luxembourg May 23-25 2011

SDMX Data & Metadata Information Model Data or Metadata Structure Definition uses specific data or metadata structure can have child categories can be linked with categories from multiple category schemes Data or Metadata Set Data or Metadata Flow Category conforms to business rules of the data or metadata flow comprised of subject or reporting categories can get data from multiple data providers Data Provider Provision Agreement Category Scheme MSIS Meeting - Luxembourg May 23-25 2011

On Information Models Different abstraction levels CORE SDMX “Higher” modelling level E.g.: statistics as tabular data SDMX “Lower” modeling level E.g.: aggregated data set with dimensions, attributes and measures MSIS Meeting - Luxembourg May 23-25 2011

Open Issues - 1 Can we use SDMX for micro and macro data exchanges in a CORE process? Need for mapping of information models MSIS Meeting - Luxembourg May 23-25 2011

On-Going Work - 1 CORE implementation scenario within Istat Main phases: Sample selection and allocation CORE wrapping of available SAS and R procedures MSIS Meeting - Luxembourg May 23-25 2011

On-Going Work - 2 Design and implementation of CORE Integration APIs Possible in/out SDMX translations CORE TOOL SDMX SDMX IAPI TOOL IAPI MSIS Meeting - Luxembourg May 23-25 2011

Open Issues - 2 What about metadata? CORE: Data and metadata managed in the same way SDMX: Distinction between structural metadata and reference metadata Dedicated effort for metadata management MSIS Meeting - Luxembourg May 23-25 2011

Collaboration between CORE/SDMX ESSnets CORE planned deliverable on “Feedbacks on SDMX Usage in CORE” Periodical meetings inside Istat between coordinators of the two ESSnets Exchanges of resources between the two ESSnets MSIS Meeting - Luxembourg May 23-25 2011

CORE and GSIM GSIM: Generic Statistical Information Model deliverable from OCMIMF Operationalising a Common Metadata/Information Management Framework activity inside Statistical Network Ambiguity on the acronym: reference to “generic statistical information model” in CORE ESSnet proposal In March started activity to clarify relationships (thanks to J.P.Kent and A.Hamilton) MSIS Meeting - Luxembourg May 23-25 2011

CORE and GSIM First analysis and discussions: the deliverables from the two initiatives are complementary in intent and do not overlap in concept Necessary to avoid gaps and/or duplications and ensure the complementary relationship MSIS Meeting - Luxembourg May 23-25 2011

CORE Information Model CORE will define a very generic information model (CORE-IM) for the interface through which statistical services will communicate with each other within the framework of the CORA model As a communication protocol, CORE-IM focuses on the “postal envelope” used when passing information between services, rather than focusing in detail on the information being communicated (ie what is inside) MSIS Meeting - Luxembourg May 23-25 2011

CORE and GSIM CORE-IM current hypothesis: to support a flag to indicate if the information being communicated is described within GSIM → without claiming to align the semantics of the content (eg, “classification”), but only to alert a consuming service which “understands” GSIM that it can relate the content to GSIM MSIS Meeting - Luxembourg May 23-25 2011

Complementary nature CORE-IM supporting semantic interoperability at a very high, abstract level (“here is an information object, along with the ‘envelope’ information about it”) where GSIM can provide greater semantic precision to a subset of information objects communicated using CORE CORE supporting communication between services → substantial interoperability benefits Information aligned with GSIM semantics → further level of interoperability MSIS Meeting - Luxembourg May 23-25 2011

Complementary nature GSBPM reference model for statistical business processes GSIM reference model for information input to, used by and produced by those processes Models are independent → it's possible to use one without the other CORE-IM recognizes and uses GSBPM and (hopefully) will do the same with regard to GSIM, giving them a potential contact point MSIS Meeting - Luxembourg May 23-25 2011

Coordination in practice Need to maximize the extent to which these synergies are achieved in practice: Members in common (no, se) ABS leader for the OCMIMF and observer in CORE CORE members external reviewers for GSIM material CORE WP2 “co-ordination input” from the OCMIMF collaboration team in regard to deliverables Half day session at METIS workshop (October) presenting CORE and OCMIMF works to external metadata specialists Common documents in preparation ... MSIS Meeting - Luxembourg May 23-25 2011