SOA initiatives at Istat Marco Silipo ISTAT (Italian National Statistical Institute) SERV project - task force meeting Eurostat October 28th, 2015

2015/2016: Istat is changing..why? Istat conducts more than 120 surveys Design of business processes is strongly base on domain specific silos: duplications of procedures, limited interoperability, similar tools in use, repeated tasks, limited exploitment of newest technologies, old style data collection systems Need to exploit also the BIG amount of DATA which is available nowadays through administrative sources Challenges coming from competitors in the private sector

The ‘Modernization project’ The key pillar of the strategy is process industrialisation based on Service-Oriented Architecture (SOA) and on the standardisation of statistical methods, tools and IT services for data collection, process and analysis, and dissemination. A centralised data collection system (including web portals) with related tools is provided in order to reduce response burden and supply tailor-based information to users A new Unified metadata system, integrated with the business architecture An architecture for dissemination, based on integrated data warehouse for data browsing, accessible micro data for research and Linked Open Data portals Need for a new organizational model Improved and more efficient governance model

The ‘Modernization project’ goal The final goal is… establishing a set of standard methods and tools that would constitute the library (a real Istat Store) to be used by analysts for their tasks, develop SOA-compliant standard software to perform data processing functions gradually replace existing instruments, so as to move away from the stovepipe model This should be supported by an improved and more efficient governance model

CORE: A tool for Statistic SOA … A brief history Common Reference Environment Environment for: i) executing standard statistical processes ii) defining processes in terms of abstract statistical services iii) mapping services to specific IT tools CSPA Common Statistical Production Architecture Reference architecture based on: i) existing standard models, such as GSBPM and GSIM, as shared vocabulary ii) the “plug and play” approach in designing, implementing and sharing statistical software solutions ESSNet CORA ESSNet CORE CORE in Istat CORE Essnet, CSPA Unece 2009 2010 2011 2012 2013 2014 2015 CSPA specification implementation CORE in Istat: Platform for the execution of statistical processes according to CSPA principles

CORE concepts match CSPA ones Tool Sas … Pl/sql Service definition Definition of a service at a conceptual level in terms of GSBPM subprocess, of the business function, of GSIM input/output. Service implementation Definition of a service at a physical level. A service definition can have one or more implementations Process Compose: compose a process with available service definitions Configure: bind each service with one of its available implementations Execute

CORE architecture CORE CORE engine R Sas Pl/sql Java Shell tools CORE is a Java web application i) implements a workflow engine ii) stores process executions iii) allows editing tool scripts iv) displays runtime logs

CORE at work 1. Define: 2. Compose: 3. Configure: 4. Prepare: - a service definition for each step in your process - a service implementation for each service definition, choosing a tool 2. Compose: conceptual design of a process, connecting service definitions 3. Configure: physical design of a process, starting from a conceptual process bind each service definition to an implementation Define a service implementation for each service definition choose a tool among the available ones define a new tool Compose your process, connecting service definitions Configure your process… Run… physical design of a process, starting from a conceptual process bind each service definition to an implementation 4. Prepare: specify input (data file and configuration parameters) for each service implementation 5. Execute: run your process!

CORE engine Running service Interruptable service

CORE tools CORE allows editing tool scripts (R, Sas, Pl/sql, shell) stored on the server, through an editing panel

Concluding remarks CORE today CORE tomorrow i) CORE has been strongly re-engeenered, becoming an “easy to use” product for non technical users ii) different surveys (COE, Anvis, Frame) are moving towards a CORE-compliant design iii) CORE is becoming a key component in the process of modernization and automation of statistical surveys CORE tomorrow i) Implementation enhancements: web services support, more complex workflow (loop, switch), data mapping between heterogeneous services ii) Governance: policies, service and tool catalogue iii) CORE as a communication platform for CSPA services