National Institute of Statistics – Italy MICROCOE: A multilevel data warehouse to support the production and expand the analysis and the dissemination of foreign trade and FATS statistics Marco Giorcelli National Institute of Statistics – Italy ESSnet on micro data linking and data warehousing in statistical production. Rome, April 6, 7 2011

Issues covered by this presentation History and key characteristics of the current data warehouse for the production and dissemination of foreign trade statistics The drivers underpinning a radical change in IT support to the production and dissemination in the domain of Globalisation statistics Key characteristics of MICROCOE, the new micro level datawarehouse for foreign trade and foreign affiliates (FATS) statistics Conclusions

Key achievements in the last 15 years: History and key characteristics of the current data warehouse for the production and dissemination of foreign trade statistics Key achievements in the last 15 years: Successful migration from Adabas/Natural and SAS to Oracle RDBMS in the late ’90s Set up and maintenance of the first and still largest on line ISTAT database for the dissemination: www.coeweb.istat.it Strong interaction between statisticians and IT people IT support to the link between foreign trade operators and the BR, and to changes in statistical production related to alteration of fiscal inputs or induced by New Foreign Trade Regulations Key characteristics: Huge amount of data uploaded every month The production data warehouse stocks on line foreign trade micro-transactions since 1991 Full reversibility between operators and product transactions, partial with respect to variables linked to BR 3

THE STRUCTURE OF THE CURRENT DATA WAREHOUSE FOR STATISTICAL PRODUCTION History and key characteristics of the current data warehouse for the production and dissemination of foreign trade statistics THE STRUCTURE OF THE CURRENT DATA WAREHOUSE FOR STATISTICAL PRODUCTION 4

History and key characteristics of the current data warehouse for the production and dissemination of foreign trade statistics IT TOOLS FOR IN DEPTH ANALYSIS OF TRADE FLOWS MICRO DATA INCLUDED IN THE PRODUCTION DATA WAREHOUSE 5

History and key characteristics of the current data warehouse for the production and dissemination of foreign trade statistics STRUCTURE OF THE CURRENT DISSEMINATION DATA WAREHOUSE 6

The drivers underpinning a radical change in IT support to production and dissemination of Globalisation statistics It is hard to reconsider an IT database from the scratch, given its strong reputation and reliability, but it is time for a radical change determined by External drivers New mission: Inward and Outward FATS surveys and trade in services intra EU have entered the Division following a reorganisation finalised to cover all statistics related to the Globalisation domain The production of statistics on exporting and importing companies is now under a EU Regulation Strong demand for calculation of indicators and figures based on micro-level trade transactions, including accessibility to micro data Convergence toward a better integration between SBS and trade statistics in the framework of MEETS Internal drivers Potential conflict between statistical rules and IT procedures given 15 years of cumulative but not always optimal IT development Strong reduction in human resources and need to support a deeper engagement in rebalancing automatic checks and manual revisions 7

MICROCOE is finalised to meet the following goals: Key characteristics of MICROCOE, the new datawarehouse on foreign trade and foreign affiliates (FATS) statistics MICROCOE is finalised to meet the following goals: To fully integrate at the micro level all Globalisation related surveys. These include trade in goods Intra and extra EU, trade in services intra EU, Inward and Outward FATS data To be open and technologically compatible to external sources, including single surveys or other firm level data warehousing To be internally consistent with respect to three different but fully hierarchical level of analysis: Level one: enterprise group level – FATS statistics Level two: enterprise level – FATS data, SBS and balance sheet administrative data, trade in goods and services reclassified by operators Level three: trade flows individual transactions – trade in goods and trade in services data potentially linked to FATS data (intra-firm trade by products!!!) To separate statistical rules from IT procedures thus improving transparency and efficiency in statistical production and IT management 8

An embryonic scheme of MICROCOE database Key characteristics of MICROCOE, the new datawarehouse on foreign trade and foreign affiliates (FATS) statistics An embryonic scheme of MICROCOE database National and European EGR Consolidated balance sheet data Parent company of MNE Level 1 1-N Italian affiliates abroad Foreign affiliates in italy Other resident enterprises BR, SBS data, unconsolidated balance sheet data Level 2 1-1 Firms engaged in trade in goods and sevices Firms not engaged in trade in goods and sevices 1-M Micro trade flows by product, country and resident / non resident operator Level 3 INTERNAL FRAME EXTERNAL SOURCES 9

A possible view of MICROCOE user interface Key characteristics of MICROCOE, the new datawarehouse on foreign trade and foreign affiliates (FATS) statistics A possible view of MICROCOE user interface Parent level data Company name Country of Residence Consolidated turnover Exports of goods Sum of unconsolidated turnover abroad Number of foreign affiliates MARIO SPA Italy 120.000 55.000 30.000 1 Company (Affiliate) level data Company name Country of residence Unconsolidated turnover Exports of goods Imports of goods MARIO 1 SPA IT 70.000 10.000 30.000 MARIO 2 SPA 50.000 20.000 25.000 MARIO GERMANY DE missing Trade flows level microdata (partial) Company name Direction of flows Type of goods (Combined nomenclature) Partner Country Monetary value Volume Trade partner operator MARIO 1 SPA Exports 45660010 US 3.000 3 45660020 DE 7.000 2 MARIO GERM Imports 45660030 CN 5.000 5 45660040 25.000 10 10

Full integration at the micro-level of all globalisation surveys Conclusions Data warehousing in foreign trade statistics has an history of success in ISTAT both for micro level production and aggregated data dissemination sides, so it has being hard for us to make a decision to change. MICROCOE benefits from what exists, but further expands along several directions: Full integration at the micro-level of all globalisation surveys Multilevel complex structure of the database Open to be integrated with other ISTAT data warehouses Complete user interface with all relevant information Key advantages: What we have is not just the “SUM” of already existing information, new information can be created (SBS figures by exporting companies and Intra-firm trade by product!!) plus more comprehensive analysis of trade flows based on new info on trade operators Threats: Entropy. This is a complex system with potentially huge cumulative knowledge processes, but it needs to be jointly managed by IT experts, statisticians and business analysts 11