1. The Data Revolution and Official Statistics
Albrecht Wirthmann
Eurostat
ESS Big Data Workshop
Ljubljana,
13-14 Oct 2016

2. Big Data and Official Statistics
What will be the impact of ubiquitous data collection and networking
Mobile Communication
Internet of [every]Things,
Cloud services,
Wearables,
Autonomous traffic,
Smart systems, …
on official statistics?

3. Scheveningen Memorandum on Big Data – September 2013
Examine the potential of Big Data sources for official statistics
Official Statistics Big Data strategy as part of wider government strategy
Address privacy and data protection
Collaboration at European and global level
Address need for skills
Partnerships between different stakeholders (government, academics, private sector)
Developments in Methodology, quality assessment and IT
Adopt action plan and roadmap for the European Statistical System
At their meeting in September 2013, the directors general of the European Statistical System agreed on a memorandum on Big Data. The so called Scheveningen memorandum called for the submission of the action plan and roadmap on Big Data. It also expresses the main points of such an action plan, such as the integration of a Big Data strategy for statistics into an overall government strategy, the need for skills, the collaboration at European as well as at global level, the need for methodological developments and the protection of personal data.
The action plan and roadmap was adopted by the ESS at its meeting on 25 Sep 2014.

4. Expected benefits of using big data ?
Outward-looking
More adequate and flexible response to user needs
Wider range of statistical products and services (without increasing burden)
Better understand quality aspects of new sources
Higher temporal and spatial resolution
Inward-looking
Acquisition of new competences for NSIs
Increase efficiency in producing statistics
We remain key players for statistical information
(self-explanatory)

5. Follow-up Activities Eurostat TF Big Data Jan 2014 ESS TF Big Data
Big data workshop in Rome
Big Data Action Plan and Roadmap 1.0 Sep 2014
Big Data Business case June 2015
Integration into Vision 2020 portfolio
ESSnet Big Data Nov May 2018
Specific Grant Agreements: 2016, 2017
Contract on legal, ethical, communication, skills issues and workshop 2016
At their meeting in September 2013, the directors general of the European Statistical System agreed on a memorandum on Big Data. The so called Scheveningen memorandum called for the submission of the action plan and roadmap on Big Data. It also expresses the main points of such an action plan, such as the integration of a Big Data strategy for statistics into an overall government strategy, the need for skills, the collaboration at European as well as at global level, the need for methodological developments and the protection of personal data.
The action plan and roadmap was adopted by the ESS at its meeting on 25 Sep 2014.

6. Ethics / Communication
Action Plan Themes
Policy
Quality
Skills
Experience sharing
Legislation IT
Infrastructures
Methods
Ethics / Communication
Partnerships
Pilots
This slide shows the topics that have been developed in the action plan and roadmap. The ESSC reinforced in its opinion the legal and ethical issues as well as the development of appropriate skills.
Some of the topics should be directly further developed such as policy, communication or skills, while other topics should be further elaborated as part of the pilots that will be run at EU level.
These topics are all somehow interrelated, but let me cluster or group them in the next few slides, and highlight some of the concrete actions that are underway.
NEXT SLIDES:
Pilots - Experience sharing – methods
Skills & IT infrastructures
Legislation – policy
4. Transversal areas: quality framework + ethics/comm.

7. Ethics / Communication
Policy
Quality
Skills
Experience sharing
Legislation IT
Infrastructures
Methods
Ethics / Communication
Partnerships
Pilots
Challenges
Exploration & tentative implementation
Scientific Approach
Cooperation, sharing of know-how
Access to data, lack of skills, new user demands
Actions
Big Data ESSnet: Pilot projects
2015 – 2019 (Grants to Statistical Offices)
Exploring different big data sources
Characteristics of data, potential outputs, Methodology and Quality, IT requirements
Partnerships with data providers, academia, users
Cooperation with UN
Conferences and workshops
A first set of challenges refers to the cooperation and exchange of best practices, the methodology and the transition into the "real use" of data. These are perhaps the areas that are closest to a statistician's heart.
One way of tackling these issues, is the launching of a series of PILOT PROJECTS.
A Framework Partnership Agreement between Eurostat and 20 NSIs was signed in Nov In Dec 2015 Eurostat launched the Special Grant Agreements that will provide the resources to the NSIs to carry out the work.
In this context close cooperation between the ESS and the GWG will be necessary in order to avoid double work and ensure synergies between the two groups.
These pilot projects will be an important pillar of the big data activities in the ESS in the coming years and should pave the way towards a data production driven by big data.

8. Analysis of websites' contents
Communication
Mobile Communication
Social Media
WWW
Web Searches
Analysis of websites' contents
Job Advertisements
Businesses' Websites
E-Commerce
Real estate
Internet Traffic
Sensors
Traffic loops
Smart meters
Automatic Vessel Identification System
Satellite Images
Webcams
Process generated data
Reservation Systems
Flight Booking transactions
Tourist lodgings
Trains
Hotels
Supermarket Cashier Data
Loyalty Cards
Financial transactions
Mobile Payments
eGovernment
Crowd sourcing
Voluntary Geographic Information websites (OpenStreetMap)
Voluntary Information websites (Wikipedia)
Community pictures collection

9. ESS Big Data Pilots List of pilot projects (Specific Grant Agreement)
Web scraping
job vacancies ; enterprise characteristics
Smart meters
electricity consumption ; temporary vacant dwellings
Automatic Identification System (Ships)
vessel identification data
Mobile phone data
Preparing for Access to data
Scenario for using multiple inputs
Modelling for now-casting statistics
A first set of challenges refers to the cooperation and exchange of best practices, the methodology and the transition into the "real use" of data. These are perhaps the areas that are closest to a statistician's heart.
One way of tackling these issues, is the launching of a series of PILOT PROJECTS.
We hope to conclude a Framework Partnership Agreement very soon and will then launch the Special Grant Agreements that will provide the resources to the countries to carry out the work.
These pilot projects will be an important pillar of the big data activities in the ESS in the coming years and should pave the way towards a data production driven by big data.

10. Ethics / Communication
Policy
Quality
Skills
Experience sharing
Legislation IT
Infrastructures
Methods
Ethics / Communication
Partnerships
Pilots
Challenges
Computing capacity, hardware ?
Analytical tools, software?
Storage ?
Actions
Sandbox IT infrastructure for experimenting
UNECE, Eurostat, EU
Cloud Infrastructure
Test of software
Secondly, important enablers for a successful move towards big data, are SKILLS and IT INFRASTRUCTURE.
Our staff will slowly but steadily need new skills and our IT architecture & infrastructure will need to adapt to the new sources.
The impact on hardware needs will be significant. Experiments are ongoing, for instance the "sandbox" environment for big data experiments hosted by the Irish Central Statistics Office – in a cooperation between among others Eurostat and UNECE.
An concrete action in the pipeline, is the set-up of a series of training courses under the umbrella of the ESTP. Our Task Force on Big Data is currently preparing the outline for such program for The courses will focus on sources and on tools and will be modulated in a way to address basic/new users or management as well as more experienced users.

11. Ethics / Communication
Policy
Quality
Skills
Experience sharing
Legislation IT
Infrastructures
Methods
Ethics / Communication
Partnerships
Pilots
Challenges
New skills for staff: statisticians vs. data scientists ?
Relations with external data providers
Fast changing conditions and situations
Actions
Training Strategy
Competency based approach
Program for European statisticians (ESTP)
In the next years: dedicated courses on big data
Focus on big data sources and on big data tools
Acquiring the skills needed to assess sources and their quality, the skills to use tools and to explore big data sources
Secondly, important enablers for a successful move towards big data, are SKILLS and IT INFRASTRUCTURE.
Our staff will slowly but steadily need new skills and our IT architecture & infrastructure will need to adapt to the new sources.
The impact on hardware needs will be significant. Experiments are ongoing, for instance the "sandbox" environment for big data experiments hosted by the Irish Central Statistics Office – in a cooperation between among others Eurostat and UNECE.
An concrete action in the pipeline, is the set-up of a series of training courses under the umbrella of the ESTP. Our Task Force on Big Data is currently preparing the outline for such program for The courses will focus on sources and on tools and will be modulated in a way to address basic/new users or management as well as more experienced users.

12. Ethics / Communication
Policy
Quality
Skills
Experience sharing
Legislation IT
Infrastructures
Methods
Ethics / Communication
Partnerships
Pilots
Challenges
Access to data & continuity of access
Data security & privacy concerns
Use of data only for statistical purposes
Impact on the public opinion of privacy and security concerns ?
Actions
Analysis of legislative situation in EU
Review of ethical guidelines inclusion into Statistics Code of Practice
Communication Strategy
Other important areas relate to the policy / political framework and the regulatory framework. Given the interaction between policy and regulation, it is very important to work on these areas in parallel and in narrow cooperation.
One aspect of policy will be the integrating of (official) statistics into any strategy related to big data. This is essential to put statistics on the map and to open doors to actually accessing of data. It should be kept in mind that big data are often held or stored by private companies, e.g. mobile network operator.
The discussion of access is not limited to the entry but should include a long term vision, in other words a certain continuity of access – this is a conditio sine qua non for a sound statistical system that is based, fully or partially, on big data sources.
A main barrier to access, is data security and privacy concerns –as was also highlighted in the feasibility study carried out with respect to tourism statistics.
Another important challenge is finding a sustainable business model for big data in official statistics, taking into account the budgetary impact for statistical offices and for those "holding" the data.
To address these questions, I can mention that Eurostat recently launched a Call for Tender with the objective of analysing the legal frameworks at EU and national level.

13. Ethics / Communication
Policy
Quality
Skills
Experience sharing
Legislation IT
Infrastructures
Methods
Ethics / Communication
Partnerships
Pilots
Challenges
Datafication
Impact on Official Statistics
Answer of Statistical System
Reaction of Government
Actions
Official Statistics (Big) Data Strategy
Roadmap and Action Plan
European Commission Communication
"Towards a thriving data driven economy"
Private Public Partnership on big data
Data4Policy Initiative
Data Revolution at UN level
Other important areas relate to the policy / political framework and the regulatory framework. Given the interaction between policy and regulation, it is very important to work on these areas in parallel and in narrow cooperation.
One aspect of policy will be the integrating of (official) statistics into any strategy related to big data. This is essential to put statistics on the map and to open doors to actually accessing of data. It should be kept in mind that big data are often held or stored by private companies, e.g. mobile network operator.
The discussion of access is not limited to the entry but should include a long term vision, in other words a certain continuity of access – this is a conditio sine qua non for a sound statistical system that is based, fully or partially, on big data sources.
A main barrier to access, is data security and privacy concerns –as was also highlighted in the feasibility study carried out with respect to tourism statistics.
Another important challenge is finding a sustainable business model for big data in official statistics, taking into account the budgetary impact for statistical offices and for those "holding" the data.
To address these questions, I can mention that Eurostat recently launched a Call for Tender with the objective of analysing the legal frameworks at EU and national level.

14. Ethics / Communication
Policy
Quality
Skills
Experience sharing
Legislation IT
Infrastructures
Methods
Ethics / Communication
Partnerships
Pilots
Challenges
Transversal challenges to all big data activities: quality, methodology
Multiple sources for multiple outputs
Sound methodology ("from design- based to model-based approach")
Big data vs. statistics : "goodness of fit" (concepts, representativeness,…)
Actions
Cooperation with UN on quality and methodological framework for big data
Transversal topics in ESSnet pilots projects
Generalisation to frameworks
As I already mentioned, all of the areas in the roadmap are interrelated.
Two areas in particular are of a more horizontal, transversal nature.
On the one hand "quality"… the quality framework as we know it, will not be adapted to the new data sources. Eurostat is contributing to the UN's work on a quality framework for big data.
Quality issues will appear in the pilots, when assessing the access to data, etc. Just think of conceptual issues (can statistical definitions be maintained when using big data?), timeliness and flexibility of access, coverage and sampling issues, etc…
On the other hand "ethics and communication" will play an important if not decisive role. Policy makers and businesses will be reluctant to cooperate or to launch big data initiatives if the "public opinion" is not supporting such approaches. Protection of data will become even more important than it already is now.

15. Conclusions The official statistics community is
Joining forces to actively work on big data
Establishing partnerships
Looking for new insights
Finding additional uses and applications
Aware of quality considerations

16. "Data are not taken for museum purposes; they are taken as a basis for doing something. … The ultimate purpose of taking data is to provide a basis for action or a recommendation for action" W. Edwards Deming, 1942 "We are in the era of big data, and big data needs statisticians to make sense of it" Eric Schmidt and Jonathan Rosenberg, 2014