Big Data and Official Statistics The UN Global Working Group http://unstats.un.org/unsd/trade/bigdata/ Ronald Jansen Chief, Trade Statistics Branch United Nations Statistics Division jansen1@un.org and BigData@un.org

Overview What is Big Data? What is Big Data for Official Statistics? What are the Challenges? UN Global Working Group on Big Data for Official Statistics

What is Big Data? Big Data characteristics Volume Velocity Variety Big Data benefits Auto-generated Timely Granular Big Data sources: Mobile Devices Digital Transactions Sensors Social Networking

Big Data for Development UN Global Pulse Research & Development Big Data Partnerships Pulse Lab Network New York Jakarta Kampala

Big Data for Development UN Global Pulse Early warning Real-time awareness Real-time feedback

What is Big Data for Official Statistics? Big Data sources / Statistics Mobile Phone Positioning Data / Tourism statistics Supermarket scanner data / Price statistics Vehicle tracking device / Transport statistics Twitter/ Consumer confidence statistics Satellite imagery / Agriculture statistics

Big Data Challenges Methodology Privacy IT infrastructure Human skills What are the Challenges? Big Data Challenges Methodology Privacy IT infrastructure Human skills Partnerships

Volatility (no time series?) Standardization ? Modelling Methodological Challenges Methodology Representativeness ? Volatility (no time series?) Standardization ? Modelling

Partnership Challenges Big Data partnerships opportunities Big Data providers Data Sources Research institutes and technology providers - Upstream processing - IT infrastructure, Academia and scientific communities - Analytical capacity and modelling National Statistical Offices - Standards and methodology

Example of use of Mobile Phone data for Official Statistics

Mobile Phone Positioning Data Call Detail Record (CDR) Data Detail Record (DDR) (Internet Protocol Detail Record) Location Updates (LA) Radio coverage updates (Abis data)

National Statistical Office Example 1: Estonia Data provider/ source Mobile phone operator Provides raw data to research company Research company Provides IT infrastructure Preparation of data Ensuring privacy and confidentiality Academia Providing modeling and analytics National Statistical Office Receives pre-processed and aggregated data Prepares and disseminates statistics Shared responsibility for data processing and analysis

National Statistical Office Example 2: Netherlands Responsibility for data Data source/ provider Vodafone Makes data available to research institute Research company Intermediate commercial partner Provides IT platform for data filtering and privacy National Statistical Office Receives pre-processed Big Data Analyze, prepare and disseminate statistics

UN Global Working Group on Big Data for Official Statistics

Milestones Feb 2013 Friday Seminar on Big Data Mar 2014 Report to Statistical Commission Decision 45/110 – GWG on Big Data Oct 2014 Conference on Big Data – Beijing Mar 2015 Report of GWG to Stat Commission Oct 2015 Conference on Big Data – Abu Dhabi

UN Global Working Group on Big Data Mandate (Decision 45/110 – 2014)   Provide strategic vision, direction and coordination of a global programme on Big Data for official statistics; Promote practical use of sources of Big Data for official statistics, while building on the existing precedents in Big Data and finding solutions for Methodological issues, Legal issues of access to data sources; Privacy issues Data security issues; Cost benefit analysis

UN Global Working Group on Big Data Mandate (continue)   Promote capacity building, training and sharing of experience; Foster communication and advocacy of use of Big Data for policy applications; Build public trust in the use of private sector Big Data for official statistics including issues of confidentiality and privacy.

International Organizations Composition of the GWG Countries Australia Denmark Italy Mexico Netherlands USA Bangladesh China Colombia Morocco Philippines Tanzania Egypt UAE Oman International Organizations ITU OECD UN Global Pulse UNECE UNESCAP UNSD World Bank

Task Teams Advocacy and Communication Big Data and SDG indicators Access and Partnerships Skills, Training, Capacity Building Cross-cutting issues (Quality framework) Mobile Phone Data Satellite Imagery Social Media Data

Strong participation Orange, Telenor, World Economic Forum, NASA, Google, Data Pop, World Pop, Flowminder, UNU-EHS, ODI (UK), Positium, UPenn, Harvard

Advocacy and Communication Advocacy/information materials on Big Data for official statistics, such as brochures, papers, presentations, videos, etc.; Good practices of using big data in official statistics, such as use cases; and Joint materials with other task teams to promote and communicate the outcomes of their work

Big Data and SDG indicators Assessment of the potential use of Big Data for monitoring progress on the 169 SDG targets by conducting a survey, taking account of the draft proposals of indicators, An inventory of research work on Big Data that could be used to estimate one or more SDGs, and Good practices and lessons learned from a pilot research in 1-2 countries on estimating 2-3 SDG indicators using Big Data.

Access and Partnerships Assessment of existing good practices of data access A set of principles as conditions for access to Big Data sources Templates for MOUs or services agreements Good practices and examples of partnerships

Training, Skills and Capacity Building Assessment tool for the needs of Big Data skills and for the institutional ‘readiness’ for Big Data skills; Corresponding training curriculum; Technical seminar on Big Data in 2015; and MOU for a global network of training institutions on Big Data

Cross-cutting issues Inventory of methodologies, data analytics /visualisation tools as well as quality assurance frameworks and refined classification of Big Data; Examples and good practices for the repository of Big Data projects; Assessment of Big data sources for the production of official statistics, determine their impact on the production process and identify best practices for managing the necessary changes within an NSO

Big Data sources Lessons learned from one or more pilot projects using mobile phone data / satellite imagery / social media data; Guidelines on the use of Big Data sources for official statistics, including template of a project plan

Big Data and Classifications

Classification of Type of Big Data Social Networks (human-sourced information) Traditional Business systems (process-mediated data) Internet of Things (machine-generated data)

Classification of Type of Big Data Social Networks (human-sourced information)   1100. Social Networks: Facebook, Twitter, Tumblr etc.   1200. Blogs and comments   1300. Personal documents   1400. Pictures: Instagram, Flickr, Picasa etc.   1500. Videos: Youtube etc.   1600. Internet searches   1700. Mobile data content: text messages   1800. User-generated maps   1900. E-Mail

Classification of Type of Big Data Traditional Business systems (process-mediated data) 21. Data produced by Public Agencies 2110. Medical records 22. Data produced by businesses 2210. Commercial transactions 2220. Banking/stock records 2230. E-commerce 2240. Credit cards

Classification of Type of Big Data Internet of Things (machine-generated data) 31. Data from sensors 311. Fixed sensors 3111. Home automation 3112. Weather/pollution sensors 3113. Traffic sensors/webcam 3114. Scientific sensors 3115. Security/surveillance videos/images 312. Mobile sensors (tracking) 3121. Mobile phone location 3122. Cars 3123. Satellite images 32. Data from computer systems 3210. Logs 3220. Web logs

Global survey (June-Aug 2015) Big data are data sources with a high volume, velocity and variety of data, which require new tools and methods to capture, curate, manage, and process them in an efficient way

Part I: Office Management of Big Data Big Data strategy Big Data strategy – Advocacy and Communication Big Data strategy – linking to the SDG indicators Access, privacy and confidentiality Part II: Current status of Big Data projects Data sources Technologies Quality assessment Part III: Guidelines on Big Data for Official Statistics

Ronald Jansen United Nations Statistics Division jansen1@un.org Questions ? Ronald Jansen United Nations Statistics Division jansen1@un.org