NIST BIG DATA WG Reference Architecture Subgroup Intermediate Report Co-chairs: Orit Levin (Microsoft) James Ketner (AT&T) Don Krapohl (Augmented Intelligence) July 24th, 2013

Reference Architecture Objectives Addresses a broad range of stakeholders (e.g., data owners, industries, academia, policy makers) Wide scope: Encompasses the whole data life cycle or in the ecosystem Can be applied to different use cases (including various verticals) Represents different system architectures (e.g., an enterprise data warehouse, distributed cloud-based system using multiple service providers) Focus Potentially with initial focus on the Big Data analytics and tools Assists in identifying security and privacy issues Agnostic to any specific technologies 2 7/24/2013NIST Big Data WG / Ref Arch Sub-group

RA Diagram Independent Submissions Different styles and perspectives, but easy to map between them Data centric (Wo Chang) Data Flow centric (Orit Levin, Bob Marcus) Technology Layers / Stack diagram (Gary Mazzaferro) The vocabulary used in these submissions and on the mailing list has been compiled and submitted as M /24/2013NIST Big Data WG / Ref Arch Sub-group

Abstract Reference Architecture by Wo Chang / NIST 7/24/2013NIST Big Data WG / Ref Arch Sub-group4

Independent RA Proposals: Big Data Sources, Usage, Transformation, and Infrastructure 7/24/2013NIST Big Data WG / Ref Arch Sub-group5 Data Flow Diagram by Bob Marcus Technology Stack / Layers Diagram by G. Mazzaferro Data Flow Ecosystem Diagram by Orit Levin

Data Sources and Usage 7/24/2013NIST Big Data WG / Ref Arch Sub-group6 Data Flow Diagram by Bob Marcus Technology Stack / Layers Diagram by G. Mazzaferro Data Flow Ecosystem Diagram by Orit Levin

Infrastructure: Storage, Security, and Management 7/24/2013NIST Big Data WG / Ref Arch Sub-group7 Data Flow Diagram by Bob Marcus Technology Stack / Layers Diagram by G. Mazzaferro Data Flow Ecosystem Diagram by Orit Levin

Data Transformation: Processing, Analytics, and Visualization 7/24/2013NIST Big Data WG / Ref Arch Sub-group8 Data Flow Diagram by Bob Marcus Technology Stack / Layers Diagram by G. Mazzaferro Data Flow Ecosystem Diagram by Orit Levin

Draft Agreement / Rough Consensus Transformation includes Processing functions Analytic functions Visualization functions Data Infrastructure includes Data stores In-memory DBs Analytic DBs 7/24/2013NIST Big Data WG / Ref Arch Sub-group9 Sources Transformation Usage Data Infrastructure Security Management Cloud Computing Network

Next Steps and AIs Deliverable I: Write the White Paper draft showing one or more (e.g., Data Flow and Stack approaches) using the same or similar terminology AI: Chairs will start the draft of the document incorporating the submissions to the Ref Arch subgroup AI: Close cooperation between “Ref Arch” and “Def&Tax” sub-groups to produce the Output: taxonomy for the RA diagrams with definitions for major entities/blocks; Input: M Deliverable II: A draft of a single RA requires more discussion and inputs based on the work of all sub-groups AI: Chairs will start the draft of the document incorporating the findings of the Ref Arch subgroup AI: Review the latest contributions to the Ref Arch and incorporate their findings (See from Yuri Demchenko / University of Amsterdam) AI: Close cooperation with the “Use Cases” and “Security” sub-groups to identify the areas of focus for “zooming” into their architecture 10 7/24/2013NIST Big Data WG / Ref Arch Sub-group

Backup Slides 11 7/24/2013NIST Big Data WG / Ref Arch Sub-group

Submitted RAs 127/24/2013NIST Big Data WG / Ref Arch Sub-group

Data Centric by Wo Chang / NIST 7/24/2013NIST Big Data WG / Ref Arch Sub-group13

Data Flow Diagram by Bob Marcus 14 7/24/2013NIST Big Data WG / Ref Arch Sub-group

Individual Data Transfer Big Data Transfer Selected Data Storage and Retrieval Big Data Storage and Retrieval Aggregation Data Objects Data Sources Data Usage Government (incl. health & financial institutions) Industries / Businesses Network Operators / Telecom Academia Data Mining Matching Collection Data Transformation Data Infrastructure Storage & Retrieval Management Security Conditioning Anonymized Pseudo- anonymized PII VOLUME VARIETY VELOCITY Aggregation 15 Data Flow Ecosystem Diagram by Orit Levin 7/24/2013NIST Big Data WG / Ref Arch Sub-group

Technology Layers / Stack diagram by Gary Mazzaferro M i c r o s o f t 16 7/24/2013NIST Big Data WG / Ref Arch Sub-group

Mapping to Technologies and Use Cases Prepared by the authors of the original RAs 7/24/2013NIST Big Data WG / Ref Arch Sub-group17

18 7/24/2013NIST Big Data WG / Ref Arch Sub-group

19 An Example of Cloud Computing Usage in Big Data Ecosystem Individual Data Transfer Big Data Transfer Selected Data Storage and Retrieval Big Data Storage and Retrieval Aggregation Data Objects Data Sources Data Usage Government (incl. health & financial institutions) Industries / Businesses Network Operators / Telecom Academia Data Mining Collection Data Transformation Data Infrastructure VOLUME VARIETY VELOCITY Data Warehouse Cloud Provider / Service Layer Cloud Provider / Service Layer SaaS PaaS IaaS Matching 7/24/2013NIST Big Data WG / Ref Arch Sub-group

Online Data Aggregator Data Subject / Person Online Sources Public Records (commons, government, etc.) Offline Sources Internal RecordsOther devices (Smart Grid, surveillance, scientific, etc.) End User devices incl. OS (mobile phones, etc.) Applications (search, publishers, etc.) Match/Bridge Service Networks Government, health, financial institutions, academia Industries / Businesses Network Operators Collection DataManagementPlatforms(DMPs) UI: Do Not Track (DNT) HTTP: DNT Analytic Cookie DMP Cookie DPI Match Cookie Appl. with customers (communications, social network, etc. Match Container Tag or Pixel request Offline Data Aggregator Web Browsers Data Mining Person Attribution Users SSP DSP AdNet AdXAgency Publisher Advertiser Advertising Industry Ecosystem DMP Container Tag or Pixel request Control Aggregated 1 st Party 2 nd Party De-identified PII 3 rd Party Contextual Data Collection Behavioral Data Creation Big Data Transfer Individual Data Transfer 20 Use Case: Advertising 7/24/2013NIST Big Data WG / Ref Arch Sub-group

Individual Data Transfer Big Data Transfer Selected Data Storage and Retrieval Big Data Storage and Retrieval Online Analytical Processing (OLAP) Data Usage Department Data Mart Regional Data Mart Subject Data MartApplication Data Mart Data Mining / Knowledge Discovery in Databases (KDD) Extraction, Transformation, and Loading (ETL) Data Transformation Data Infrastructure Central Data Warehouse Management Security Archives FilesOnline Transaction Processing (OLTP) Systems MS Office Documents Functional Data Mart Operational Data Store Staging Area Data Sources Manual Managed Report Environment (MRE) Data Objects 21 Use Case: Enterprise Data Warehouse 7/24/2013NIST Big Data WG / Ref Arch Sub-group

7/24/2013NIST Big Data WG / Ref Arch Sub-group22