5/14/2018 11:44 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Big Data Workloads with Azure Blob Storage 5/14/2018 11:44 AM Session Code Here Big Data Workloads with Azure Blob Storage Vamshidhar Kommineni Principal PM Manager Chris Brooks Principal Program Manager © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

What we’ll cover today Brief introduction to Azure Storage and Blob Storage Big Data workloads on Azure Integration of Blob Storage with Big Data services 2017 Roadmap for Blob Storage

Azure Storage Services Microsoft Build 2016 5/14/2018 11:44 AM Azure Storage Services IaaS Virtual machines Storage Networking PaaS Existing frameworks Web and mobile Serverless Compute Microservices Disks Persistent disks for Azure IaaS VMs Premium Storage Disks option: SSD based, high IOPS, low latency Files Fully Managed File Shares in the Cloud SMB and REST access “Lift and shift” legacy apps Blobs Highly scalable, REST based cloud object store Block Blobs: Sequential file I/O Cool Tier Available Page Blobs: Random-write pattern data Append Blobs Tables Massive auto-scaling NoSQL store Dynamic scaling based on load Scale to PBs of table data Fast key/value lookups Queues Reliable queues at scale for cloud services Decouple and scale components Message visibility timeout and update message to protect against unreliable dequeuers Built on a unified Distributed Storage System Durability, Encryption at Rest, Strongly Consistent Replication, Fault Tolerance, Auto Load-Balancing © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

What is the Blob Storage Service? Microsoft Build 2017 5/14/2018 11:44 AM What is the Blob Storage Service? Azure’s Object Storage platform Store and serve unstructured data App and Web scale data Backups and Archive Big Data from IoT, Genomics, etc. Types of Blobs Block Blobs - Most object storage scenarios Append Blobs - Multi-writer append only scenarios Page Blobs - Page aligned random reads and writes Block 1 Block 4 Block 3 Block 2 Image.jpg Sparse File 512 byte aligned 512 1024 1536 2048 2560 © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Blob Storage – Key Capabilities Microsoft Build 2017 5/14/2018 11:44 AM Blob Storage – Key Capabilities Strong consistency Multiple Redundancy types – LRS, GRS, RA-GRS Tiered Storage – Hot & Cool tiers Broad platform and language support © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Blob Storage – Key Technical Considerations Microsoft Build 2017 5/14/2018 11:44 AM Blob Storage – Key Technical Considerations Supports top level directory/isolation structure – “containers” Supports parallel “multipart” uploads and range gets Blobs are mutable Optimistic concurrency support with ETAG and timestamps Supports locks – Timed and infinite https://channel9.msdn.com/Events/Build/2016/B816 © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Analytics & Machine Learning Microsoft Build 2017 5/14/2018 11:44 AM Big Data on Azure Big Data Use Cases Ingest & ETL Streaming Analytics & Machine Learning Data Aggregation Presentation Machine Learning Stream Analytics Batch Functions Data Lake Analytics Data Factory App Insights Log Analytics Monitor IoT Hub Event Hubs Data Warehouse CDN Search Power BI Azure HDInsight Blob Storage Storage © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Big Data on Azure Blob Storage Microsoft Build 2017 5/14/2018 11:44 AM Big Data on Azure Blob Storage Geospatial GlobeSpotter - 360 degree, high resolution, highly accurate (10cm on position) street level views for every single street in the Netherlands, large portions of Germany, parts of Scandinavia and the US. “By having everything on the cloud in Azure, we can take a very data-intensive solution that is complex for customers to manage and turn it into an offering that they only need an Internet connection with a browser to access.” --Hans van Eijk: Global Product Strategy Director Case Study IoT 1.1 Million elevators in 150 countries. Connect ThyssenKrupp’s sensors and systems that monitor everything from motor temperature to shaft alignment, cab speed and door functioning to Azure “When the elevator reports that it has a problem, it sends out an error code and the three or four most probable causes of that error code. In effect, our field technician is being coached by this expert citizen.” --” Rory Smith, director of strategic development Case Study Genomics Stanford Medicine’s Center for Genomics and Personalized Medicine (SCGPM) adopts DNANexus’ solution on Azure to process PBs of genomics data with millions of core hours “By working closely with Microsoft though our relationship with Stanford, we now offer a tailored solution for incorporating Azure into state-of-the-art infrastructure for biomedical innovation,” --Richard Daly, CEO DNANexus Press Release © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Azure Storage Integration with Big Data services Microsoft Build 2017 5/14/2018 11:44 AM Demo Azure Storage Integration with Big Data services © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Big Data with Azure Blob Storage Microsoft Build 2017 5/14/2018 11:44 AM Big Data with Azure Blob Storage Big Data Use Cases Ingest & ETL Streaming Analytics & Machine Learning Data Aggregation Presentation Machine Learning Stream Analytics Batch Functions Data Lake Analytics Data Factory App Insights Log Analytics Monitor IoT Hub Event Hubs Data Warehouse CDN Search Power BI Azure HDInsight Blob Storage Blob Storage Pillars Open & Interoperable Manageable & Cost Efficient Scalable & Performant Secure & Compliant Durable & Available Storage © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Manageable & Cost Efficient Microsoft Build 2017 5/14/2018 11:44 AM Blob Storage Pillars Open & Interoperable Manageable & Cost Efficient Scalable & Performant Secure & Compliant Durable & Available © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

SDK/Tools Developers Tools Open Source client libraries on GitHub Retry/resume, Parallel upload/downloads, etc. Mobile platform support Data movement library High performance data transfers Cross platform compliant – CoreCLR based Tools Storage Powershell – Linux support coming soon CLI – Python based, Linux friendly AZCopy – Robocopy for Azure Storage Storage Explorer – Cross platform UX

SDK/Tools - New since Build 2016 Linux support .NET Client library on CoreCLR Data Movement Library on CoreCLR Azure Storage CLI 2.0 built on Python Mobile/Web support Xamarin & Android library GA Node.js with support for Typescript GA Javascript library preview Python client library preview Other Azure Storage Ruby library for Cloud Foundry Client library and tools support for new service features

AzCopy on Linux Familiar Unix syntax Built on CoreCLR and Storage Data Movement Library Fast data transfer with pause/resume Preview available now - https://aka.ms/downloadazcopyprlinux

Manageable & Cost Efficient Microsoft Build 2017 5/14/2018 11:44 AM Blob Storage Pillars Open & Interoperable Manageable & Cost Efficient Scalable & Performant Secure & Compliant Durable & Available © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Storage Metrics/Logs - Azure Monitor Integration Microsoft Build 2017 5/14/2018 11:44 AM Storage Metrics/Logs - Azure Monitor Integration Benefits Access from unified Azure Monitor APIs Setup charts and alerts based on metrics Archive analytics data into storage account Stream analytics data to Event Hub, OMS, etc. Features Capacity and Transaction Metrics Diagnostic Logs Access from Azure Portal, REST, SDK, Powershell, CLI Timeline Limited Preview: Starting this month © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data Protection Recycle Bin – Recovery from Microsoft Build 2017 5/14/2018 11:44 AM Data Protection Recycle Bin – Recovery from accidental deletions/overwrites Configurable Number of days to retain (max: 365) Number of overwritten versions (max: 10) Can be turned off Blob level DeleteBlob or PutBlob/PutBlockList results in an entry GA: Q3 CY17 Future: Object Versioning, WORM support © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Lower Transaction cost Microsoft Build 2017 5/14/2018 11:44 AM Cool Storage Blob Tiers Hot – for commonly used data Cool – for rarely used data (GA: 4/16) API is 100% identical; similar throughput and latency Same durability options: LRS, GRS, RA-GRS Pricing to match your workload Hot: Lower access prices for frequent use Cool: Lower storage prices for high volume Switch account tiers as needed Blob REST API Blob Storage Account Hot Tier Lower Transaction cost Cool Tier Lower Capacity cost Account level conversion © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Lower Transaction cost Microsoft Build 2017 5/14/2018 11:44 AM Tiered Storage Object Level Tiering Individual blobs can move between tiers All tiers of blobs co-exist in a storage account Blob REST API Hot Blobs Lower Transaction cost Cool Blobs Lower Capacity cost Blob Storage Account Object-Level Tiering © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Lower Transaction cost Microsoft Build 2017 5/14/2018 11:44 AM Tiered Storage Object Level Tiering Individual blobs can move between tiers All tiers of blobs co-exist in a storage account New Storage Tier – Archive Storage Cold storage for long term data Retrieval latency is hours Consistent API among Storage tiers Access through Blob REST API Support direct writes to Archive Timeline Preview coming soon – AzStorageFeedback@microsoft.com Blob REST API Hot Blobs Lower Transaction cost Blob Storage Account Cool Blobs Lower Capacity cost Archive Blobs Lowest cost Offline storage Object-Level Tiering © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5/14/2018 11:44 AM Tiered Storage © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Tiered Storage New API to set blob tier: SetBlobTier 5/14/2018 11:44 AM Tiered Storage New API to set blob tier: SetBlobTier Acknowledged immediately from service Get APIs return current tier and archive status New headers "x-ms-access-tier” and “x-ms-archive-status” GetBlobMetaData GetBlobProperties ListBlobs Future: Automated Lifecycle Management © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Big Data with Azure Blob Storage Microsoft Build 2017 5/14/2018 11:44 AM Big Data with Azure Blob Storage Big Data Use Cases Ingest & ETL Streaming Analytics & Machine Learning Data Aggregation Presentation Machine Learning Stream Analytics Batch Functions Data Lake Analytics Data Factory App Insights Log Analytics Monitor IoT Hub Event Hubs Data Warehouse CDN Search Power BI Azure HDInsight Blob Storage Blob Storage Pillars Open & Interoperable Manageable & Cost Efficient Scalable & Performant Secure & Compliant Durable & Available Storage © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Azure Storage Integration with Big Data services Microsoft Build 2017 5/14/2018 11:44 AM Demo Azure Storage Integration with Big Data services © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Big Data with Azure Blob Storage Microsoft Build 2017 5/14/2018 11:44 AM Big Data with Azure Blob Storage Big Data Use Cases Ingest & ETL Streaming Analytics & Machine Learning Data Aggregation Presentation Machine Learning Stream Analytics Batch Functions Data Lake Analytics Data Factory App Insights Log Analytics Monitor IoT Hub Event Hubs Data Warehouse CDN Search Power BI Azure HDInsight Blob Storage Blob Storage Pillars Open & Interoperable Manageable & Cost Efficient Scalable & Performant Secure & Compliant Durable & Available Storage © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Hadoop File System and Azure Storage Microsoft Build 2017 5/14/2018 11:44 AM Hadoop File System and Azure Storage Applications Hadoop Shell Commands Hadoop FileSystem API Specification HDFS Azure Storage Integration (WASB) … Coming soon to WASB: Improved support for HDFS Flush and Sync semantics Support for larger blobs and other new features Azure Blob Storage © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Manageable & Cost Efficient Microsoft Build 2017 5/14/2018 11:44 AM Blob Storage Pillars Open & Interoperable Manageable & Cost Efficient Scalable & Performant Secure & Compliant Durable & Available © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Storage Blobs Durability & Availability Microsoft Build 2017 5/14/2018 11:44 AM Azure Storage Blobs Durability & Availability Strong Consistency 3 replicas + erasure coding Data Integrity MD5 hash on ingress/egress CRC checksum & “bit rot” protection Disaster Recovery (BCDR) Geo-redundant storage (GRS) High Availability (HA) 99.9% availability SLA 99.99% for reads with RA-GRS © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Manageable & Cost Efficient Microsoft Build 2017 5/14/2018 11:44 AM Blob Storage Pillars Open & Interoperable Manageable & Cost Efficient Scalable & Performant Secure & Compliant Durable & Available © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Encryption At Rest – Storage Service Encryption Microsoft Build 2017 5/14/2018 11:44 AM Encryption At Rest – Storage Service Encryption Prevents data disclosure from physical disk compromise Available now for Blobs with Microsoft managed keys H2 2017 – Encryption enabled for all accounts Customer managed encryption keys - Preview H2 2017 © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Encryption In Transit Storage REST APIs support HTTPS Microsoft Build 2017 5/14/2018 11:44 AM Encryption In Transit Storage REST APIs support HTTPS SAS Tokens can be restricted for HTTPS only New - “Secure Transfer” option Limit all access to HTTPS only Enables control via ARM Policy and monitoring via Azure Security Center Available in H2 of 2017 © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Network Access Control Microsoft Build 2017 5/14/2018 11:44 AM Network Access Control Layered security for Storage Protection from key disclosure threats Limit access to specific Azure VNETs or external internet IP address ranges Preview in H2 2017 © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Apache Ranger Integration Microsoft Build 2017 5/14/2018 11:44 AM Apache Ranger Integration Granular access control policies in Hadoop Now including Azure Storage Blobs through WASB driver Secure HDInsight Preview July 2017 © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

AAD Authentication and RBAC for Blob Storage Microsoft Build 2017 5/14/2018 11:44 AM AAD Authentication and RBAC for Blob Storage Currently support AAD, OAuth and RBAC on Storage Resource Provider via ARM Adding AAD and OAuth support to Blob Storage REST API Integrating with Azure RBAC for Authorization Role assignments down to container scope Preview in H2 2017 © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Manageable & Cost Efficient Microsoft Build 2017 5/14/2018 11:44 AM Blob Storage Pillars Open & Interoperable Manageable & Cost Efficient Scalable & Performant Secure & Compliant Durable & Available © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

VM Storage Performance & Scale For IaaS based big data workloads 5/14/2018 11:44 AM VM Storage Performance & Scale For IaaS based big data workloads Preview in West Central US with CLI/PS support Azure Portal coming soon Up to 4TB Up to 1TB Now Premium: 128GB, 512GB, 1 TB Standard: 32 GB, 64 GB, 128 GB, 512 GB, 1 TB June 2017 4TB Disk Size for Premium and Standard (7,500IOPS, 250MBps per Premium Disk) Max disk size per VM 256TB Max IOPS per VM 80,000 IOPS Max throughput per VM 2GB/s Additional smaller disk sizes for Premium: 32GB, 64GB © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Individual Blob Performance & Scale Improvements Microsoft Build 2017 5/14/2018 11:44 AM Individual Blob Performance & Scale Improvements Large Block Blobs Single Blob max increased 25x to 5TB Write throughput improvements Small Block Blobs Improved read latency for blobs <4MB 5 TB 200 GB © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Storage Account Scalability Microsoft Build 2017 5/14/2018 11:44 AM Storage Account Scalability Preview this summer 5+ PB storage capacity 100k+ Requests/sec 100+ Gbps ingress and egress bandwidth Optimize for rapid storage scale out for HDInsight workloads 5+ PB 500 TB © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Big Data with Azure Blob Storage Microsoft Build 2017 5/14/2018 11:44 AM Big Data with Azure Blob Storage Big Data Use Cases Ingest & ETL Streaming Analytics & Machine Learning Data Aggregation Presentation Machine Learning Stream Analytics Batch Functions Data Lake Analytics Data Factory App Insights Log Analytics Monitor IoT Hub Event Hubs Data Warehouse CDN Search Power BI Azure HDInsight Blob Storage Blob Storage Pillars Open & Interoperable Manageable & Cost Efficient Scalable & Performant Secure & Compliant Durable & Available Storage © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5/14/2018 11:44 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.