Download presentation
Presentation is loading. Please wait.
Published byClaire Fletcher Modified over 6 years ago
1
Microsoft Build 2016 9/15/2018 1:12 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
2
Store and Serve PB of data with Azure Block Blobs
Session code Store and Serve PB of data with Azure Block Blobs Jason Vamshi Azure Storage Team
3
What we’ll cover today Brief Introduction to Azure Storage
Overview of Azure Blob Storage Deep Dive into Azure Block Blobs Key Concepts Architecture and Internals What’s New and Service Announcements Best Practices and Design Patterns
4
Azure Scale 22 Regions ONLINE…huge datacenter capacity around the world…and we’re growing UK (2) TBA Canada Central Toronto North Europe Ireland West Europe Netherlands Central US Iowa North Central US Illinois Canada East Quebec City US Gov Iowa Germany East Magdeburg China North Beijing East US Virginia Japan East Saitama West US California China South Shanghai East US 2 Virginia Germany Central Frankfurt Japan West Osaka India Central Pune US Gov Virginia South Central US Texas India South Chennai India West Mumbai East Asia Hong Kong SE Asia Singapore Australia East Sydney Brazil South Sao Paulo Australia SE Melbourne Operational 1+ million servers One of the top 3 networks in the world (coverage, speed, connections) More regions than AWS and Google combined US Gov (Operational) Announced
5
Azure Storage Foundational Building Block of Azure Hyper Scale Open
Microsoft Build 2016 9/15/2018 1:12 PM Azure Storage Foundational Building Block of Azure Azure Services: SQL Data Warehouse, HDInsight, Data Lake Store, Event Hubs, IoT Hubs… Microsoft Services: Office 365, OneDrive, XBox, Skype… Hyper Scale >100 Trillion Objects, >13 million transactions per second Open REST API, Open sourced Client Libraries – .NET, Java, C++, Python, Node.js, iOS, Android, Xamarin… Hybrid Azure Stack in Windows Server 2016 © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
6
Azure Storage Services
Microsoft Build 2016 9/15/2018 1:12 PM Azure Storage Services Blobs Highly scalable, REST based cloud object store Block Blobs: Sequential file I/O Page Blobs: Random-write pattern data Tables Massive auto-scaling NoSQL store Dynamic scaling based on load Scale to PBs of table data Fast key/value lookups Queues Reliable queues at scale for cloud services Decouple and scale components Message visibility timeout and update message to protect against unreliable dequeuers Disks Persistent disks for Azure IaaS VMs Built on page blobs Premium Storage Disks: SSD based, high IOPS, low latency Files Fully Managed File Shares in the Cloud Map to file share, standard file system semantics “Lift and shift” legacy apps Code against (REST API) Use on Windows & Linux VMs Azure Storage Documentation © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
7
Azure Blob Storage 9/15/2018 1:12 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
8
What is the Blob Storage Service?
Object Storage: Store and serve unstructured data App and Web scale data Big Data from IoT, Genomics, etc. Backups and Archive Key Features Store PBs of data Durable – Choose level of durability: LRS, ZRS, GRS Highly Available % SLA, 99.99% for reads (RA-GRS) Strongly consistent Dynamically scale up on bandwidth and TPS Cost Effective
9
Microsoft Build 2016 9/15/2018 1:12 PM Blob Storage Concepts Microsoft Azure Subscription 1 Storage Account 1 Container 1 Subscription 2 Storage Account 2 Container 2 Blobs Subscription...n Storage Account...n Container…n Contoso © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
10
Types of Blobs Block Blobs Append Blobs Page Blobs
Microsoft Build 2016 9/15/2018 1:12 PM Types of Blobs Image.jpg Block Blobs Most object storage scenarios Documents, images, video, etc. Append Blobs Multi-writer append only scenarios Logging, Big Data Analytics output Page Blobs Page aligned random reads and writes IaaS Disks, Event Hub, Block level backup Block 1 Block 4 Block 3 Block 2 Block 1 Block 4 Block 3 Block 2 Sparse File 512 byte aligned 512 1024 1536 2048 2560 © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
11
Block Blobs: Deep Dive 9/15/2018 1:12 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
12
Demo Block Blob Getting Started Sample Microsoft Build 2016
9/15/2018 1:12 PM Demo Block Blob Getting Started Sample © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
13
Blob Names Account Name: 3-24 characters, lower case only
Microsoft Build 2016 9/15/2018 1:12 PM Blob Names Account Name: 3-24 characters, lower case only Container Name: 3-63 characters, lower case only Blob Name: characters, case sensitive Virtual directories within blob namespace © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
14
Blob Structure Standard Properties User Defined Metadata Blob Data
Blobtype: BlockBlob|PageBlob|AppendBlob Etag & Last-Modified – Used for conditional operations Content-Length & Content-MD5 – Size of blob in bytes and checksum User Defined Metadata Up to 8KB of unique names and associated values (valid HTTP headers) No partial updates – Retrieve, then set Can set on containers as well Blob Data Up to 200GB per blob
15
Uploading Block Blobs Single REST call up to 64 MB
Azure Storage Image.jpg (<64 MB) Image.jpg (<64 MB) REST Calls PutBlob(Image.jpg) Single REST call up to 64 MB New PutBlob will replace existing blob
16
Uploading Large Block Blobs
Azure Storage Movie.mp4 (1 GB) Movie.mp4 Movie.mp4 BL001 BL002 BL003 BLN00 Up to 4MB per block, variable size Up to 50,000 blocks BlockId – constant length per block Benefits: Efficient Continuation and Retry Parallel, out of order uploads REST Calls blobName = “Movie.mp4”; PutBlock(blobName, BL001, BL001Data); PutBlock(blobName, BL002, BL002Data); ………… PutBlock(blobName, BLN00, BLN00Data); PutBlockList(blobName, BL001,…,BL00N);
17
Modifying Block Blobs GetBlockList to get both block lists
Committed Uncommitted Committed BL001 BL002 BL003 BL004 BL005 BL006 BL003 BL004 BL002 BL004 BL002 BL001 BL004 BL006 BL001 REST Calls PutBlock(BL006); PutBlock(BL003); PutBlock(BL004); PutBlock(BL002); PutBlockList( u=BL002,c=BL001,u=BL004,u=BL006, u=BL001); GetBlockList to get both block lists Blobs are mutable Modify and delete blocks Efficient duplicate data upload
18
Downloading and Copying Blobs
GetBlob Returns blob properties and metadata in addition to data Checksums in response allow data integrity validation Range reads are supported Can build parallel downloading of large blobs CopyBlob Full copy – Uncommitted blocks (and snapshots) are discarded Copies can complete asynchronously CopyId to check progress with GetBlob/GetBlobProperties or Abort Copy Within an account, copies are fast Copy from other external sources – eg – web sites, S3 etc
19
Listing and Deleting Blobs
ListBlobs Container level operation Up to 5000 blobs at a time, with NextMarker Filtering – Prefix for “Begins with” Traversing – Delimiter returns list of prefixes for use with filtering DeleteBlob Consider DeleteContainer for large numbers of Blobs
20
Client Libraries Simplify working with Azure Storage Services
Implement convenient features Automatic chunking into blocks Parallel uploads Retry policies on failures Client side encryption …
21
Versioning Snapshots Can be read, copied or deleted
Microsoft Build 2016 9/15/2018 1:12 PM Versioning Snapshots Read only copy of a blob at a point in time Same name as blob with a datetime stamp Can be read, copied or deleted CopyBlob to make a write-able new blob from a snapshot CopyBlob to restore a snapshot over the current version Snapshots are not copied with a normal CopyBlob operation Blob Deletion requires deletion of all snapshots © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
22
Concurrency Optimistic concurrency – Timestamps/ETags
Microsoft Build 2016 9/15/2018 1:12 PM Concurrency Optimistic concurrency – Timestamps/ETags Timestamp based – If-Modified-Since and If-UnModified-Since ETag based – If-Match and If-None-Match Conditional update with supplied Timestamp or ETag will fail if conditions not met Pessimistic Concurrency - Leases Lease Blob for exclusive write and delete access 15-60s lease duration (can be renewed) or infinite lease (locks) Can change lease id to acquire ownership in a chain/workflow Can also acquire on containers to prevent container deletion Last Writer wins © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
23
Scaling on Azure Storage
Overview of Storage Architecture Per account and per partition limits
24
Azure Storage Architecture
Microsoft Build 2016 9/15/2018 1:12 PM Azure Storage Architecture Front End Stateless, front door for request handling (auth, request metering/throttling, validation) Partition Layer Serves data based on partitions, enables batch transactions and strong consistency Stream Layer Stores multiple replicas of the data, deals with failures, bit rot, etc. FE 1 FE 2 FE 3 FE 4 FE 5 Partition 1 (A-C) Partition 2 (D-E) Partition 3 (F-J) Partition 4 (K-T) Partition 5 (U-Z) Stream 1 Stream 2 Stream 3 Stream 4 Stream 5 © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
25
What are Partitions? Most important concept in designing for scale
Storage is a range based partitioning system based on partition keys Partition key is determined by you Partition Key For Blobs: Account Name + Container Name + Blob Name (in that order) Every Blob is an independent partition
26
Partitions and Scale A partition is served by a single server
Limited by the server slice allocated to the blob A server serves many partitions Container and Blob Naming is important – Bias to different prefixes Scale targets provide isolation in a multi tenant system Single Blob Limits: 60MB/s Account Limits: Up to 500TB, 20K tps, 20Gbps in/30Gbps out (for LRS)
27
Block Blobs: What’s New?
9/15/2018 1:12 PM Block Blobs: What’s New? © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
28
Changes in 2015 GA two new Azure Storage Services: For Block Blobs:
Azure Premium Disks Azure Files For Block Blobs: Append Blobs* Shared access signature improvements* Android library GA C++ library GA Client side encryption libraries - .Net (GA) and Java (preview) * Used in the demos in this talk
29
2016 Announcements Client Libraries Encryption at Rest Preview*
Xamarin Library* iOS Library Python Library* Encryption at Rest Preview*
30
2016 Announcements – Cool Storage*
New “Blob Storage account” for Block Blobs Two tiers Hot – for commonly used data Cool – for rarely used data API is 100% identical; similar throughput and latency Durability options: LRS, GRS, RA-GRS Availability: Cool - 99%, Hot %. Higher for RA-GRS reads Pricing to match your workload Hot: Lower access prices for frequent use Cool: Lower storage prices for high volume Account level switch from Hot to Cool
31
Blob Storage Scenarios and Design Patterns
9/15/2018 1:12 PM Blob Storage Scenarios and Design Patterns © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
32
Data Management and Mobile Applications
Design challenges Different device types Limited local storage Untrusted devices Number of devices Personal data SAS Token Blob Storage https
33
Demo Azure Storage for Mobile Apps with Xamarin Microsoft Build 2016
9/15/2018 1:12 PM Demo Azure Storage for Mobile Apps with Xamarin © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
34
Blob Storage for your Apps and Services
Cost efficient data persistence Support a broad set of scenarios Mobile Apps – Xamarin, Android, iOS, WP Web services – .NET, Java, Node.js Web sites - .NET, Ruby, PHP Scientific computing and Big Data – Python, C++ Archive and data management – AzCopy, DMLib, PowerShell, AzureCLI
35
Blob Storage Patterns Data Movement Delegated Authorization
Multi-writer Appends Encryption at Rest Full Text Search
36
Data Movement AzCopy tool simplifies data migration at scale
Microsoft Build 2016 9/15/2018 1:12 PM Data Movement AzCopy tool simplifies data migration at scale Efficient means of copying millions of files – large or small Includes journaling for reliability Supports Blob Storage, Table Storage and File Storage Data Movement library supports developers Open source core framework from AzCopy Enables custom applications to be developed Storage Services NAS Storage Services © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
37
Demo Data Movement Library Microsoft Build 2016 9/15/2018 1:12 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
38
Delegated Authorization
Microsoft Build 2016 9/15/2018 1:12 PM Delegated Authorization SAS Tokens enable direct access by client apps Supports least privilege principle Removes the need to manage a data access layer Mobile / IOT – Enable millions of untrusted clients Web / Javascript – CORS + SAS enables website support Token Issuer Request / Renew SAS Token Storage Services REST Request with SAS token © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
39
Multi-writer Appends AppendBlob enables efficient appends to a blob
One REST call up to 4MB vs multiple REST calls for Block Blob Enables concurrency management with multiple writers Enables duplicate detection for single writers SAS permissions include Append only permission Blob Storage
40
Encryption at Rest Blob Storage Service Encryption (Preview)
Microsoft Build 2016 9/15/2018 1:12 PM Encryption at Rest Blob Storage Service Encryption (Preview) Blob Storage Service automatically encrypts your data SSL used to secure data sent to Storage Service Microsoft manages all key management practices including compliance Preview: Asia East, Japan East – More regions coming Application Encryption Applications can encrypt data prior to sending to Storage Service Includes support for KeyVault based key management Supports Blob, Table and Queue storage Available in .NET and Java Storage Client libraries KeyVault SSL Storage Services © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
41
Full Text Search Azure Search enables full-text search for Blob Storage Formats: PDF, Office, XML, ZIP, plain text, JSON .NET SDK and Portal support coming soon In public preview Table Storage being evaluated Contact:
42
Summary Azure Blob Storage:
Highly Scalable, Durable and Available Secure Open and multi-platform Hybrid Cost Effective Documentation: Re-visit Build on Channel 9 Continue your education at Microsoft Virtual Academy online
43
Please Complete An Evaluation Form Your input is important!
9/15/2018 Please Complete An Evaluation Form Your input is important! or © 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.