1. Microsoft Build 2016
9/15/2018 1:12 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Store and Serve PB of data with Azure Block Blobs
Session code
Store and Serve PB of data with Azure Block Blobs
Jason
Vamshi
Azure Storage Team

3. What we’ll cover today Brief Introduction to Azure Storage
Overview of Azure Blob Storage
Deep Dive into Azure Block Blobs
Key Concepts
Architecture and Internals
What’s New and Service Announcements
Best Practices and Design Patterns

4. Azure Scale
22 Regions ONLINE…huge datacenter capacity around the world…and we’re growing
UK (2)
TBA
Canada Central
Toronto
North Europe
Ireland
West Europe
Netherlands
Central US
Iowa
North Central US
Illinois
Canada East
Quebec City
US Gov
Iowa
Germany East
Magdeburg
China North
Beijing
East US
Virginia
Japan East
Saitama
West US
California
China South
Shanghai
East US 2
Virginia
Germany Central
Frankfurt
Japan West
Osaka
India Central
Pune
US Gov
Virginia
South Central US
Texas
India South
Chennai
India West
Mumbai
East Asia
Hong Kong
SE Asia
Singapore
Australia East
Sydney
Brazil South
Sao Paulo
Australia SE
Melbourne
Operational
1+ million servers
One of the top 3 networks in the world (coverage, speed, connections)
More regions than AWS and Google combined
US Gov (Operational)
Announced

5. Azure Storage Foundational Building Block of Azure Hyper Scale Open
Microsoft Build 2016
9/15/2018 1:12 PM
Azure Storage
Foundational Building Block of Azure
Azure Services: SQL Data Warehouse, HDInsight, Data Lake Store, Event Hubs, IoT Hubs…
Microsoft Services: Office 365, OneDrive, XBox, Skype…
Hyper Scale
>100 Trillion Objects, >13 million transactions per second
Open
REST API, Open sourced Client Libraries – .NET, Java, C++, Python, Node.js, iOS, Android, Xamarin…
Hybrid
Azure Stack in Windows Server 2016
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Azure Storage Services
Microsoft Build 2016
9/15/2018 1:12 PM
Azure Storage Services
Blobs
Highly scalable, REST based cloud object store
Block Blobs: Sequential file I/O
Page Blobs: Random-write pattern data
Tables
Massive auto-scaling NoSQL store
Dynamic scaling based on load
Scale to PBs of table data
Fast key/value lookups
Queues
Reliable queues at scale for cloud services
Decouple and scale components
Message visibility timeout and update message to protect against unreliable dequeuers
Disks
Persistent disks for Azure IaaS VMs
Built on page blobs
Premium Storage Disks: SSD based, high IOPS, low latency
Files
Fully Managed File Shares in the Cloud
Map to file share, standard file system semantics
“Lift and shift” legacy apps
Code against (REST API)
Use on Windows & Linux VMs
Azure Storage Documentation
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Azure Blob Storage 9/15/2018 1:12 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. What is the Blob Storage Service?
Object Storage: Store and serve unstructured data
App and Web scale data
Big Data from IoT, Genomics, etc.
Backups and Archive
Key Features
Store PBs of data
Durable – Choose level of durability: LRS, ZRS, GRS
Highly Available % SLA, 99.99% for reads (RA-GRS)
Strongly consistent
Dynamically scale up on bandwidth and TPS
Cost Effective

9. Microsoft Build 2016
9/15/2018 1:12 PM
Blob Storage Concepts
Microsoft Azure
Subscription 1
Storage Account 1
Container 1
Subscription 2
Storage Account 2
Container 2
Blobs
Subscription...n
Storage Account...n
Container…n
Contoso
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Types of Blobs Block Blobs Append Blobs Page Blobs
Microsoft Build 2016
9/15/2018 1:12 PM
Types of Blobs
Image.jpg
Block Blobs
Most object storage scenarios
Documents, images, video, etc.
Append Blobs
Multi-writer append only scenarios
Logging, Big Data Analytics output
Page Blobs
Page aligned random reads and writes
IaaS Disks, Event Hub, Block level backup
Block 1
Block 4
Block 3
Block 2
Block 1
Block 4
Block 3
Block 2
Sparse File
512 byte aligned
512
1024
1536
2048
2560
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Block Blobs: Deep Dive 9/15/2018 1:12 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Demo Block Blob Getting Started Sample Microsoft Build 2016
9/15/2018 1:12 PM
Demo
Block Blob Getting Started Sample
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Blob Names Account Name: 3-24 characters, lower case only
Microsoft Build 2016
9/15/2018 1:12 PM
Blob Names
Account Name: 3-24 characters, lower case only
Container Name: 3-63 characters, lower case only
Blob Name: characters, case sensitive
Virtual directories within blob namespace
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Blob Structure Standard Properties User Defined Metadata Blob Data
Blobtype: BlockBlob|PageBlob|AppendBlob
Etag & Last-Modified – Used for conditional operations
Content-Length & Content-MD5 – Size of blob in bytes and checksum
User Defined Metadata
Up to 8KB of unique names and associated values (valid HTTP headers)
No partial updates – Retrieve, then set
Can set on containers as well
Blob Data
Up to 200GB per blob

15. Uploading Block Blobs Single REST call up to 64 MB
Azure Storage
Image.jpg
(<64 MB)
Image.jpg
(<64 MB)
REST Calls
PutBlob(Image.jpg)
Single REST call up to 64 MB
New PutBlob will replace existing blob

16. Uploading Large Block Blobs
Azure Storage
Movie.mp4 (1 GB)
Movie.mp4
Movie.mp4
BL001
BL002
BL003
BLN00
Up to 4MB per block, variable size
Up to 50,000 blocks
BlockId – constant length per block
Benefits:
Efficient Continuation and Retry
Parallel, out of order uploads
REST Calls
blobName = “Movie.mp4”;
PutBlock(blobName, BL001, BL001Data);
PutBlock(blobName, BL002, BL002Data);
…………
PutBlock(blobName, BLN00, BLN00Data);
PutBlockList(blobName,
BL001,…,BL00N);

17. Modifying Block Blobs GetBlockList to get both block lists
Committed
Uncommitted
Committed
BL001
BL002
BL003
BL004
BL005
BL006
BL003
BL004
BL002
BL004
BL002
BL001
BL004
BL006
BL001
REST Calls
PutBlock(BL006);
PutBlock(BL003);
PutBlock(BL004);
PutBlock(BL002);
PutBlockList(
u=BL002,c=BL001,u=BL004,u=BL006, u=BL001);
GetBlockList to get both block lists
Blobs are mutable
Modify and delete blocks
Efficient duplicate data upload

18. Downloading and Copying Blobs
GetBlob
Returns blob properties and metadata in addition to data
Checksums in response allow data integrity validation
Range reads are supported
Can build parallel downloading of large blobs
CopyBlob
Full copy – Uncommitted blocks (and snapshots) are discarded
Copies can complete asynchronously
CopyId to check progress with GetBlob/GetBlobProperties or Abort Copy
Within an account, copies are fast
Copy from other external sources – eg – web sites, S3 etc

19. Listing and Deleting Blobs
ListBlobs
Container level operation
Up to 5000 blobs at a time, with NextMarker
Filtering – Prefix for “Begins with”
Traversing – Delimiter returns list of prefixes for use with filtering
DeleteBlob
Consider DeleteContainer for large numbers of Blobs

20. Client Libraries Simplify working with Azure Storage Services
Implement convenient features
Automatic chunking into blocks
Parallel uploads
Retry policies on failures
Client side encryption …

21. Versioning Snapshots Can be read, copied or deleted
Microsoft Build 2016
9/15/2018 1:12 PM
Versioning
Snapshots
Read only copy of a blob at a point in time
Same name as blob with a datetime stamp
Can be read, copied or deleted
CopyBlob to make a write-able new blob from a snapshot
CopyBlob to restore a snapshot over the current version
Snapshots are not copied with a normal CopyBlob operation
Blob Deletion requires deletion of all snapshots
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Concurrency Optimistic concurrency – Timestamps/ETags
Microsoft Build 2016
9/15/2018 1:12 PM
Concurrency
Optimistic concurrency – Timestamps/ETags
Timestamp based – If-Modified-Since and If-UnModified-Since
ETag based – If-Match and If-None-Match
Conditional update with supplied Timestamp or ETag will fail if conditions not met
Pessimistic Concurrency - Leases
Lease Blob for exclusive write and delete access
15-60s lease duration (can be renewed) or infinite lease (locks)
Can change lease id to acquire ownership in a chain/workflow
Can also acquire on containers to prevent container deletion
Last Writer wins
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Scaling on Azure Storage
Overview of Storage Architecture
Per account and per partition limits

24. Azure Storage Architecture
Microsoft Build 2016
9/15/2018 1:12 PM
Azure Storage Architecture
Front End
Stateless, front door for request handling (auth, request metering/throttling, validation)
Partition Layer
Serves data based on partitions, enables batch transactions and strong consistency
Stream Layer
Stores multiple replicas of the data, deals with failures, bit rot, etc.
FE 1
FE 2
FE 3
FE 4
FE 5
Partition 1
(A-C)
Partition 2
(D-E)
Partition 3
(F-J)
Partition 4
(K-T)
Partition 5
(U-Z)
Stream 1
Stream 2
Stream 3
Stream 4
Stream 5
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. What are Partitions? Most important concept in designing for scale
Storage is a range based partitioning system based on partition keys
Partition key is determined by you
Partition Key For Blobs:
Account Name + Container Name + Blob Name (in that order)
Every Blob is an independent partition

26. Partitions and Scale A partition is served by a single server
Limited by the server slice allocated to the blob
A server serves many partitions
Container and Blob Naming is important – Bias to different prefixes
Scale targets provide isolation in a multi tenant system
Single Blob Limits: 60MB/s
Account Limits: Up to 500TB, 20K tps, 20Gbps in/30Gbps out (for LRS)

27. Block Blobs: What’s New?
9/15/2018 1:12 PM
Block Blobs: What’s New?
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Changes in 2015 GA two new Azure Storage Services: For Block Blobs:
Azure Premium Disks
Azure Files
For Block Blobs:
Append Blobs*
Shared access signature improvements*
Android library GA
C++ library GA
Client side encryption libraries - .Net (GA) and Java (preview)
* Used in the demos in this talk

29. 2016 Announcements Client Libraries Encryption at Rest Preview*
Xamarin Library*
iOS Library
Python Library*
Encryption at Rest Preview*

30. 2016 Announcements – Cool Storage*
New “Blob Storage account” for Block Blobs
Two tiers
Hot – for commonly used data
Cool – for rarely used data
API is 100% identical; similar throughput and latency
Durability options: LRS, GRS, RA-GRS
Availability: Cool - 99%, Hot %. Higher for RA-GRS reads
Pricing to match your workload
Hot: Lower access prices for frequent use
Cool: Lower storage prices for high volume
Account level switch from Hot to Cool

31. Blob Storage Scenarios and Design Patterns
9/15/2018 1:12 PM
Blob Storage Scenarios and Design Patterns
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Data Management and Mobile Applications
Design challenges
Different device types
Limited local storage
Untrusted devices
Number of devices
Personal data
SAS Token
Blob Storage
https

33. Demo Azure Storage for Mobile Apps with Xamarin Microsoft Build 2016
9/15/2018 1:12 PM
Demo
Azure Storage for Mobile Apps with Xamarin
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

34. Blob Storage for your Apps and Services
Cost efficient data persistence
Support a broad set of scenarios
Mobile Apps – Xamarin, Android, iOS, WP
Web services – .NET, Java, Node.js
Web sites - .NET, Ruby, PHP
Scientific computing and Big Data – Python, C++
Archive and data management – AzCopy, DMLib, PowerShell, AzureCLI

35. Blob Storage Patterns Data Movement Delegated Authorization
Multi-writer Appends
Encryption at Rest
Full Text Search

36. Data Movement AzCopy tool simplifies data migration at scale
Microsoft Build 2016
9/15/2018 1:12 PM
Data Movement
AzCopy tool simplifies data migration at scale
Efficient means of copying millions of files – large or small
Includes journaling for reliability
Supports Blob Storage, Table Storage and File Storage
Data Movement library supports developers
Open source core framework from AzCopy
Enables custom applications to be developed
Storage Services
NAS
Storage Services
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

37. Demo Data Movement Library Microsoft Build 2016 9/15/2018 1:12 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

38. Delegated Authorization
Microsoft Build 2016
9/15/2018 1:12 PM
Delegated Authorization
SAS Tokens enable direct access by client apps
Supports least privilege principle
Removes the need to manage a data access layer
Mobile / IOT – Enable millions of untrusted clients
Web / Javascript – CORS + SAS enables website support
Token Issuer
Request / Renew
SAS Token
Storage Services
REST Request with
SAS token
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

39. Multi-writer Appends AppendBlob enables efficient appends to a blob
One REST call up to 4MB vs multiple REST calls for Block Blob
Enables concurrency management with multiple writers
Enables duplicate detection for single writers
SAS permissions include Append only permission
Blob Storage

40. Encryption at Rest Blob Storage Service Encryption (Preview)
Microsoft Build 2016
9/15/2018 1:12 PM
Encryption at Rest
Blob Storage Service Encryption (Preview)
Blob Storage Service automatically encrypts your data
SSL used to secure data sent to Storage Service
Microsoft manages all key management practices including compliance
Preview: Asia East, Japan East – More regions coming
Application Encryption
Applications can encrypt data prior to sending to Storage Service
Includes support for KeyVault based key management
Supports Blob, Table and Queue storage
Available in .NET and Java Storage Client libraries
KeyVault
SSL
Storage Services
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

41. Full Text Search
Azure Search enables full-text search for Blob Storage
Formats: PDF, Office, XML, ZIP, plain text, JSON
.NET SDK and Portal support coming soon
In public preview
Table Storage being evaluated
Contact:

42. Summary Azure Blob Storage:
Highly Scalable, Durable and Available
Secure
Open and multi-platform
Hybrid
Cost Effective
Documentation:
Re-visit Build on Channel 9
Continue your education at Microsoft Virtual Academy online

43. Please Complete An Evaluation Form Your input is important!
9/15/2018
Please Complete An Evaluation Form Your input is important! or
© 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.