5/9/2018 7:28 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demystifying Cloud Data Services for an App Developer 5/9/2018 7:28 AM P4016 Demystifying Cloud Data Services for an App Developer Romit Girdhar Software Engineer Wee Hyong Tok Program Manager © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Evolving Requirements Evolving Data Processing Requirements VALUE Growth of data Machine learning and AI Any data In-memory Internet of Things INTERNET CONNECTED Hadoop Dashboards Ad hoc analysis Operational reporting CLOUD DIGITAL ANALOG Enterprise data warehouse OLAP ETL MOBILE [wh] Good day, everyone! Thank you for tuning in to this session. As we work with customers, one of the things we have been hearing a lot is the evolving data processing requirements. From transactional systems, to doing ETL and bringing the data into an enterprise data warehouse To wanting to reduce the end to end latency to derive insights (from days to minutes, to even seconds). And to be able to do this @ scale… working on big data And wanting to build intelligent systems using Machine Learning/AI on data of various shapes and forms… This session will help demystify the data services that you can use to achieve all of this… Super exciting! Complex implementations Spreadmarts Siloed data Transactional systems Evolving Requirements

Transforming Data into Intelligent Action 5/9/2018 7:28 AM Transforming Data into Intelligent Action EXTRACT Original data TRANSFORM ETL tool (SSIS, etc.) LOAD (SQL Sever, Teradata, etc.) EDW Transformed data OLTP, ERP, LOB, ... BI tools Data marts Apps Dashboards (On-premise and in the cloud) INGEST Original data Scale-out, storage, and compute (HDFS, Blob storage, etc.) TRANSFORM AND LOAD Devices, social, sensors, web #MSBuild © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Common Components of Data-driven Solutions Microsoft Build 2017 5/9/2018 7:28 AM Common Components of Data-driven Solutions Ingestion – Incoming data Processing – Analyze/Act on your data Storing – Access your raw/processed data Serving – Serve your data © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Build 2017 5/9/2018 7:28 AM Ingestion Patterns Real-time Ingestion - Capable of ingesting millions of messages/sec Near-real time Ingestion – Data received from web forms, etc., ingesting less than 1000msg/sec One-time Load Scheduled Periodic Loads [romit] © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Build 2017 5/9/2018 7:28 AM Processing Patterns Real-time Processing – Need to process each event as it arrives. Batch Processing – Process large amounts of data on a periodic schedule [romit] © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data Serving Patterns Dashboarding/Reporting Microsoft Build 2017 5/9/2018 7:28 AM Data Serving Patterns Dashboarding/Reporting Ad-hoc Data Access – Access your data in your app Interactive Data Access – Slice/Dice, Drill-down © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Build 2017 5/9/2018 7:28 AM Data Storage Patterns OLTP/Hot Store – Great for quick access of data & light-weight Interactive queries Warm Store – Cheap to store ; Great for batch processing Archival/Cold Store – Cheap to store ; Data only needs to be accessed as an exception. OLAP/Analytical Store – Similar to a hot store, but, provides fast and interactive access to your data © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Types of Data Stores Requirements Transient / Staging Big Data Microsoft Build 2017 5/9/2018 7:28 AM Types of Data Stores Transient / Staging Big Data Relational Non-Relational JSON Documents Column Key-Value Graph Time-series Requirements Data Volume and Velocity Real time vs Batch Read/Write Latency Types of data to be stored © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Solution Patterns 5/9/2018 7:28 AM [wh] © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Solution Pattern #1 - Real-time Serving Microsoft Build 2017 5/9/2018 7:28 AM Solution Pattern #1 - Real-time Serving Hot/ OLTP Store Application Inbound Data Application Cold/ Archival © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Solution Pattern #1 - Real-time Serving Microsoft Build 2017 5/9/2018 7:28 AM Solution Pattern #1 - Real-time Serving Hot/ OLTP Store Application Inbound Data Application Cold/ Archival Ingest Serve and Consume Store © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Solution Pattern #2 – Real-time Processing Microsoft Build 2017 5/9/2018 7:28 AM Solution Pattern #2 – Real-time Processing Warm Store (for later processing) Inbound Data Message Queue Near real-time processing engine Dashboarding/Reporting Hot/OLTP Store Application © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Solution Pattern #2 – Real-time Processing Microsoft Build 2017 5/9/2018 7:28 AM Solution Pattern #2 – Real-time Processing Warm Store (for later processing) Inbound Data Message Queue Near real-time processing engine Dashboarding/Reporting Hot/OLTP Store Application Serve and Consume Ingest Process Store © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Solution Pattern #3 - Batch Processing Microsoft Build 2017 5/9/2018 7:28 AM Solution Pattern #3 - Batch Processing Reports and Dashboard Other Data Sources Data Processing Advanced Analytics OLAP/Analytical Store Application Inbound Data Data in the warm store © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

OLAP/Analytical Store Microsoft Build 2017 5/9/2018 7:28 AM Solution Pattern #3 Reports and Dashboard Other Data Sources Data Processing [romit] Advanced Analytics OLAP/Analytical Store Application Inbound Data Data in the warm store Process Ingest Store Serve and Consume © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

What are my Technology Options? 5/9/2018 7:28 AM What are my Technology Options? [romit] © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

What are my tech options? Microsoft Build 2017 5/9/2018 7:28 AM What are my tech options? Callouts: This is not an exhaustive list (we’ve listed the common ones) SQL DW is a big data processing tool Hot store and Analytical stores are similar, though some like AAS are better for interactive querying Benefit that Hadoop & Spark bring to the table © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Let’s test our knowledge… Microsoft Build 2017 5/9/2018 7:28 AM Let’s test our knowledge… ? ? [romit] Support Call Logs Service Telemetry Data Social Data Ingest Process Store Serve and Consume © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Let’s test our knowledge… Microsoft Build 2017 5/9/2018 7:28 AM Let’s test our knowledge… Power BI Event Hubs Stream Analytics Real-time Insights and Alerts Federated U-SQL Join Query (Reference Data) Intelligence @ Scale Built-in Cognition Capabilities Azure Data Lake Store and Analytics Azure SQL Database Azure SQL Data Warehouse PolyBase over Azure Data Lake Store Azure Analysis Services [romit] -> Support Call Logs Service Telemetry Data Social Data Ingest Process Store Serve and Consume © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

How should I get started? 5/9/2018 7:28 AM How should I get started? [romit] © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data Processing Flow – Real-time processing Microsoft Build 2017 5/9/2018 7:28 AM Data Processing Flow – Real-time processing Call-outs: For ASA, the only queue that currently works is EventHubs Storm/Spark as supported as a part of HDInsight “this” data means data in the current state/whether raw or processed Difference/similarities between “Hot” store and “Serving” Store © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data Processing Flow – Batch Processing Microsoft Build 2017 5/9/2018 7:28 AM Data Processing Flow – Batch Processing Yes No No No Callouts: - Partition your “big” data. For the most part, by Date-time © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Call to action Get started in minutes http://aka.ms/cisolutions Reference architecture for common scenarios Built on best practice design patterns Automated deployment on your Azure subscription Customizable for your needs Supported by a global partner ecosystem Code Labs – Building your First App https://github.com/Microsoft-Build-2016/CodeLabs-Data Get started in minutes

Related sessions Learn more about building Data-driven Apps for Scale: Microsoft Build 2017 5/9/2018 7:28 AM Related sessions Learn more about building Data-driven Apps for Scale: B8040: How JCI built next-gen Data-driven applications at scale Thursday, May 11th, 2017: 4.00pm – 5.00pm PST B8081: How to serve AI with Data: The future of the data platform Wednesday, May 10th, 2017: 2.00pm – 3.00pm PST [Live streaming available!] B8018: How to build global scale IoT applications with Azure SQL DB Thursday, May 11th, 2017: 3.30pm – 4.30pm PST Code Stories Theater sessions #MSBuild © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5/9/2018 7:28 AM © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.