1. Analytics on Azure What to Use When Christina E. Leo
Cloud Solution Architect | Data & AI | One Commercial Partner | United Kingdom
@christinaleo

3. Why is this important? There is a lot of data 30 years CLOUD CLOUD
1985
1990
1995
2000
2005
2010
2015
2020
Why is this important?
30 years
CLOUD
CLOUD
There is a lot of data
MOBILE
MOBILE
INTERNET CONNECTED
DIGITAL
ANALOG

4. Data gives us new ways of knowing and new things to know.
Data is important
Data makes software smarter through AI. It is how machines know.

5. It is a huge opportunity
$1.2tr Estimated additional revenue/shifting revenue driven by AI in three years
“$2.5tr Potential contribution to the global economy in Europe by 2030 from AI”
- PwC analysis
It is a huge opportunity
now!
“According to ‘Research and Markets’, the global chat bot market is set to reach $1.2B by 2025.”
~ CognitionX News letter.
AI Opportunity
“Natural language processing among technology segment are expected to dominate the market”
- Stratistics MRC
“38% Boost in rates of profitability in developed nations by 2034 through AI potential.” ~ Accenture
“40% boost in labour productivity in developed nations by 2035 through AI potential” ~ Accenture

6. The Azure Data Landscape
11/12/2019 7:31 PM
The Azure Data Landscape
People
Automated Systems
Apps
Web
Mobile
Bots
Data Sources
Sensors and devices
Orchestration
Data Catalog
Data Factory
Event Hubs
Storage Blob
Table Storage
Integrations Services
Data Storage
Data Lake Store
CosmosDB
SQL Server 2017
SQL Data
Warehouse
SQL Azure
SQL Managed Instance
Machine Learning
Data Science &
Deep Learning VM
Studio
Azure ML Services
ML Development Environments
Server
Prebuilt Models
Cognitive
Services
Operationalization
Bot Framework
Functions & Serverless
Azure Search
Big Data & Analytics
HDInsight
Stream Analytics
Analysis
Services
Dashboards & Visualizations
Power BI
IoT
IoT Hub
DataBricks
Data Storage Intelligence Action
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Solution Scenarios Big Data & Advanced Analytics
Modern Data Warehousing
“We want to integrate all our data including ‘big data” with our data warehouse”
Real-Time Analytics
“We are trying to get insights from our devices in real-time, etc.”
Advanced Analytics
“We are trying to predict when our customers churn.”

8. Uses of Data BI AA AI Value Descriptive Diagnostic Predictive
11/12/2019 7:31 PM
Uses of Data BI AA AI
Value
Descriptive
Diagnostic
Predictive
Prescriptive
Data Engineering, Warehousing & Preparation
Descriptive analytics — the "simplest class of analytics," said Lithium Technologies' chief scientist Michael Wu — is your raw data in summarized form. It's your social engagement counts, sales numbers, customer statistics and other metrics that show you what's happening in your business in an easy-to-understand way.
Predictive and prescriptive analytics are the next steps that help you turn descriptive metrics into insights and decisions. 
"Prescriptive analytics provide intelligent recommendations for the optimal next steps for almost any application or business process to drive desired outcomes or accelerate results.“
"Predictive analytics forecasts what will happen in the future. Prescriptive analytics can help companies alter the future,“
"Prescriptive analytics builds on [predictive] by informing decision makers about different decision choices with their anticipated impact on a specific key performance indicators," said Thomas Mathew, chief product officer at influencer engagement platform Zoomph. "Think of [traffic navigation app] Waze. Pick an origin and a destination — a multitude of factors get mashed together, and [it advises] you on different route choices, each with a predicted ETA. This is everyday prescriptive analytics at work."
Dashboards and Reports
Detailed Statistics
Machine Learning
Deep Learning
Analyst
Data Scientist
Data Engineer
ML Engineer
AI Engineer
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Modern Data Warehousing
The Modern Data Warehouse extends the scope of the data warehouse to serve “big data” that is prepared with techniques beyond relational ETL.
Modern Data Warehousing
“We want to integrate all our data including ‘big data” with our data warehouse”
Real-Time Analytics
“We are trying to get insights from our devices in real-time, etc.”
Advanced Analytics
“We are trying to predict when our customers churn.”

10. Modern Data Warehousing Canonical Operations
Tech Ready 15
Modern Data Warehousing Canonical Operations
11/12/2019
Ingest
Transfer, Store
Load & Ingest
Process
Process, Clean
Serve
Serve, Analyze
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Data Warehousing Pattern IN Azure
Tech Ready 15
Data Warehousing Pattern IN Azure
11/12/2019 7:31 PM
Loading and preparing data for analysis with a data warehouse
DATA LOADING
DATA FACTORY
Azure Import/Export Service
API’s, CLI & GUI Tools
SERVING STORAGE
DATA PROCESSING
INGEST STORAGE
APPLICATIONS
AZURE DATABRICKS
DATA LAKE STORE
AZURE STORAGE
COSMOS DB r
LOGS, FILES AND MEDIA (UNSTRUCTURED)
AZURE SQL DW
HDINSIGHT
In modern data warehousing, data may be collected from various sources including flat files that are uploaded to storage like Data Lake Store or Azure Storage, or the source data may come from applications that are writing to one more transaction databases.
These “ingest stores” form the source for the data that is processed and ultimately served to applications and dashboards either in a data warehouse or by an analytic store that gets its data from the data warehouse.
It is worth observing that the modern data warehouse can serve two functions. It can participate in the data processing in addition to being the data store serving analytic clients.
Azure Analysis Services
BUSINESS / CUSTOM APPS
(STRUCTURED)
DASHBOARDS
SQL DB
COSMOS DB
OPERATIONAL DATA
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Real-Time Analytics
Real-time Analytics (aka Stream Analytics) is the phenomenon of processing data as soon as it is generated, to derive very quick analysis/insight for timely action.
Modern Data Warehousing
“We want to integrate all our data including ‘big data” with our data warehouse”
Real-Time Analytics
“We are trying to get insights from our devices in real-time, etc.”
Advanced Analytics
“We are trying to predict when our customers churn.”

13. STREAMING -Canonical Operations
Tech Ready 15
STREAMING -Canonical Operations
11/12/2019
Ingest
CONNECT, Collect, STORE
Ingest
Analytics
Process, Analyze
Actions
Report, Visualize, ACT
Ingest Use Case: Blob Historian
The expectations for fast and agile execution in businesses continue to grow. Businesses and developers are now opting for easy-to-use, cloud-based platforms to cope with the demand for more agility, and are looking for platforms that enable them to ingest and process a continuous stream of data generated by their systems in near real-time. The canonical Blob Historian scenario can be described as follows: Data from various devices and platforms which are geo-distributed across the world are pushed to a centralized data collector. Once the data is at the central location, some stateless transformation is performed on them such as scrubbing PII information, adding geo-tagging, IP lookup etc. The transformed data is then archived into Blob storage in a fashion which can be readily consumed by HDInsight for offline processing. Also Replay for RCA etc.
Analytics Use Case: Telemetry/Log Processing
Monitoring to reduce TTD, TTM
As the volume of devices, machines and applications grow, the most common enterprise use case to run businesses is the need to monitor and respond to changing business needs by creating rich analytics near real- time.
The canonical Telemetry/Log Processing scenario can be described using the Online Service or Application example, however the pattern is commonly seen across businesses that collect and report on application or device telemetry. The application or service regularly collects health data (data representing the current status of the application or infrastructure at a point in time) in addition to user request logs and other data representing actions or activities performed within the application. The data is historically saved to a blob or other type of data store for further processing. With the recent trend towards real-time dashboarding, in addition to saving the data to a blob or other type of store for historical analysis, customers are looking to process and transform the stream of incoming data directly such that it can be immediately provided to end users in the form of Dashboards and/or Notifications when action needs to be taken, for example if the site goes down operations personnel can be notified to begin investigation and resolve the issue quickly.
As more data is gathered and processed Machine Learning can also be used to develop and learn from patterns seen in the system such that it is possible to better predict when machines may need to be serviced or when things are about to go wrong.
Operations Use Case: IoT Scenario
Command and Control, Maintenance
As devices become smarter and more devices are built with communication capabilities, the expectation of what can be done with the data generated and collected from these devices continues to evolve both in the commercial and consumer spaces. It is expected that with so much data available, we can quickly combine and process the data, gaining more insight into the environment around us, and the devices we use regularly.
The canonical IoT Scenario is often described using the Vending Machine example, however the pattern is commonly seen across IoT use cases. The Devices, the Vending machines, regularly send information (product stock, status, temperature, etc. data) to either a field gateway (if the device is Non-IP capable) or to a cloud gateway (IP Capable) for ingestion into the system. The incoming data stream is processed and transformed such that it can be immediately provided to end users in the form of Dashboards and Notifications when action needs to be taken, for example when product in a specific vending machine gets low, the relevant representative can be notified to restock the machine, or if the machine is in need of repair a technician can be scheduled. In some cases, the action that needs to be taken may be as simple as rebooting the machine or pushing down a firmware upgrade which can be done without the need for human interaction.
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Big Data Streaming Pattern with Azure
Tech Ready 15
Big Data Streaming Pattern with Azure
11/12/2019 7:31 PM
EVENT HUBS
IoT HUB
KAFKA on HDINSIGHT
STREAM
ANALYTICS
STORM on
HDINSIGHT
AZURE DATABRICKS
(Spark Streaming)
AZURE ML STUDIO
R SERVER
(Spark ML)
MACHINE LEARNING
STREAM INGESTION
LONG-TERM STORAGE
STREAM ANALYTICS
SENSORS AND IOT
(UNSTRUCTURED)
REAL-TIME
APPLICATIONS r
LOGS, FILES AND MEDIA (UNSTRUCTURED)
Note that there are multiple options for all functional categories. In this deck the focus is on:
Stream Ingestion: The options are Azure Event Hubs, Azure IoT Hub and Apache Kafka on HDInsight
Stream Analytics i.e. querying, filtering and transforming streaming data: The options are Azure Stream Analytics, Azure Databricks Structured Streaming and Apache Storm on HDInsight
BUSINESS / CUSTOM APPS
(STRUCTURED)
REAL-TIME
DASHBOARDS
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Data Warehousing Pattern IN Azure
Tech Ready 15
Data Warehousing Pattern IN Azure
11/12/2019 7:31 PM
Big Data Lambda Architecture
AZURE HDINSIGHT
(Kafka)
WEB & MOBILE APPS
AZURE DATABRICKS
(Spark ML, SparkR, SparklyR)
LOGS, FILES AND MEDIA (UNSTRUCTURED) r
AZURE EventHub
Stream Analytics
AZURE DATA FACTORY
event
AZURE COSMOS DB
AZURE STORAGE
AZURE DATABRICKS
(Spark)
BUSINESS / CUSTOM APPS
(STRUCTURED)
ANALYTICAL DASHBOARDS
PolyBase
AZURE SQL DATA WAREHOUSE
AZURE DATA FACTORY
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Advanced Analytics
Advanced Analytics is the process of applying machine learning and/or deep learning techniques to data for the purpose of creating predictive/prescriptive insights.
Modern Data Warehousing
“We want to integrate all our data including ‘big data” with our data warehouse”
Real-Time Analytics
“We are trying to get insights from our devices in real-time, etc.”
Advanced Analytics
“We are trying to predict when our customers churn.”

17. Advanced analytics-Canonical Operations
Tech Ready 15
Advanced analytics-Canonical Operations
11/12/2019
Ingest
Acquire, understand
Data Acquisition & Understanding
Modeling
Training, validation
Deployment
Deploy, integrate
Data Acquisition is the process of collecting the source data so that it can explored and processed. Data Understanding is the process of exploring the data to understand the value it provides, its content and its flaws.
Modeling is the process of creating a predictive model from a collection of historical data that provides the context and outcomes.
Deployment or operationalization is the process of exposing the predictive capabilities of the trained model in a format that applications can integrate and consume.
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Advanced Analytics Pattern IN Azure
Tech Ready 15
Advanced Analytics Pattern IN Azure
11/12/2019 7:31 PM
Performing data collection/understanding, modeling and deployment
MODEL TRAINING
SERVING STORAGE
AZURE ML
AZURE ML STUDIO
ML SERVER
AZURE DATABRICKS (Spark ML)
SQL Server (In-database ML)
DATA SCIENCE VM
BATCH AI
SENSORS AND IOT
(UNSTRUCTURED)
COSMOS DB
APPLICATIONS
LONG TERM STORAGE
DATA PROCESSING
SQL DB r
LOGS, FILES AND MEDIA (UNSTRUCTURED)
SQL DW
SQL DB
DATA LAKE STORE
AZURE STORAGE
COSMOS DB
AZURE DATABRICKS
HDINSIGHT
SQL DW
TRAINED MODEL HOSTING
ORCHESTRATION
AZURE ANALYSIS SERVICES
BUSINESS / CUSTOM APPS
(STRUCTURED)
DASHBOARDS
SQL Server (In-database ML)
AZURE CONTAINER SERVICE
DATA FACTORY
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. considerations Customer: Regionality VNET requirements
IaaS
PaaS
SaaS
Customer:
Regionality
VNET requirements
Speed of business
Data Volumes
Data Types
Governance
Team
Skills
considerations

20. Proximity to existing practice
Quality of Data
Proximity to existing practice
Access to Data
Access to Tech.
Availability of
R&D
considerations
Access to Talent
Experience of Talent
Ability to Act

21. There are no right or wrong solutions, only optimal solutions
We lead with certain solutions and customise based on customer scenarios
Customer voice, product, and service maturity govern lead solutions
Consider price and performance, ease of use, and ecosystem acceptance as factors
Everything is fluid - a lead solution today might be non-optimal tomorrow, based on the factors above and new releases
Things to note