Introduction to Azure Streaming Analytics Warren Sifre Introduction to Azure Streaming Analytics April 4, 2017

Who am I? Warren Sifre Data Analytics Solution Architect Allegient Email: wsifre@allegient.com Twitter: @WAS_SQL LinkedIn: www.linkedin.com/in/wsifre

About me… In the IT Industry since 1998. Developed system integration solutions against many different database platforms for various applications across many industries. Passion in Solutions Architecture at both hardware and software levels. Interests in SQL Server, MongoDB, Hadoop, Python/C#/PowerShell and Information Security (Hacking) Indy BI PASS User Group Founder and Chapter Leader PASS SQL Saturday / Indy PASS User Group presenter MSCE: Data Platforms/Business Intelligence, Teradata 14 CTP and many more…

What is Streaming Analytics? How it works? Use Cases Configuration and Dependencies SAQL Demo

Typical Environment

Challenges… Real-Time Analytics Environment Scalability Many Steps in between the Source Data and the Visualization/Reporting Output. Environment Scalability Months/Years of planning is needed to plan out equipment procurement and scale out to meet increasing demand. Resource Cost Management Ideal configuration would require the purchasing of enough equipment to handle peak performance requirements. Although those peak requirements may only be for a few hours of any given day. Disaster Recovery Strategy Architecting and maintaining a DR strategy where performance, RTOs, and RPOs are met can be challenging and leave the organization with a lot of underutilized resources.

What is Streaming Analytics? A way to evaluate data before it has reached its final repository destination. Why? Hours to weeks can be the time it takes for data to be transmitted, received, processed, aggregated, then visualized in the traditional Data Warehouse architecture. Business requirements have changed and the desire to glean insights from this data sooner is now becoming a requirement, not a nice to have.

How it can work… Streaming Analytics Job Event Hub Data Factory Process and Deliver data to multiple end points Transmit Data Event Hub Queues data for processing Data Factory Gather and process data for Predictive Analytics Process Machine Learning Predictive Analytics Processing Power BI Visualize real-time data stream Azure SQL Store data

Reimagined Environment

Use Cases Transportation Energy Manufacturing Medical Device Reduce the need to pull vehicles from service for routine inspections by using sensors to determine when actual anomalies are occurring. Energy Monitor equipment from central locations such as Wind Turbines and Power Generators, thus reducing time spent on manual/physical inspection or replacement of parts just because of time instead of actual degraded performance. Manufacturing Monitor equipment and plant conditions for optimal performance. Medical Device Through remote monitoring expensive replacement parts can be ordered closer to the end-of- life of an equipment than by a schedule. This can reduce the cost of having an overstock of parts on-hand.

Configuration Options… Add Input(s) Data Stream Event Hub Blob Storage IoT Hub Reference Data Add Query Streaming Analytic Query Language (SAQL) - Similar to T-SQL Add Output(s) SQL Database Blob Storage Event Hub Power BI Table Storage Service Bus Queue Service Bus Topic DocumentDB Data Lake Store Settings Scale – How much processing power desired for SA Job? Exception Handling– What is the definition of Late Data? What to do with late or out of order data? Alerts – When do you want to receive a notification? Functions – AzureML Integration

SAQL - Elements DML SELECT FROM WHERE GROUP BY HAVING CASE WHEN THEN ELSE INNER JOIN LEFT OUTER JOIN UNION CROSS APPLY OUTER APPLY CAST INTO ORDER BY ASC, DSC String Functions Len ConCat CharIndex Substring PatIndex Date and Time Functions DateName DatePart Day Month Year DateTimeFromParts DateDiff DateAdd Aggregate Functions Sum Count Avg Min Max StDev StDevP Var VarP Windowing Extensions TumblingWindow HoppingWindow SlidingWindow Scaling Extensions With Partition By Over Temporal Functions Lag IsFirst CollectTop

SAQL - Elements DML SELECT FROM WHERE GROUP BY HAVING CASE WHEN THEN ELSE INNER JOIN LEFT OUTER JOIN UNION CROSS APPLY OUTER APPLY CAST INTO ORDER BY ASC, DSC String Functions Len ConCat CharIndex Substring PatIndex Date and Time Functions DateName DatePart Day Month Year DateTimeFromParts DateDiff DateAdd Aggregate Functions Sum Count Avg Min Max StDev StDevP Var VarP Windowing Extensions TumblingWindow HoppingWindow SlidingWindow Scaling Extensions With Partition By Over Temporal Functions Lag IsFirst CollectTop

Tumbling Window Fixed window of time with no overlap

Hopping Window Fixed window of time with a fix time of overlap

Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window.

Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window.

Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window.

Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window.

Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window.

Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window.

Sliding Window A Fixed window time, but a window is defined as the moment an event enters or exits an existing window.

Demonstration

Helpful Links More Information on Streaming Analytics SAQL https://msdn.microsoft.com/en-us/library/azure/dn834998.aspx SAQL Query Patterns https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-stream-analytics-query-patterns/ Azure Portal Link Azure Portal

Warren Sifre Email: wsifre@allegient.com Twitter: @WAS_SQL LinkedIn: www.linkedin.com/in/wsifre