AZURE STREAM ANALYTICS & DATA FACTORY

Canonical Architecture for processing events Tech Ready 15 Canonical Architecture for processing events 9/14/2018 Ingestor (broker) Collection Presentation and action Event producers Transformation Long-term storage Event Hubs Storage adapters Stream processing Cloud gateways (web APIs) Field gateways Applications Legacy IOT (custom protocols) Devices IP-capable devices (Windows/Linux) Low-power devices (RTOS) Search and query Data analytics (Excel) Web/thick client dashboards Service bus Azure DBs Azure storage HDInsight Stream Analytics Devices to take action © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Canonical scenarios Archiving Dashboarding Triggering Workflows Tech Ready 15 9/14/2018 Canonical scenarios Archiving Dashboarding Triggering Workflows © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Service Bus New! Azure Service Bus Relay Queue Topic 9/14/2018 Azure Service Bus Relay NAT and Firewall Traversal Service Request/Response Services Un-buffered with TCP Throttling Queue Transactional Cloud AMQP/HTTP Broker High-Scale, High-Reliability Messaging Sessions, Scheduled Delivery, etc. Topic Transactional Message Distribution Up to 2000 subscriptions per Topic Up to 2K/100K filter rules per subscription Notification Hub High-scale notification distribution Most mobile push notification services Millions of notification targets Event Hub New! © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Up to 32 partitions via portal, more on request Introducing Azure Event Hubs IEventProcessor Event Processor Host Azure Event Hub Direct Receivers Consumer Group(s) Partitions PartitionKey Event Producers Hash AMQP 1.0 Credit-based flow control Client-side cursors Offset by Id or Timestamp Throughput Units: 1 ≤ TUs ≤ Partition Count TU: 1 MB/s writes, 2 MB/s reads Up to 32 partitions via portal, more on request

Azure Stream Analytics Потоковая обработка в реальном времени в облаке Обработка миллиона потоков событий в секунду Проведение анализа в режиме реального времени Коррелирование множества потоков данных Быстрая разработка на знакомом языке на основе SQL. Интеграция с PowerBI

Stream Analytics, End-to-End Overview Azure Stream Analytics Outputs SQL Azure Azure Blobs Event Hub  Temporal Semantics Guaranteed delivery Guaranteed up time Transform Temporal joins Filter Aggregates Projections Windows Etc. Enrich Correlate  Event Inputs Event Hub Azure Blob BI Dashboards Predictive Analytics ☁ Reference Data Azure Blob Azure Storage Data Source Consume Collect Process Deliver

Real-time analytics Real-time Analytics Fully Managed Cloud Service Intake millions of events per second (up to 1 GB/s) Low processing latency, auto adaptive (sub-second to seconds) Correlate between different streams, or with reference data Find patterns or lack of patterns in data in real-time Fully Managed Cloud Service No hardware acquisition and maintenance No platform/infrastructure deployment and maintenance Easily expand your business globally leveraging Azure regions

Mission critical reliability Guaranteed event delivery Guaranteed business continuity: Automatic and fast recovery Effective Audits Privacy and security properties of solutions are evident Azure integration for monitoring and ops alerting Easy To Scale Scale from small to large on demand

Scheduling and monitoring built in Built-in monitoring View your system’s performance at a glance Help you find the cost-optimal way of deployment

Query Language You write declarative queries in SQL Tech Ready 15 9/14/2018 You write declarative queries in SQL No code compilation, easy to author and deploy Unified programming model Brings together event streams, reference data and machine learning extensions Temporal Semantics All operators respect, and some use, the temporal properties of events Built-in operators and functions These should (mostly) look familiar if you know relational databases Filters, projections, joins, windowed (temporal) aggregates, text and date manipulation Query Language © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

SQL – Language & Library Build 2015 9/14/2018 11:37 AM SQL – Language & Library DML SELECT FROM WHERE GROUP BY HAVING CASE WHEN THEN ELSE INNER/LEFT OUTER JOIN UNION CROSS/OUTER APPLY CAST INTO ORDER BY ASC, DSC Date and Time Functions DateName DatePart Day Month Year DateTimeFromParts DateDiff DateAdd Aggregate Functions Sum Count Avg Min Max StDev StDevP Var VarP Temporal Functions Lag, IsFirst CollectTop String Functions Len Concat CharIndex Substring PatIndex Windowing Extensions TumblingWindow HoppingWindow SlidingWindow Scaling Extensions WITH PARTITION BY OVER © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Windowing Concepts Windows can be tumbling, hopping, or sliding Tech Ready 15 9/14/2018 Windowing Concepts Windows can be tumbling, hopping, or sliding Windows are fixed length Must be used in a GROUP BY clause Output event will have the timestamp of the end of the window 1 5 4 2 6 8 t1 t2 t5 t6 t3 t4 Time Window 1 Window 2 Window 3 Aggregate Function (Sum) 18 14 Output Events

Tech Ready 15 9/14/2018 Tumbling Windows How many vehicles entered each toll both every 5 minutes? SELECT TollId, COUNT(*) FROM EntryStream GROUP BY TollId, TumblingWindow(minute,5) © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

A 10-second Hopping Window with a 5-second “Hop” Tech Ready 15 9/14/2018 Hopping Windows A 10-second Hopping Window with a 5-second “Hop” 1 5 4 6 2 8 6 5 3 6 1 “Every 5 seconds give me the count of tweets over the last 10 seconds” 5 10 15 20 25 30 Time (secs) 1 5 4 6 2 4 6 2 8 6 8 6 5 3 5 3 6 1 SELECT Topic, Count(*) AS TotalTweets FROM TwitterStream TIMESTAMP BY CreatedAt GROUP BY Topic, HoppingWindow(second, 10, 5) © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Joining multiple streams Tech Ready 15 9/14/2018 Joining multiple streams “List all users and the topics on which they switched their sentiment within a minute“ Twitter Stream: {“XO”, 4, “Win10”} {“Dip”, 2, “XBox”} {“Jo”, 0, “Surface”} {“Foo”,4, “Bing”} Twitter Stream: (same stream, further down the timeline) {“XO”, 0, “Win10”} {“Dip”, 0, “Xbox”} {“Jo”, 4, “Surface”} {“Foo”, 0, “Bing”} time SELECT TS1.UserName, TS1.Topic FROM TwitterStream TS1 TIMESTAMP BY CreatedAt JOIN TwitterStream TS2 TIMESTAMP BY CreatedAt ON TS1.UserName = TS2.UserName AND TS1.Topic = TS2.Topic AND DateDiff(second, TS1, TS2) BETWEEN 1 AND 60 WHERE TS1.SentimentScore != TS2.SentimentScore © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Detecting absence of events Build 2015 9/14/2018 11:37 AM Detecting absence of events “Show me if a topic is not tweeted for 10 seconds since it was last tweeted” Twitter Stream: {“XO”, 4, “Win10”} {“WAA”, 2, “Microsoft”} {“AB”, 0, “Bing} {“Dip”, 4, “Xbox”} Twitter Stream: (same stream, further down the timeline) {“Foo”, 0, “Win10”} {“Tim”, 2, “Microsoft”} {“AB”, 0, “Bing”} time SELECT TS1.CreatedAt, TS1.Topic, TS1.UserName FROM TwitterStream TS1 TIMESTAMP BY CreatedAt LEFT OUTER JOIN TwitterStream TS2 TIMESTAMP BY CreatedAt ON TS1.Topic = TS2.Topic AND DateDiff(second, TS1, TS2) BETWEEN 1 AND 10 WHERE TS2.Topic IS NULL © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Build 2015 9/14/2018 11:37 AM Reference Data Seamless correlation of event streams with reference data Static or slowly-changing data stored in blobs CSV and JSON files in Azure Blobs; scanned for new snapshots on a settable cadence JOIN (INNER or LEFT OUTER) between streams and reference data sources Reference data appears like another input: SELECT myRefData.Name, myStream.Value FROM myStream JOIN myRefData ON myStream.myKey = myRefData.myKey © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Scaling using Partitions Partitioning allows for parallel execution over scaled-out resources SELECT Count(*) AS Count, Topic FROM TwitterStream PARTITION BY PartitionId GROUP BY TumblingWindow(minute, 3), Topic, PartitionId Stream Analytics Event Hub PartitionId = 1 Query Result 1 Result 2 Result 3 PartitionId = 2 PartitionId = 3 PartitionId = 1 PartitionId = 3 PartitionId = 2

Multiple steps, multiple outputs Tech Ready 15 9/14/2018 Multiple steps, multiple outputs WITH Step1 AS ( SELECT Count(*) AS CountTweets, Topic FROM TwitterStream PARTITION BY PartitionId GROUP BY TumblingWindow(second, 3), Topic, PartitionId ), Step2 AS ( SELECT Avg(CountTweets) FROM Step1 GROUP BY TumblingWindow(minute, 3) ) SELECT * INTO Output1 FROM Step1 SELECT * INTO Output2 FROM Step2 SELECT * INTO Output3 FROM Step2 A query can have multiple steps to enable pipeline execution A step is a sub-query defined using WITH (“common table expression”) Can be used to develop complex queries more elegantly by creating a intermediary named result Creates unit of execution for scaling out when PARTITION BY is used Each step’s output can be sent to multiple output targets using INTO

Pricing Stream Analytics is priced on two variables: Build 2015 9/14/2018 11:37 AM Pricing Stream Analytics is priced on two variables: Volume of data processed Streaming units required to process the data stream Meter Price (USD) Volume of Data Processed Volume of data processed by the streaming job (in GB) $.001 per GB Streaming Unit Blended measure of cores, memory, and bandwidth $0.031 per hour * Streaming unit is a unit of compute capacity with a maximum throughput of 1MB/s © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Event Hubs Прием телеметрии, масштабируемой в облаке, с веб-сайтов, приложений и устройств Запись миллионов событий в секунду практически в режиме реального времени Подключение устройств с гибкой авторизацией и регулированием количества запросов Буферизация событий на базе времени Подключаемые адаптеры для других облачных служб. Клиентские библиотеки

Azure Data Factory Служба фабрики данных Azure - это сложная платформа для организации работы с традиционными EDW и в условиях меняющейся среды данных. Она предоставляет предприятиям возможность использовать все доступные им данные для принятия решений на основе этих данных. Служба фабрики данных Azure позволяет предприятиям использовать все это разнообразие, предоставляя платформу для организации процессов обработки, хранения и перемещения данных и объединения их в конвейеры производства информации, а также для управления массивами достоверных данных.

Возможности Azure DF Мониторинг конвейеров данных в одном окне. Служба фабрики данных предоставляет полное и достоверное представление о службах хранения, обработки и перемещения данных. Она помогает быстро оценить работоспособность конвейера данных от начала до конца, выявить проблемы и при необходимости предпринять корректирующие действия. Вы также можете в визуальном режиме отслеживать происхождение данных и взаимосвязи между разными данными во всех ваших источниках. Комфортная работа с различными системами хранения и обработки данных. Служба фабрики данных позволяет создавать конвейеры производства информации, которые перемещают и обрабатывают как локальные (например, SQL Server), так и облачные источники данных, такие как база данных SQL Azure, таблица Azure и BLOB-объекты. Преобразование данных в достоверную информацию. Служба фабрики данных поддерживает обработку Hive, Pig и C#, а также ключевые функции обработки, такие как автоматическое управление кластером Hadoop (HDInsight), повторные попытки при временных ошибках, настраиваемые политики времени ожидания и предупреждения.

Workflow

Build 2015 9/14/2018 11:37 AM Resource Library Business Overview http://azure.microsoft.com/en-us/services/stream-analytics/ Documentation http://azure.microsoft.com/en-us/documentation/services/stream-analytics/ Samples https://github.com/streamanalytics/samples ASA Blog http://blogs.msdn.com/b/streamanalytics/rss.aspx Follow us on Twitter https://twitter.com/AzureStreaming (follow @AzureStreaming) ASA Forum https://social.msdn.microsoft.com/Forums/en-US/home?forum=AzureStreamAnalytics   Vote for ideas http://feedback.azure.com/forums/270577-azure-stream-analytics Email ASA Team azstream@microsoft.com © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.