1. AZURE STREAM ANALYTICS & DATA FACTORY

2. Canonical Architecture for processing events
Tech Ready 15
Canonical Architecture for processing events
9/14/2018
Ingestor (broker)
Collection
Presentation and action
Event producers
Transformation
Long-term storage
Event Hubs
Storage adapters
Stream processing
Cloud gateways (web APIs)
Field gateways
Applications
Legacy IOT (custom protocols)
Devices
IP-capable devices (Windows/Linux)
Low-power devices (RTOS)
Search and query
Data analytics (Excel)
Web/thick client dashboards
Service bus
Azure DBs
Azure storage
HDInsight
Stream Analytics
Devices to take action
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Canonical scenarios Archiving Dashboarding Triggering Workflows
Tech Ready 15
9/14/2018
Canonical scenarios
Archiving
Dashboarding
Triggering Workflows
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Azure Service Bus New! Azure Service Bus Relay Queue Topic
9/14/2018
Azure Service Bus
Relay
NAT and Firewall Traversal Service Request/Response Services Un-buffered with TCP Throttling
Queue
Transactional Cloud AMQP/HTTP Broker High-Scale, High-Reliability Messaging Sessions, Scheduled Delivery, etc.
Topic
Transactional Message Distribution
Up to 2000 subscriptions per Topic
Up to 2K/100K filter rules per subscription
Notification Hub
High-scale notification distribution
Most mobile push notification services
Millions of notification targets
Event Hub
New!
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Up to 32 partitions via portal, more on request
Introducing Azure Event Hubs
IEventProcessor
Event Processor Host
Azure Event Hub
Direct
Receivers
Consumer Group(s)
Partitions
PartitionKey
Event Producers
Hash
AMQP 1.0
Credit-based flow control
Client-side cursors
Offset by Id or Timestamp
Throughput Units:
1 ≤ TUs ≤ Partition Count
TU: 1 MB/s writes, 2 MB/s reads
Up to 32 partitions via portal, more on request

6. Azure Stream Analytics Потоковая обработка в реальном времени в облаке
Обработка миллиона потоков событий в секунду
Проведение анализа в режиме реального времени
Коррелирование множества потоков данных
Быстрая разработка на знакомом языке на основе SQL. Интеграция с PowerBI

7. Stream Analytics, End-to-End Overview
Azure Stream Analytics
Outputs
SQL Azure
Azure Blobs
Event Hub 
Temporal Semantics
Guaranteed delivery
Guaranteed up time
Transform
Temporal joins
Filter
Aggregates
Projections
Windows
Etc.
Enrich
Correlate 
Event Inputs
Event Hub
Azure Blob
BI Dashboards
Predictive Analytics ☁
Reference Data
Azure Blob
Azure
Storage
Data Source
Consume
Collect
Process
Deliver

8. Real-time analytics Real-time Analytics Fully Managed Cloud Service
Intake millions of events per second (up to 1 GB/s)
Low processing latency, auto adaptive (sub-second to seconds)
Correlate between different streams, or with reference data
Find patterns or lack of patterns in data in real-time
Fully Managed Cloud Service
No hardware acquisition and maintenance
No platform/infrastructure deployment and maintenance
Easily expand your business globally leveraging Azure regions

9. Mission critical reliability
Guaranteed event delivery
Guaranteed business continuity: Automatic and fast recovery
Effective Audits
Privacy and security properties of solutions are evident
Azure integration for monitoring and ops alerting
Easy To Scale
Scale from small to large on demand

10. Scheduling and monitoring built in
Built-in monitoring
View your system’s performance at a glance
Help you find the cost-optimal way of deployment

11. Query Language You write declarative queries in SQL
Tech Ready 15
9/14/2018
You write declarative queries in SQL
No code compilation, easy to author and deploy
Unified programming model
Brings together event streams, reference data and machine learning extensions
Temporal Semantics
All operators respect, and some use, the temporal properties of events
Built-in operators and functions
These should (mostly) look familiar if you know relational databases
Filters, projections, joins, windowed (temporal) aggregates, text and date manipulation
Query Language
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. SQL – Language & Library
Build 2015
9/14/ :37 AM
SQL – Language & Library
DML
SELECT
FROM
WHERE
GROUP BY
HAVING
CASE WHEN THEN ELSE
INNER/LEFT OUTER JOIN
UNION
CROSS/OUTER APPLY
CAST
INTO
ORDER BY ASC, DSC
Date and Time Functions
DateName
DatePart
Day
Month
Year
DateTimeFromParts
DateDiff
DateAdd
Aggregate Functions
Sum
Count
Avg
Min
Max
StDev
StDevP
Var
VarP
Temporal Functions
Lag, IsFirst
CollectTop
String Functions
Len
Concat
CharIndex
Substring
PatIndex
Windowing Extensions
TumblingWindow
HoppingWindow
SlidingWindow
Scaling Extensions
WITH
PARTITION BY
OVER
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Windowing Concepts Windows can be tumbling, hopping, or sliding
Tech Ready 15
9/14/2018
Windowing Concepts
Windows can be tumbling, hopping, or sliding
Windows are fixed length
Must be used in a GROUP BY clause
Output event will have the timestamp of the end of the window 1 5 4 2 6 8 t1 t2 t5 t6 t3 t4
Time
Window 1
Window 2
Window 3
Aggregate
Function (Sum) 18 14
Output Events

14. Tech Ready 15
9/14/2018
Tumbling Windows
How many vehicles entered each toll both every 5 minutes?
SELECT TollId, COUNT(*) FROM EntryStream
GROUP BY TollId, TumblingWindow(minute,5)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. A 10-second Hopping Window with a 5-second “Hop”
Tech Ready 15
9/14/2018
Hopping Windows
A 10-second Hopping Window with a 5-second “Hop” 1 5 4 6 2 8 6 5 3 6 1
“Every 5 seconds give me the count of tweets over the last 10 seconds” 5 10 15 20 25 30
Time
(secs) 1 5 4 6 2 4 6 2 8 6 8 6 5 3 5 3 6 1
SELECT Topic, Count(*) AS TotalTweets
FROM TwitterStream TIMESTAMP BY CreatedAt
GROUP BY Topic, HoppingWindow(second, 10, 5)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Joining multiple streams
Tech Ready 15
9/14/2018
Joining multiple streams
“List all users and the topics on which they switched their sentiment within a minute“
Twitter Stream:
{“XO”, 4, “Win10”}
{“Dip”, 2, “XBox”}
{“Jo”, 0, “Surface”}
{“Foo”,4, “Bing”}
Twitter Stream:
(same stream,
further down the timeline)
{“XO”, 0, “Win10”}
{“Dip”, 0, “Xbox”}
{“Jo”, 4, “Surface”}
{“Foo”, 0, “Bing”}
time
SELECT TS1.UserName, TS1.Topic
FROM TwitterStream TS1 TIMESTAMP BY CreatedAt
JOIN TwitterStream TS2 TIMESTAMP BY CreatedAt
ON TS1.UserName = TS2.UserName AND TS1.Topic = TS2.Topic AND DateDiff(second, TS1, TS2) BETWEEN 1 AND 60
WHERE TS1.SentimentScore != TS2.SentimentScore
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Detecting absence of events
Build 2015
9/14/ :37 AM
Detecting absence of events
“Show me if a topic is not tweeted for 10 seconds since it was last tweeted”
Twitter Stream:
{“XO”, 4, “Win10”}
{“WAA”, 2, “Microsoft”}
{“AB”, 0, “Bing}
{“Dip”, 4, “Xbox”}
Twitter Stream:
(same stream,
further down the timeline)
{“Foo”, 0, “Win10”}
{“Tim”, 2, “Microsoft”}
{“AB”, 0, “Bing”}
time
SELECT TS1.CreatedAt, TS1.Topic, TS1.UserName
FROM TwitterStream TS1 TIMESTAMP BY CreatedAt
LEFT OUTER JOIN TwitterStream TS2 TIMESTAMP BY CreatedAt
ON TS1.Topic = TS2.Topic AND DateDiff(second, TS1, TS2) BETWEEN 1 AND 10
WHERE TS2.Topic IS NULL
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Build 2015
9/14/ :37 AM
Reference Data
Seamless correlation of event streams with reference data
Static or slowly-changing data stored in blobs
CSV and JSON files in Azure Blobs; scanned for new snapshots on a settable cadence
JOIN (INNER or LEFT OUTER) between streams and reference data sources
Reference data appears like another input:
SELECT myRefData.Name, myStream.Value
FROM myStream
JOIN myRefData
ON myStream.myKey = myRefData.myKey
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Scaling using Partitions
Partitioning allows for parallel execution over scaled-out resources
SELECT Count(*) AS Count, Topic
FROM TwitterStream PARTITION BY PartitionId
GROUP BY TumblingWindow(minute, 3), Topic, PartitionId
Stream Analytics
Event Hub
PartitionId = 1
Query
Result 1
Result 2
Result 3
PartitionId = 2
PartitionId = 3
PartitionId = 1
PartitionId = 3
PartitionId = 2

20. Multiple steps, multiple outputs
Tech Ready 15
9/14/2018
Multiple steps, multiple outputs
WITH Step1 AS (
SELECT Count(*) AS CountTweets, Topic
FROM TwitterStream PARTITION BY PartitionId
GROUP BY TumblingWindow(second, 3), Topic, PartitionId ),
Step2 AS (
SELECT Avg(CountTweets)
FROM Step1
GROUP BY TumblingWindow(minute, 3) )
SELECT * INTO Output1 FROM Step1
SELECT * INTO Output2 FROM Step2
SELECT * INTO Output3 FROM Step2
A query can have multiple steps to enable pipeline execution
A step is a sub-query defined using WITH (“common table expression”)
Can be used to develop complex queries more elegantly by creating a intermediary named result
Creates unit of execution for scaling out when PARTITION BY is used
Each step’s output can be sent to multiple output targets using INTO

21. Pricing Stream Analytics is priced on two variables:
Build 2015
9/14/ :37 AM
Pricing
Stream Analytics is priced on two variables:
Volume of data processed
Streaming units required to process the data stream
Meter
Price (USD)
Volume of Data Processed
Volume of data processed by the streaming job (in GB)
$.001 per GB
Streaming Unit
Blended measure of cores, memory, and bandwidth
$0.031 per hour
* Streaming unit is a unit of compute capacity with a maximum throughput of 1MB/s
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Azure Event Hubs Прием телеметрии, масштабируемой в облаке, с веб-сайтов, приложений и устройств
Запись миллионов событий в секунду практически в режиме реального времени
Подключение устройств с гибкой авторизацией и регулированием количества запросов
Буферизация событий на базе времени
Подключаемые адаптеры для других облачных служб. Клиентские библиотеки

23. Azure Data Factory
Служба фабрики данных Azure - это сложная платформа для организации работы с традиционными EDW и в условиях меняющейся среды данных. Она предоставляет предприятиям возможность использовать все доступные им данные для принятия решений на основе этих данных.
Служба фабрики данных Azure позволяет предприятиям использовать все это разнообразие, предоставляя платформу для организации процессов обработки, хранения и перемещения данных и объединения их в конвейеры производства информации, а также для управления массивами достоверных данных.

24. Возможности Azure DF
Мониторинг конвейеров данных в одном окне. Служба фабрики данных предоставляет полное и достоверное представление о службах хранения, обработки и перемещения данных. Она помогает быстро оценить работоспособность конвейера данных от начала до конца, выявить проблемы и при необходимости предпринять корректирующие действия. Вы также можете в визуальном режиме отслеживать происхождение данных и взаимосвязи между разными данными во всех ваших источниках.
Комфортная работа с различными системами хранения и обработки данных. Служба фабрики данных позволяет создавать конвейеры производства информации, которые перемещают и обрабатывают как локальные (например, SQL Server), так и облачные источники данных, такие как база данных SQL Azure, таблица Azure и BLOB-объекты.
Преобразование данных в достоверную информацию. Служба фабрики данных поддерживает обработку Hive, Pig и C#, а также ключевые функции обработки, такие как автоматическое управление кластером Hadoop (HDInsight), повторные попытки при временных ошибках, настраиваемые политики времени ожидания и предупреждения.

25. Workflow

26. Build 2015
9/14/ :37 AM
Resource Library
Business Overview
Documentation
Samples
ASA Blog
Follow us on Twitter
ASA Forum
Vote for ideas
ASA Team
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.