1. Microsoft Ignite NZ
25-28 October 2016
SKYCITY, Auckland

2. Microsoft Ignite 2016
11/17/2018 2:56 PM
Massive scale, Real-time processing, Optimized Cost & Ever changing requirements: Are you ready?
M234
Shweta Gupta
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Microsoft Ignite 2016 11/17/2018 2:56 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Identifying a scenario
Microsoft Ignite 2016
11/17/2018 2:56 PM
Identifying a scenario
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Microsoft Ignite 2016
11/17/2018 2:56 PM
So our scenario…
Change the draft of the air automatically based on the temperature in cabins/bay areas.
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Let’s define our next steps
Create our data sources
Step 2
Data Ingestion
Step 3:
Data processing to identify the extreme temperature areas
Step 4
Take action

7. Let’s define our next steps
Create our data sources
Step 2
Data Ingestion
Step 3:
Data processing to identify the extreme temperature areas
Step 4
Take action

8. Aspects of data sources
Microsoft Ignite 2016
11/17/2018 2:56 PM
Aspects of data sources
Choosing the right hardware
Placement
Number of sensors
Costs
Power & voltage requirements
Data sources
Data capture
Connectivity
Data format, structure
Security
Device identification
Authentication
Authorization
Control
How will the sensors send data?
Wifi, Bluetooth, ethernet
Field gateway?
Protocols MQTT/AMQP/Http/Custom
JSON/csv/text
Aggregation
Frequency
Compression
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Data capture Choosing the right hardware Placement of the devices
Microsoft Ignite 2016
11/17/2018 2:56 PM
Data capture
Choosing the right hardware
Placement of the devices
Power, voltage & costs
Number of sensors
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Data capture Choosing the right hardware Placement of the devices
Microsoft Ignite 2016
11/17/2018 2:56 PM
Data capture
Choosing the right hardware
Placement of the devices
Power, voltage & costs
Number of sensors
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Our scenario Average sensor range requires 1 in each bay area
Microsoft Ignite 2016
11/17/2018 2:56 PM
Our scenario
Average sensor range requires 1 in each bay area
Each floor has 10 such bays
Each building has 15 such floors
Each campus has 50 buildings
10 such campuses to manage
Sum: 2 x 10 x 15 x 50 x 50 = 750,000
Sum: 2 x 10 x 15 x 50 x 50 = 7500
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Connectivity Define the layout of devices
Microsoft Ignite 2016
11/17/2018 2:56 PM
Connectivity
Define the layout of devices
How will they send data? – wifi, Bluetooth, Ethernet
Where will the data be consolidated before sending it to the cloud
Will the sensors be directly connecting to the cloud?
Protocols – MQTT, AMQP, HTTP
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Our scenario
Each floor has bays

14. Our scenario
One sensor per bay

15. One hub per floor that consolidates data for the floor
Microsoft Ignite 2016
11/17/2018 2:56 PM
Our scenario
Bluetooth for sending data to the hub
One hub per floor that consolidates data for the floor
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Gateway for each building
Microsoft Ignite 2016
11/17/2018 2:56 PM
Our scenario
Cables/wifi for sending data to the hub
Gateway for each building
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Data structure JSON/CSV/Text Collection or single messages? Compressed
Microsoft Ignite 2016
11/17/2018 2:56 PM
Data structure
JSON/CSV/Text
Collection or single messages?
Compressed
Our scenario (JSON collection)
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Frequency Factors to consider Criticality Timeliness Connectivity
Microsoft Ignite 2016
11/17/2018 2:56 PM
Frequency
Factors to consider
Criticality
Timeliness
Connectivity
Costs
We will cover the costs in detail a little later down our journey…
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Security
Let’s talk this along with Data Ingestion!

20. Let’s define our next steps
Microsoft Ignite 2016
11/17/2018 2:56 PM
Let’s define our next steps
Step 1
Create our data sources
Step 2
Data Ingestion
Step 3:
Data processing to identify the extreme temperature areas
Step 4
Take action
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Factors to consider Location Throughput Security
Microsoft Ignite 2016
11/17/2018 2:56 PM
Factors to consider
Location
Throughput
Security
Authentication & Authorization
Encryption
Compression
Collection parsing
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Iot HubCost per month (USD)
Microsoft Ignite 2016
11/17/2018 2:56 PM
Our scenario
No. of sensors: 750,000
Throughput requirements:
Scenario
Frequency (in mins)
Message size (in bytes)
Throughput (Mb/day)
Iot HubCost per month (USD)
direct messages 5
240
2000 30
823.97
450
Aggregate messages
34816
956.25 50
aggregate messages
159.83
compressed & aggregate messages
5120
140.63
23.44
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Secured ingestion Device registry Security token Authentication
Microsoft Ignite 2016
11/17/2018 2:56 PM
Secured ingestion
Device registry
Security token
Authentication
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Device registration Register device with X.509 certificate
Microsoft Ignite 2016
11/17/2018 2:56 PM
Device registration
Register device with X.509 certificate
Return device
key details
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Secured ingestion Microsoft Ignite 2016 11/17/2018 2:56 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Secured ingestion Microsoft Ignite 2016 11/17/2018 2:56 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Microsoft Ignite 2016
11/17/2018 2:56 PM
Generating SAS Token
Note: X.509 certificate does not include skn in the string
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. De-compression & De-aggregation
Microsoft Ignite 2016
11/17/2018 2:56 PM
De-compression & De-aggregation
Generic compression & de-compression code which can be deployed (github)
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. De-compression & De-aggregation
Microsoft Ignite 2016
11/17/2018 2:56 PM
De-compression & De-aggregation
Generic de-aggregation code, based on configurable de- aggregation pattern
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. Data Ingestion – Our scenario
Microsoft Ignite 2016
11/17/2018 2:56 PM
Data Ingestion – Our scenario
Building gateway consolidate & compress data
Gateway generate SaS token
Data is sent over to Azure IoT Hub over tls using the SaS tokens
IoT hub trigger function de- compresses & de-aggregates the data
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. Data sources & Ingestion
Microsoft Ignite 2016
11/17/2018 2:56 PM
Data sources & Ingestion
In action
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Let’s define our next steps
Create our data sources
Step 2
Data Ingestion
Step 3:
Data processing to identify the extreme temperature areas
Step 4
Take action

33. Microsoft Ignite 2016
11/17/2018 2:56 PM
Decisions:
We need to check if any of the sensors is sending data that is above the benchmark temperature
Or below the benchmark temperature
Probably store the recorded temperature… observe user patterns
Alert if a sensor is not functioning or not sending data
Manually corrected temperature
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

34. So we have two paths: Hot path Cold path Near-real time
Microsoft Ignite 2016
11/17/2018 2:56 PM
So we have two paths:
Near-real time
processing
Hot path
Benchmark alerts
Inactive sensor alerts
Streaming data
Data storage
Cold path
Store for future use
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

35. Comparing various options
Microsoft Ignite 2016
11/17/2018 2:56 PM
Comparing various options
Option
Description
Best suited
Spark/HDInsight
Spark running on custom Hadoop or HDInsight clusters that could process streaming data into both storages
Complex processing. For ex, running NLP algorithms on oncoming twitter feeds
Application on Virtual machines
Custom applications deployed on Virtual machines
Migration of legacy applications
App services
Custom code written, but managed by Azure.
Legacy code that could be deployed on PaaS
Functions
Server less lightweight compute
Trigger based operations
Stream Analytics
Queries in T-SQL that process streaming data in time windows
Conditional checking / processing in windows
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. Our scenario: Microsoft Ignite 2016 11/17/2018 2:56 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

37. Our scenario: Hot path Cold path Microsoft Ignite 2016
11/17/2018 2:56 PM
Our scenario:
Hot path
Cold path
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

38. Data processing In action Microsoft Ignite 2016 11/17/2018 2:56 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

39. Let’s define our next steps
Create our data sources
Step 2
Data Ingestion
Step 3:
Data processing to identify the extreme temperature areas
Step 4
Take action

40. Step 4: Take Action

41. Microsoft Ignite 2016
11/17/2018 2:56 PM
Take action
What do we want to do if we a bay sensor records too hot or cold?
What do we want to do if a sensor is not sending data
What do we want to do with the sensor temperature that is being recorded
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

42. Send instructions to move the air draft
Take Action
What if a bay’s sensor records too hot or too cold?
Update live dashboard
Alert triggered
Send instructions to move the air draft

43. Our scenario: Microsoft Ignite 2016 11/17/2018 2:56 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

44. Take action

45. Data action In action Microsoft Ignite 2016 11/17/2018 2:56 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

46. Step 5: Predict? Microsoft Ignite 2016 11/17/2018 2:56 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

47. Going beyond reactive analytics
Microsoft Ignite 2016
11/17/2018 2:56 PM
Going beyond reactive analytics
Temperature data
by bays, floor, building & campus
captured every half hour
manually triggered
external weather details
occupancy
build a temperature prediction model
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

48. Going beyond reactive analytics
Weather forecasts
Outlook
calendar
Temperature prediction model
Set the
temperature
Attendance
(card swipe details)

49. Take action

50. Microsoft Ignite 2016 11/17/2018 2:56 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

51. Reference
Github for content: (will go public by Nov 5th, 2016)
Github for De-aggregation & De-compression:
Whitepaper on various hardware devices:

52. 11/17/2018 2:56 PM
Questions?
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.