Microsoft Ignite NZ 25-28 October 2016 SKYCITY, Auckland

Microsoft Ignite 2016 11/17/2018 2:56 PM Massive scale, Real-time processing, Optimized Cost & Ever changing requirements: Are you ready? M234 Shweta Gupta © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Ignite 2016 11/17/2018 2:56 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Identifying a scenario Microsoft Ignite 2016 11/17/2018 2:56 PM Identifying a scenario © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Ignite 2016 11/17/2018 2:56 PM So our scenario… Change the draft of the air automatically based on the temperature in cabins/bay areas. © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Let’s define our next steps Create our data sources Step 2 Data Ingestion Step 3: Data processing to identify the extreme temperature areas Step 4 Take action

Let’s define our next steps Create our data sources Step 2 Data Ingestion Step 3: Data processing to identify the extreme temperature areas Step 4 Take action

Aspects of data sources Microsoft Ignite 2016 11/17/2018 2:56 PM Aspects of data sources Choosing the right hardware Placement Number of sensors Costs Power & voltage requirements Data sources Data capture Connectivity Data format, structure Security Device identification Authentication Authorization Control How will the sensors send data? Wifi, Bluetooth, ethernet Field gateway? Protocols MQTT/AMQP/Http/Custom JSON/csv/text Aggregation Frequency Compression © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data capture Choosing the right hardware Placement of the devices Microsoft Ignite 2016 11/17/2018 2:56 PM Data capture Choosing the right hardware Placement of the devices Power, voltage & costs Number of sensors © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data capture Choosing the right hardware Placement of the devices Microsoft Ignite 2016 11/17/2018 2:56 PM Data capture Choosing the right hardware Placement of the devices Power, voltage & costs Number of sensors © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Our scenario Average sensor range requires 1 in each bay area Microsoft Ignite 2016 11/17/2018 2:56 PM Our scenario Average sensor range requires 1 in each bay area Each floor has 10 such bays Each building has 15 such floors Each campus has 50 buildings 10 such campuses to manage Sum: 2 x 10 x 15 x 50 x 50 = 750,000 Sum: 2 x 10 x 15 x 50 x 50 = 7500 © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Connectivity Define the layout of devices Microsoft Ignite 2016 11/17/2018 2:56 PM Connectivity Define the layout of devices How will they send data? – wifi, Bluetooth, Ethernet Where will the data be consolidated before sending it to the cloud Will the sensors be directly connecting to the cloud? Protocols – MQTT, AMQP, HTTP © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Our scenario Each floor has bays

Our scenario One sensor per bay

One hub per floor that consolidates data for the floor Microsoft Ignite 2016 11/17/2018 2:56 PM Our scenario Bluetooth for sending data to the hub One hub per floor that consolidates data for the floor © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Gateway for each building Microsoft Ignite 2016 11/17/2018 2:56 PM Our scenario Cables/wifi for sending data to the hub Gateway for each building © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data structure JSON/CSV/Text Collection or single messages? Compressed Microsoft Ignite 2016 11/17/2018 2:56 PM Data structure JSON/CSV/Text Collection or single messages? Compressed Our scenario (JSON collection) © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Frequency Factors to consider Criticality Timeliness Connectivity Microsoft Ignite 2016 11/17/2018 2:56 PM Frequency Factors to consider Criticality Timeliness Connectivity Costs We will cover the costs in detail a little later down our journey… © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Security Let’s talk this along with Data Ingestion!

Let’s define our next steps Microsoft Ignite 2016 11/17/2018 2:56 PM Let’s define our next steps Step 1 Create our data sources Step 2 Data Ingestion Step 3: Data processing to identify the extreme temperature areas Step 4 Take action © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Factors to consider Location Throughput Security Microsoft Ignite 2016 11/17/2018 2:56 PM Factors to consider Location Throughput Security Authentication & Authorization Encryption Compression Collection parsing © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Iot HubCost per month (USD) Microsoft Ignite 2016 11/17/2018 2:56 PM Our scenario No. of sensors: 750,000 Throughput requirements: Scenario Frequency (in mins) Message size (in bytes) Throughput (Mb/day) Iot HubCost per month (USD) direct messages 5 240 4943.85 2000 30 823.97 450 Aggregate messages 34816 956.25 50 aggregate messages 159.83 compressed & aggregate messages 5120 140.63 23.44 © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Secured ingestion Device registry Security token Authentication Microsoft Ignite 2016 11/17/2018 2:56 PM Secured ingestion Device registry Security token Authentication © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Device registration Register device with X.509 certificate Microsoft Ignite 2016 11/17/2018 2:56 PM Device registration Register device with X.509 certificate Return device key details © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Secured ingestion Microsoft Ignite 2016 11/17/2018 2:56 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Secured ingestion Microsoft Ignite 2016 11/17/2018 2:56 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Ignite 2016 11/17/2018 2:56 PM Generating SAS Token Note: X.509 certificate does not include skn in the string © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

De-compression & De-aggregation Microsoft Ignite 2016 11/17/2018 2:56 PM De-compression & De-aggregation Generic compression & de-compression code which can be deployed (github) https://github.com/Microsoft/iot-samples/blob/master/AzureFunctions/nodejs/DecompShred.js © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

De-compression & De-aggregation Microsoft Ignite 2016 11/17/2018 2:56 PM De-compression & De-aggregation Generic de-aggregation code, based on configurable de- aggregation pattern https://github.com/Microsoft/iot-samples/blob/master/CollectionParsing/AzureFunctions/NodejsIotHubTrigger/index.js © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data Ingestion – Our scenario Microsoft Ignite 2016 11/17/2018 2:56 PM Data Ingestion – Our scenario Building gateway consolidate & compress data Gateway generate SaS token Data is sent over to Azure IoT Hub over tls using the SaS tokens IoT hub trigger function de- compresses & de-aggregates the data © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data sources & Ingestion Microsoft Ignite 2016 11/17/2018 2:56 PM Data sources & Ingestion In action © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Let’s define our next steps Create our data sources Step 2 Data Ingestion Step 3: Data processing to identify the extreme temperature areas Step 4 Take action

Microsoft Ignite 2016 11/17/2018 2:56 PM Decisions: We need to check if any of the sensors is sending data that is above the benchmark temperature Or below the benchmark temperature Probably store the recorded temperature… observe user patterns Alert if a sensor is not functioning or not sending data Manually corrected temperature © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

So we have two paths: Hot path Cold path Near-real time Microsoft Ignite 2016 11/17/2018 2:56 PM So we have two paths: Near-real time processing Hot path Benchmark alerts Inactive sensor alerts Streaming data Data storage Cold path Store for future use © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Comparing various options Microsoft Ignite 2016 11/17/2018 2:56 PM Comparing various options Option Description Best suited Spark/HDInsight Spark running on custom Hadoop or HDInsight clusters that could process streaming data into both storages Complex processing. For ex, running NLP algorithms on oncoming twitter feeds Application on Virtual machines Custom applications deployed on Virtual machines Migration of legacy applications App services Custom code written, but managed by Azure. Legacy code that could be deployed on PaaS Functions Server less lightweight compute Trigger based operations Stream Analytics Queries in T-SQL that process streaming data in time windows Conditional checking / processing in windows © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Our scenario: Microsoft Ignite 2016 11/17/2018 2:56 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Our scenario: Hot path Cold path Microsoft Ignite 2016 11/17/2018 2:56 PM Our scenario: Hot path Cold path © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data processing In action Microsoft Ignite 2016 11/17/2018 2:56 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Let’s define our next steps Create our data sources Step 2 Data Ingestion Step 3: Data processing to identify the extreme temperature areas Step 4 Take action

Step 4: Take Action

Microsoft Ignite 2016 11/17/2018 2:56 PM Take action What do we want to do if we a bay sensor records too hot or cold? What do we want to do if a sensor is not sending data What do we want to do with the sensor temperature that is being recorded © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Send instructions to move the air draft Take Action What if a bay’s sensor records too hot or too cold? Update live dashboard Alert triggered Send instructions to move the air draft

Our scenario: Microsoft Ignite 2016 11/17/2018 2:56 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Take action

Data action In action Microsoft Ignite 2016 11/17/2018 2:56 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Step 5: Predict? Microsoft Ignite 2016 11/17/2018 2:56 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Going beyond reactive analytics Microsoft Ignite 2016 11/17/2018 2:56 PM Going beyond reactive analytics Temperature data by bays, floor, building & campus captured every half hour manually triggered external weather details occupancy build a temperature prediction model © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Going beyond reactive analytics Weather forecasts Outlook calendar Temperature prediction model Set the temperature Attendance (card swipe details)

Take action

Microsoft Ignite 2016 11/17/2018 2:56 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Reference Github for content: https://github.com/shwetams/IgniteNZ (will go public by Nov 5th, 2016) Github for De-aggregation & De-compression: https://github.com/Microsoft/iot-samples/tree/master/ Whitepaper on various hardware devices: http://www.yole.fr/iso_upload/Samples/Yole_IoT_June_2014_Sample.pdf

11/17/2018 2:56 PM Questions? © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.