Kent Weare Integration MVP Real World Industrial IoT

Agenda Intro to Industrial IoT Our Journey Our Reference Architecture Demos

About Me Senior Enterprise Architect Azure (Integration) MVP Author Multi-National Power Generation Company – Calgary, Alberta, Canada Azure (Integration) MVP Author InfoQ – Cloud Editor Blogger

What this talk is not about!

What this talk is about Image Source: http://powerplacesoftware.com/wp-content/uploads/2014/01/shutterstock_110533598.jpg http://myriverside.sd43.bc.ca/sohas2015/files/2015/11/Hydro_Electric_Plant-2m4taz2.jpg

What is Industrial IoT (IIoT)? The Industrial Internet of Things (Industrial IoT) is made up of a multitude of devices connected by communications software. The resulting systems, and even the individual devices that comprise it, can monitor, collect, exchange, analyze, and instantly act on information to intelligently change their behavior or their environment -- all without human intervention. Source: https://www.rti.com/industries/iot-faq.html Each windmill in the farm is equipped with many sensors and software that allows it to communicate with all the other windmills and even request repairs. To optimize power generation, each windmill automatically adjusts its settings and behavior based on data it receives and processes from the system, such as changes in wind speed and direction. Such environmental changes may result not only directly from the weather, but also from the behavior of other windmills.

IIoT is not a new discipline… Each windmill in the farm is equipped with many sensors and software that allows it to communicate with all the other windmills and even request repairs. To optimize power generation, each windmill automatically adjusts its settings and behavior based on data it receives and processes from the system, such as changes in wind speed and direction. Such environmental changes may result not only directly from the weather, but also from the behavior of other windmills.

It is the evolution of existing technologies…

IIoT’s roots are in SCADA… What is SCADA? SCADA is an acronym for Supervisory Control and Data Acquisition. It is software and hardware that can be used for process control and gathering real time data from remote locations for exercising control on equipment and conditions. Orlando Moreno

SCADA System Image Source: https://rtidds.files.wordpress.com/2014/06/genericscadasystem.png

Are IIoT and SCADA the same? Image Source: http://blog.sparna.fr/wp-content/uploads/2012/05/comparison.jpg

No, their prescribed implementations are very different vs Image Source: http://ww3.hdnux.com/photos/06/76/34/1828634/7/920x920.jpg But they do address some of the same use case.

SCADA Systems are highly dependent on: Provisioning of Infrastructure Multi-level Networks Firewalls Expensive, proprietary software or protocols An unwillingness to change Which ultimately leads to…

Image Source: http://ww3. hdnux. com/photos/06/76/34/1828634/7/920x920

Ask BMW about their connected car… Am I being too harsh? Ask BMW about their connected car… http://www.bmwblog.com/2016/03/31/bmw-connected-north-america-makes-world-debut-microsoft-build-2016/

Our Journey… Image Source: http://beautyharmonylife.com/wp-content/uploads/2014/12/image-1.jpg

What prompted our change in Architecture? Image Source: http://thisisagoodsign.com/wp-content/uploads/2015/04/organizational-change.jpg

Image Source: https://website-designs

We could have delivered a point solution… http://bodyliterate.org/wp-content/uploads/2015/02/point-a-to-b.gif

But opted for a flexible, layered architecture… Delivered Data Collection Real Time Analytics Future Opportunity Pro-active Alerts Pre-emptive Maintenance Work Order Integration

Disaster Recovery

Disaster Recovery

But can traditional SCADA and IOT Architectures co-exist? http://bodyliterate.org/wp-content/uploads/2015/02/point-a-to-b.gif Arsenal-Tottenham 

But can these Architectures co-exist? http://blog.crisp.se/wp-content/uploads/2013/07/TechnicalDebt-KeepOldCrap-vs-CleanUpOldCrap.png It would not be feasible, from a time or cost perspective, to retro fit everything. Instead we replace the components that make sense within the context of time and budget

What did we implement? http://www.bmwblog.com/2016/03/31/bmw-connected-north-america-makes-world-debut-microsoft-build-2016/

A Hybrid messaging platform that: Feeds existing Historian and allows Plant Operation personnel to monitor that state of devices and Power Generation output (MW) Provides telemetry for downstream reporting, compliance and commercial purposes. Real-time visibility for Field IT Support Flexibility to evolve http://www.bmwblog.com/2016/03/31/bmw-connected-north-america-makes-world-debut-microsoft-build-2016/

What were our timelines? http://www.bmwblog.com/2016/03/31/bmw-connected-north-america-makes-world-debut-microsoft-build-2016/

January – POC, Architecture Reviews, Contract Negotiations February/March – Build and Testing April – Parallel Systems Cutover > 56 million Device Reads processed May – Decommission Legacy Solution

Our Reference Architecture

Microsoft Azure IoT Services Devices Device Connectivity Storage Analytics Presentation & Action Event Hubs SQL Database Machine Learning App Service Service Bus Table/Blob Storage Stream Analytics Power BI External Data Sources DocumentDB HDInsight Notification Hubs Data Factory Mobile Services BizTalk { }

Event Hubs Cloud-scale telemetry ingestion Build 2014 4/11/2018 Event Hubs Cloud-scale telemetry ingestion Compatible with more than a million publishers supporting HTTP, AMQP and MQTT Ingress millions of events per second SAS based security, with unique token per publisher Configurable data retention (1-30 days) Low latency (<10 ms for volatile data) Pluggable with other cloud services like Stream Analytics Event Hub © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Stream Analytics Real-time analytics for Internet of Things solutions Build 2014 4/11/2018 Stream Analytics Stream Analytics Real-time analytics for Internet of Things solutions Calculation Engine Sum, Average, Count, Max, Min Intelligent Windows Tumbling Hopping Sliding Stream millions of events per second Mission critical reliability, performance and predictable results Rapid development with familiar SQL-based language Routes information © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Stream Analytics - Windows Build 2014 4/11/2018 Stream Analytics - Windows Tumbling Window Hopping Window Sliding Window SELECT sensorId, COUNT(*) AS Count FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, TumblingWindow(second, 10) SELECT sensorId, MIN(temp) as temp FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, SlidingWindow(second, 5) HAVING MIN(temp) > 75  SELECT sensorId, COUNT(*) AS Count, AVG(temp) FROM SensorReadings TIMESTAMP BY time GROUP BY sensorId, HoppingWindow(second, 10 , 5) Source: https://blogs.technet.microsoft.com/machinelearning/2015/06/01/the-azure-stream-analytics-query-language/ Windowing Windowing is a core requirement for stream processing applications to perform set-based operations like counts or aggregations over events that arrive within a specified period of time. ASA supports three types of windows: Tumbling, Hopping, and Sliding. Tumbling Windows define a repeating, non-overlapping window of time.   Like Tumbling Windows, Hopping Windows move forward in time by a fixed period but they can overlap with one another. With a Sliding Window, the system is asked to logically consider all possible windows of a given length and output events for cases when the content of the window actually changes – that is, when an event entered or existed the window. © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Service Bus Topics Pub-Sub Messaging Load-levelling Build 2014 4/11/2018 Service Bus Topics Pub-Sub Messaging Load-levelling Output for Stream Analytics Cost effective Cloud Messaging © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Power Bi Datasets Reports Dashboards Real Time & Scheduled Update Self-service data exploration and analysis Office-like experience Cloud and Mobile friendly

BizTalk Server On-Premises Integration Engine Protocol Mediation Data format transformation Routing

Azure Logic Apps Cloud-based Orchestration Notifications (SMS/Email) SaaS Connectivity

Demo 1 - Telemetry Image Source: http://ftsenviro.webfactional.com/products/telemetry/ubicom/files/icon_reliability.png

Demo 1 – Telemetry Architecture Our core use case – deliver telemetry

Demo 1 – Telemetry Architecture C# Simulator – Publish to Event Hub Create Event Hub Connection Instantiate Message Stream Analytics Populate Randomized Tag Values Serialize and Send (Async)

Demo 1 – Telemetry Architecture Message Payload Stream Analytics

Demo 1 – Telemetry Architecture Stream Analytics Input Stream Analytics

Demo 1 – Telemetry Architecture Stream Analytics Output Stream Analytics

Demo 1 – Telemetry Architecture Stream Analytics Query Stream Analytics

Demo 1 – Telemetry Architecture BizTalk Processing Receive Port Send Port BizTalk Pipeline (XML-CSV) Service Bus Topic BizTalk Pipeline (JSON-XML) BizTalk MsgBox Historian File Collector Stream Analytics

Demo 1 – Telemetry Architecture BizTalk Output (de-identified) Stream Analytics

Demo 2 – Real Time Visualizations Image Source: http://www.reverse-it.net/wp-content/uploads/2013/12/module-icon-telemetry.png

Demo 2 – Real Time Visualizations Architecture

Demo 2 – Real Time Visualizations Stream Analytics Inputs Stream Analytics

Demo 2 – Real Time Visualizations Stream Analytics Inputs – Reference Data Stream Analytics

Demo 2 – Real Time Visualizations Stream Analytics Outputs Stream Analytics

Demo 2 – Real Time Visualizations Stream Analytics Output – Power BI Stream Analytics

Demo 2 – Real Time Visualizations Stream Analytics Query Stream Analytics

Demo 2 – Real Time Visualizations Power BI Stream Analytics

Demo 3 – Detecting Absence of Event(s) Stream Analytics

Demo 3 –Absence of Event(s) Architecture Stream Analytics

Demo 3 –Absence of Event(s) Architecture Stream Analytics Query Stream Analytics

Demo 3 –Absence of Event(s) Architecture Logic Apps Stream Analytics

Demo 3 –Absence of Event(s) Architecture Azure Functions Stream Analytics

Demo 3 –Absence of Event(s) Architecture Text Message Stream Analytics

Monitoring Stream Analytics

Monitoring with BizTalk360 Web Endpoint Monitoring BizTalk360 Process Monitoring BizTalk Server BizTalk360

Monitoring with BizTalk360 BizTalk360 Web Endpoint Monitoring Stream Analytics

Monitoring with BizTalk360 API Management Policy Stream Analytics

Monitoring with BizTalk360 Stream Analytics Query Stream Analytics

Monitoring with BizTalk360 BizTalk Receive Location Stream Analytics

Monitoring with BizTalk360 BizTalk360 Process Monitoring Stream Analytics

Lessons Learned Drive adoption through POCs (Fail Fast/Test & Learn) Decision makers like the option to say ‘No’ – give them that opportunity Frame benefits holistically Involve Security early Write ASA queries defensively to avoid job failures Don’t tie Power BI accounts to a user Decouple data collection and analytic streams Leverage Event Hub ‘re-play’ capabilities Power BI can only be configured after you have data Watch for ACS (BizTalk 2013) Don’t forget about monitoring

Image Source: https://memoirsofanadventurer

What’s Next? Challenge Status Quo Focus on plant modernization Need flexibility (investments/divestures) Reduce dependency on expensive proprietary software OPC UA – AMQP Bridge http://tinyurl.com/h93mvdu Build Historian “like” alternative Build out advanced analytics capabilities

Questions? Blog: http://www.MiddlewareInTheCloud.com Twitter: @wearsy InfoQ: http://www.infoq.com BMW Connected Car: https://channel9.msdn.com/events/Build/2016/KEY02 (36m:25s mark)