Building Open & Scalable Multi-Site Enterprise Architectures Travis Cox Co-Director of Sales Engineering, Inductive Automation

Agenda What does an Enterprise architecture look like? Site/plant Remote locations Corporate, DMZ Cloud Enterprise challenges Goals & key factors Understanding your requirements, objectives, and network Building an enterprise architecture Configuration, best practices, security

Enterprise Architecture

Enterprise Challenges Amount of devices & data Faster rates Loss of communication, slow communication, high latency Maintaining local control Centralizing all data (real-time & historical) Security Management Scalability Business demands (data, machine learning, analytics, cloud) and more…

Today’s Goals Unlimited Possibilities Understand Ignition’s products, modules, and features Provide examples Provide tools and best practices Provide tuning tips

Key Factors Requirements Configuration & design Data flow Bandwidth Network latency Security Administration

Understanding Requirements, Objectives, and Network Understand requirements at all levels (machine, site, corporate, cloud) What functionality do I need locally? Centrally? Cloud? Do I need redundancy? Understand minimum requirements for Ignition CPU, Memory, Disk, NIC Physical vs. Virtual Understand network (architecture, bandwidth, latency, firewalls) Purdue model, DMZ Understand all connections and data flow Outbound/inbound, firewalls, ports, protocols Understand Ignition, modules, configuration, Gateway Network, MQTT What happens when I lose communication to corporate? What do I need locally? Store & forward, visualization, control, alarm notification, local history? What do I need centrally? Realtime data? Historical data? Do I need to tie into the cloud for storage, machine learning, analytics? How is the virtual environment setup? Does Ignition have the proper resources? Dedicated or pooled resources? Is the VM host over resourced? Don’t take on everything. Take on the architecture in strides.

Building an Enterprise Architecture Each plant is independent and can run on its own. Talk about what is in the picture.

Site / Plant Components 5 Critical Components Ignition Edge Ignition’s Gateway Network MQTT Critical Asset Redundancy

Easily Extend Ignition to the Edge of Your Network What is Ignition Edge? Easily Extend Ignition to the Edge of Your Network Ignition Edge is a new line of lightweight, limited, low-cost Ignition products designed specifically for embedding into field and OEM devices at the edge of the network. Ignition Edge products are priced for the edge of your network so it’s more affordable than ever to extend Ignition all the way to the edge of your network.

What is Ignition Edge? Edge Panel Edge Enterprise Edge MQTT

Ignition Edge Features & Benefits Access data from PLCs & OPC-UA servers Features unlimited tags (as of 7.9.9) Equipped with OPC-UA, including Modbus, Siemens, and Allen-Bradley drivers (Other Ignition-supported drivers, such as DNP3, can be added onto Ignition Edge for an additional cost) Work on Windows (any version), and OSX, Linux & more Work seamlessly with Ignition systems

Site / Plant Components Ignition Edge Ignition Edge Enterprise Ignition Edge MQTT Ignition Edge Panel

What is Ignition’s Gateway Network? The Gateway Network allows you to connect multiple Gateways together over a wide area network, and opens up many distributed features between gateways. The Gateway Network provides the following features: Web sockets provide fast, firewall-friendly 2-way communication over a single configured connection Setup proxy node Security and SSL Remote tags, history, alarming, and EAM

Gateway Network Setup Gateway Network Just for Ignition Outbound connection Bi-directional Web sockets RBE Secure (port 8060)

Gateway Network Setup Each plant is independent and can run on its own. Talk about what is in the picture.

Tip #1: Name Ignition Servers Name each server uniquely and properly Used to identify servers for tag history and Gateway Network services Important for remote services & EAM Configure names before setting up tag history or Gateway Network

Tip #2: Connect Up Connect local servers to central servers Easier to open firewalls on central servers vs. local firewalls

Tip #3: Connect Only to Master Only connect to master node of redundant pair Connection is aware of both servers Don’t make 2 outgoing connections from the local server

Gateway Network Services Remote tags Remote alarm notification Remote history Enterprise Administration Module (EAM)

Gateway Network Services: Remote Tags Tags exist on local Gateway Setup remote tag provider on higher level server Real-time status and control Alarm status & acknowledgement Query historical data Only subscribes to tags needed Remote tag management

Gateway Network Services: Remote Tags

Tip #4: Name Real-time Tag Providers Properly Never use “default” Give proper names for each Ignition server Make sure names are unique across all Ignition servers in the enterprise Make sure the remote tag provider has the same name edge1 (local) edge1 (remote)

Tip #5: Use Fully Qualified Tag Paths Real-time Tag Binding: [edge1]path/to/my/tag History Tag Path: [splitter/ignition-system-name:edge1]path/to/my/tag edge1 (local) edge1 (remote) Use tag indirection and include tag provider name Local tag path is identical to remote tag path

Gateway Network Setup edge1 edge1 edge1 (local) edge1 (remote) edge1 (remote) Each plant is independent and can run on its own. Talk about what is in the picture.

Tip #6: Use Subscribed Mode for Alarms Alarms held in memory Better performance Heavier on memory Lighter on bandwidth (WAN) Configured on remote tag provider

Remote Tag History Querying Gateway Network Queries through Gateway Network Heavier on bandwidth (WAN) No need to mirror data Doesn’t require remote tag history provider to be setup, just simply configured on remote tag provider

“Gateway Network” History Access Mode

Tip #6: Remote Tag History Querying Database Queries from local database No bandwidth (WAN) Requires mirroring or replication Specify remote driver and provider Doesn’t require remote tag history provider to be setup, just simply configured on remote tag provider

Tag History Splitter Mirrors tag historian data to 2 databases at the same time Both connections go through store & forward Local database should be specified first Ability to query local database first for specific amount of time Keep local database small

Gateway Network Services: Tag History Splitter

Tip #7: Use “Database” History Access Mode

Gateway Network Services: Remote Alarm Notification

Gateway Network Services: Remote Alarm Notification A single remote alarm notification profile unlocks 2 features Local pipeline, remote alarm notification profile on notification block Send alarm to remote pipeline directly All remote pipelines visible in alarm configuration

Gateway Network Services: Remote History Store history on central database No local database required Store & Forward Compresses data over Gateway Network Ignition Edge Enterprise = 1 week history buffer

Gateway Network Services: Remote History

Remote Tag History Bandwidth & Latency Concerns

Tip #8: Remote Tag History Bandwidth & Latency Concerns If latency is high increase write size and write time Slower connections = send more data slower Don’t send data faster than latency time Configured on store & forward connection

Gateway Network Services: EAM Manage multiple Gateways from one Gateway. Use the Controller Gateway to coordinate and automate many administrative tasks for Agent Gateways, including: Monitor Agent health and performance Automate Gateway backup and recovery Synchronization projects and resources Deploy modules Central licensing Remote upgrades

Gateway Network Services: EAM

Gateway Network Services: EAM Agents Agents Agents Agents Agents Controller Agents Agents Proxy through Gateway

Tip #9: Best Practices for Security Use HTTPS/TLS for everything Gateway Network (use SSL, ApprovedOnly connection policy) Security Zones (lock down access by IP or hostname) Security Policies (tag access, alarm acknowledgement, tag history) Gateway/Project Role-based Policies

Gateway Network Security

Gateway Network Service Security Lock down: Tag Access / Management History Access / Storage Alarm Notification Alarm Status (ack, shelve)

Non Gateway Network Services Alarm history (journal) Audit logs Transaction groups Requires direct database access from remote site (highly requested feature)

What is MQTT? Message Queuing Telemetry Transport MQTT is a machine-to-machine (M2M) data transfer protocol that is quickly becoming the leading messaging protocol for the Industrial Internet of Things (IIoT)

MQTT Architecture

Why MQTT? Decouples devices from applications Low bandwidth Report by Exception (RBE) TLS security (port 8883) Access Control Lists (ACLs) Outbound connection only (no inbound firewall rules) Stateful awareness Single source of truth Plug and play functionality Eliminates cutovers (parallel applications)

Leading Protocol

MQTT = Future Proofing Vibrant ecosystem Gateways Sensors Applications RTUs, PLCs Remote I/O

MQTT Sparkplug Specification Sparkplug is a specification that defines how to use MQTT in a mission critical, real time environment. Eclipse Tahu Project Defines MQTT Topic Namespace spBv1.0/group/DDATA/edgenode/device Defines MQTT Payload Definition Defines MQTT State Management High Availability / Redundancy / Scale

Why MQTT & Gateway Network? Ignition only Management (projects & tags) EAM Remote history (select tags) Remote alarm ack & notification Open standard Decouple devices from applications Future proofing (ecosystem) Access to 3rd party Tag exists centrally Get data to cloud It is not about vs. but about best of both worlds

Building an Enterprise Architecture

Frontend Gateways & Load Balancer Hardware or software (F5 Load Balancer) Turn on sticky sessions No state (memory tags, alarms, SFC engines, timer scripts, etc.). Requires dedicated server for that. Get data from I/O servers through Gateway Network and SQL databases Handle authentication through shared authorization such as Active Directory or federated identity.

Building an Enterprise Architecture

Building an Enterprise Architecture Customers who want to migrate to the cloud Hosting (SaaS model) Leverage cloud IoT platforms for machine learning and business intelligence Unlimited storage Easy to maintain (no physical machines) Perspective is a game changer