Building Scalable Ignition Enterprise Architectures Travis Cox Co-Director of Sales Engineering Kent Melville Sales Engineer

Agenda Enterprise challenges Building an enterprise architecture Special considerations Data flow Bandwidth & latency Organization & configuration Remote tags & history Load Balancer Security considerations Connections, configuration, and projects

Enterprise Challenges Lots of devices and data Network communication issues Local control and visibility Local history Visibility of real-time data centrally Centralizing historical data Firewall considerations Bandwidth considerations Multi-site analysis Scalability Security Lots of projects & templates (central configuration)

Enterprise Challenges Locking down templates and windows Health checks for each Ignition server Managing licenses Backups & disaster recovery Upgrading Ignition servers Cost considerations

Building an Enterprise Architecture Understand customers requirements at all locations OT (Machine, Building, Site) IT (Corporate, Cloud) Understand Ignition’s modules and products Understand connections and data flow Understand all challenges Think about redundancy and backups Lots of questions to ask

Standalone HMI

Which Ignition HMI product do I use? Question Which Ignition HMI product do I use?

Standalone HMI Choice Ignition Edge Panel 1 Client Vision Benefits 1 local / 1 remote client 1 week of historical data 1 way email notification Includes drivers (AB, Siemens, Modbus) Restrictions 500 tags No database access No server side scripting Benefits 1 local / 1 remote client Unlimited tags Database access Server side scripting Restrictions Communication drivers separate No history without module No alarm notification without module

Ignition Edge Solutions Ignition Edge Panel Create local HMIs for field devices Ignition Edge Enterprise Synchronize data to a central enterprise server Ignition Edge MQTT Publish field-device data through MQTT

Standard Architecture - Site

Question Do we need redundancy?

Standard Architecture – Site w/ Redundancy

Question What happens at a critical machine when there is communication loss to the central Ignition server at the site?

No Visibility or Control Local HMI

Local History Store & Forward Loss of Data Local History Store & Forward

Store & Forward Choice Ignition Edge Enterprise Tag Historian Module Benefits Cost effective Includes drivers (AB, Siemens, Modbus) Restrictions 500 tags 1-week of cache Benefits Unlimited tags Cache until disk is full Restrictions Communication drivers separate

Question The local Ignition server is now a critical part of the architecture, do I need to redundancy? Do I need to poll the PLC twice?

Loss of Data

Local / Plant Remote / Central Hub & Spoke Local / Plant Remote / Central

Gateway Network The Gateway Network allows you to connect multiple Gateways together over a wide area network, and opens up many distributed features between gateways. The Gateway Network provides the following features: Web sockets provide fast, firewall-friendly 2-way communication over a single configured connection. Setup proxy node Security and SSL Remote tags, history, alarming, and EAM

Hub & Spoke – Many Possibilities

Question What happens when the size of the project gets large? # of tags and/or # of clients

Scale-Out

How do we manage all of these Ignition servers? Question How do we manage all of these Ignition servers?

Enterprise Administration Module

Enterprise Administration Module Manage multiple Gateways from one Gateway. Use the Controller Gateway to coordinate and automate many administrative tasks for Agent Gateways, including: Monitor Agent health and performance Automate Gateway backup and recovery Synchronization projects and resources Deploy modules Central licensing Remote upgrades

Who else needs the data? Realtime? Historical? Corporate? Cloud? Question Who else needs the data? Realtime? Historical? Corporate? Cloud?

MQTT - Pub/Sub Protocol

MQTT

MQTT vs. Gateway Network Pub/sub Get data to more applications Leverage cloud IoT platforms Use with IT/cloud Just for Ignition Maintains single project Alarm acknowledgement at the source Use with OT (Ignition locally and centrally) Both have Single source of tags RBE (report by exception) Store & forward Security and SSL Outbound traffic

Use of Cloud Customers who want to migrate to the cloud Hosting (SaaS model) Leverage cloud IoT platforms for machine learning and business intelligence Unlimited storage Easy to maintain (no physical machines)

Key Factors Requirements Configuration & design Data flow Bandwidth Network latency Security Administration

Data Flow – PLC to Ignition Where is the PLC in relation to Ignition Ports must be open in firewall Polling is heavy on bandwidth Latency factors into speed RBE is better faster but requires more hardware

Data Flow – Gateway Network & MQTT Outbound only No inbound ports need to be opened in the firewall

Data Flow – Client Communication Realtime data Historical data Acknowledging alarms

Bandwidth & Network Latency - Clients Latency impacts speed Best to have local clients (if possible) Reduce number of Gateway calls Decrease client poll rate

Bandwidth & Network Latency – Historical Data Avoid querying data over WAN Have a local database near the Vision server Use tag history splitter (mirror data)

Tag History Splitter Mirrors tag historian data to 2 databases at the same time Both connections go through store & forward Local database should be specified first Ability to query local database first for specific amount of time Keep local database small

Bandwidth & Network Latency – Store & Forward If latency is high increase write size and write time Don’t send data faster than latency time

Tag Paths Organize tag databases Use fully qualified tag paths for real-time and history Use indirection for templates and popups Realtime Tag Binding: [default]Realistic/Realistic0 History Tag Path: [Splitter/ignition-system-name:default]realistic/realistic0

Remote Tag Providers

Remote Tag Providers - Alarms Queried Queried when necessary Heavier on bandwidth (WAN) Lighter on memory Subscribed (recommended) Alarms held in memory Better performance Heavier on memory Lighter on bandwidth (WAN)

Remote Tag Providers – History Querying Gateway Network Queries through Gateway Network Heavier on bandwidth (WAN) No need to mirror data Database (recommended) Queries from local database No bandwidth (WAN) Requires mirroring or replication Specify remote driver and provider

Remote History Storage

Remote History Providers – DB vs. Gateway Network Direct to Database Data is not compressed Latency impacts performance Database must be opened in firewall Through Gateway Network (recommended) Data is compressed Send more data No need to open database in firewall

Remote History Providers – Tag History Splitter Tag History Splitter can send data to: Direct database connection Remote history provider (Gateway Network) Tag history splitter (splitter of splitter)

Non-Gateway Network Services Alarm history (journal) Audit logs Transaction groups Requires direct database access from remote site (highly requested feature)

Frontend Gateways & Load Balancer Hardware or software (F5 Load Balancer) Turn on sticky sessions No state (memory tags, alarms, SFC engines, timer scripts, etc.). Requires dedicated server for that. Get data from I/O servers through Gateway Network and SQL databases Handle authentication through Active Directory or database since it is shared across all frontend Gateways

Security Considerations Attend Steps for Protecting Your Ignition System today at 2:30-3:30 pm by Carl Gould Security Hardening Guide

Security Considerations - Connections Use HTTPS and force SSL (get a trusted certificate) for Gateway webpage, designers, and clients Gateway Network (use SSL, ApprovedOnly connection policy) Security Zones (lock down access by IP or hostname) Security Policies (tag access, alarm acknowledgement, tag history)

Security Considerations - Configuration Lock down access, by role, to: Gateway status Gateway configuration Designer Creating new projects Editing global protected resources Editing tags

Security Considerations – Project Security Designer Roles to view, save, publish, edit protected resources, delete Protect resources (windows, templates, scripts, etc..) Client Access to client Don't use shared passwords or auto-login Allowed messages (7.9.4 lock downs) Role based security on windows and components