2. Building Open & Scalable Multi-Site Enterprise Architectures
Travis Cox
Co-Director of Sales Engineering, Inductive Automation

3. Agenda What does an Enterprise architecture look like?
Site/plant
Remote locations
Corporate, DMZ
Cloud
Enterprise challenges
Goals & key factors
Understanding your requirements, objectives, and network
Building an enterprise architecture
Configuration, best practices, security

4. Enterprise Architecture

5. Enterprise Challenges
Amount of devices & data
Faster rates
Loss of communication, slow communication, high latency
Maintaining local control
Centralizing all data (real-time & historical)
Security
Management
Scalability
Business demands (data, machine learning, analytics, cloud)
and more…

6. Today’s Goals Unlimited Possibilities
Understand Ignition’s products, modules, and features
Provide examples
Provide tools and best practices
Provide tuning tips

7. Key Factors Requirements Configuration & design Data flow Bandwidth
Network latency
Security
Administration

8. Understanding Requirements, Objectives, and Network
Understand requirements at all levels (machine, site, corporate, cloud)
What functionality do I need locally? Centrally? Cloud?
Do I need redundancy?
Understand minimum requirements for Ignition
CPU, Memory, Disk, NIC
Physical vs. Virtual
Understand network (architecture, bandwidth, latency, firewalls)
Purdue model, DMZ
Understand all connections and data flow
Outbound/inbound, firewalls, ports, protocols
Understand Ignition, modules, configuration, Gateway Network, MQTT
What happens when I lose communication to corporate? What do I need locally? Store & forward, visualization, control, alarm notification, local history?
What do I need centrally? Realtime data? Historical data? Do I need to tie into the cloud for storage, machine learning, analytics?
How is the virtual environment setup? Does Ignition have the proper resources? Dedicated or pooled resources? Is the VM host over resourced?
Don’t take on everything. Take on the architecture in strides.

9. Building an Enterprise Architecture
Each plant is independent and can run on its own. Talk about what is in the picture.

10. Site / Plant Components
5 Critical Components
Ignition Edge
Ignition’s Gateway Network
MQTT
Critical Asset
Redundancy

11. Easily Extend Ignition to the Edge of Your Network
What is Ignition Edge?
Easily Extend Ignition to the Edge of Your Network
Ignition Edge is a new line of lightweight, limited, low-cost Ignition products designed specifically for embedding into field and OEM devices at the edge of the network.
Ignition Edge products are priced for the edge of your network so it’s more affordable than ever to extend Ignition all the way to the edge of your network.

12. What is Ignition Edge?
Edge Panel
Edge Enterprise
Edge MQTT

13. Ignition Edge Features & Benefits
Access data from PLCs & OPC-UA servers
Features unlimited tags (as of 7.9.9)
Equipped with OPC-UA, including Modbus, Siemens, and Allen-Bradley drivers (Other Ignition-supported drivers, such as DNP3, can be added onto Ignition Edge for an additional cost)
Work on Windows (any version), and OSX, Linux & more
Work seamlessly with Ignition systems

14. Site / Plant Components
Ignition Edge
Ignition Edge Enterprise
Ignition Edge MQTT
Ignition Edge Panel

15. What is Ignition’s Gateway Network?
The Gateway Network allows you to connect multiple Gateways together over a wide area network, and opens up many distributed features between gateways. The Gateway Network provides the following features:
Web sockets provide fast, firewall-friendly 2-way communication over a single configured connection
Setup proxy node
Security and SSL
Remote tags, history, alarming, and EAM

16. Gateway Network Setup Gateway Network Just for Ignition
Outbound connection
Bi-directional
Web sockets
RBE
Secure (port 8060)

17. Gateway Network Setup
Each plant is independent and can run on its own. Talk about what is in the picture.

18. Tip #1: Name Ignition Servers
Name each server uniquely and properly
Used to identify servers for tag history and Gateway Network services
Important for remote services & EAM
Configure names before setting up tag history or Gateway Network

19. Tip #2: Connect Up Connect local servers to central servers
Easier to open firewalls on central servers vs. local firewalls

20. Tip #3: Connect Only to Master
Only connect to master node of redundant pair
Connection is aware of both servers
Don’t make 2 outgoing connections from the local server

21. Gateway Network Services
Remote tags
Remote alarm notification
Remote history
Enterprise Administration Module (EAM)

22. Gateway Network Services: Remote Tags
Tags exist on local Gateway
Setup remote tag provider on higher level server
Real-time status and control
Alarm status & acknowledgement
Query historical data
Only subscribes to tags needed
Remote tag management

23. Gateway Network Services: Remote Tags

24. Tip #4: Name Real-time Tag Providers Properly
Never use “default”
Give proper names for each Ignition server
Make sure names are unique across all Ignition servers in the enterprise
Make sure the remote tag provider has the same name
edge1
(local)
edge1
(remote)

25. Tip #5: Use Fully Qualified Tag Paths
Real-time Tag Binding:
[edge1]path/to/my/tag
History Tag Path:
[splitter/ignition-system-name:edge1]path/to/my/tag
edge1
(local)
edge1
(remote)
Use tag indirection and include tag provider name
Local tag path is identical to remote tag path

26. Gateway Network Setup edge1 edge1 edge1
(local)
edge1
(remote)
edge1
(remote)
Each plant is independent and can run on its own. Talk about what is in the picture.

27. Tip #6: Use Subscribed Mode for Alarms
Alarms held in memory
Better performance
Heavier on memory
Lighter on bandwidth (WAN)
Configured on remote tag provider

28. Remote Tag History Querying
Gateway Network
Queries through Gateway Network
Heavier on bandwidth (WAN)
No need to mirror data
Doesn’t require remote tag history provider to be setup, just simply configured on remote tag provider

29. “Gateway Network” History Access Mode

30. Tip #6: Remote Tag History Querying
Database
Queries from local database
No bandwidth (WAN)
Requires mirroring or replication
Specify remote driver and provider
Doesn’t require remote tag history provider to be setup, just simply configured on remote tag provider

31. Tag History Splitter
Mirrors tag historian data to 2 databases at the same time
Both connections go through store & forward
Local database should be specified first
Ability to query local database first for specific amount of time
Keep local database small

32. Gateway Network Services: Tag History Splitter

33. Tip #7: Use “Database” History Access Mode

34. Gateway Network Services: Remote Alarm Notification

35. Gateway Network Services: Remote Alarm Notification
A single remote alarm notification profile unlocks 2 features
Local pipeline, remote alarm notification profile on notification block
Send alarm to remote pipeline directly
All remote pipelines visible in alarm configuration

36. Gateway Network Services: Remote History
Store history on central database
No local database required
Store & Forward
Compresses data over Gateway Network
Ignition Edge Enterprise = 1 week history buffer

37. Gateway Network Services: Remote History

38. Remote Tag History Bandwidth & Latency Concerns

39. Tip #8: Remote Tag History Bandwidth & Latency Concerns
If latency is high increase write size and write time
Slower connections = send more data slower
Don’t send data faster than latency time
Configured on store & forward connection

40. Gateway Network Services: EAM
Manage multiple Gateways from one Gateway. Use the Controller Gateway to coordinate and automate many administrative tasks for Agent Gateways, including:
Monitor Agent health and performance
Automate Gateway backup and recovery
Synchronization projects and resources
Deploy modules
Central licensing
Remote upgrades

41. Gateway Network Services: EAM

42. Gateway Network Services: EAM
Agents
Agents
Agents
Agents
Agents
Controller
Agents
Agents
Proxy through Gateway

43. Tip #9: Best Practices for Security
Use HTTPS/TLS for everything
Gateway Network (use SSL, ApprovedOnly connection policy)
Security Zones (lock down access by IP or hostname)
Security Policies (tag access, alarm acknowledgement, tag history)
Gateway/Project Role-based Policies

44. Gateway Network Security

45. Gateway Network Service Security
Lock down:
Tag Access / Management
History Access / Storage
Alarm Notification
Alarm Status (ack, shelve)

46. Non Gateway Network Services
Alarm history (journal)
Audit logs
Transaction groups
Requires direct database access from remote site (highly requested feature)

47. What is MQTT? Message Queuing Telemetry Transport
MQTT is a machine-to-machine (M2M) data transfer protocol that is quickly becoming the leading messaging protocol for the Industrial Internet of Things (IIoT)

48. MQTT Architecture

49. Why MQTT? Decouples devices from applications Low bandwidth
Report by Exception (RBE)
TLS security (port 8883)
Access Control Lists (ACLs)
Outbound connection only (no inbound firewall rules)
Stateful awareness
Single source of truth
Plug and play functionality
Eliminates cutovers (parallel applications)

50. Leading Protocol

51. MQTT = Future Proofing Vibrant ecosystem Gateways Sensors Applications
RTUs, PLCs
Remote I/O

52. MQTT Sparkplug Specification
Sparkplug is a specification that defines how to use MQTT in a mission critical, real time environment.
Eclipse Tahu Project
Defines MQTT Topic Namespace
spBv1.0/group/DDATA/edgenode/device
Defines MQTT Payload Definition
Defines MQTT State Management
High Availability / Redundancy / Scale

53. Why MQTT & Gateway Network?
Ignition only
Management (projects & tags)
EAM
Remote history (select tags)
Remote alarm ack & notification
Open standard
Decouple devices from applications
Future proofing (ecosystem)
Access to 3rd party
Tag exists centrally
Get data to cloud
It is not about vs. but about best of both worlds

54. Building an Enterprise Architecture

55. Frontend Gateways & Load Balancer
Hardware or software (F5 Load Balancer)
Turn on sticky sessions
No state (memory tags, alarms, SFC engines, timer scripts, etc.). Requires dedicated server for that.
Get data from I/O servers through Gateway Network and SQL databases
Handle authentication through shared authorization such as Active Directory or federated identity.

56. Building an Enterprise Architecture

57. Building an Enterprise Architecture
Customers who want to migrate to the cloud
Hosting (SaaS model)
Leverage cloud IoT platforms for machine learning and business intelligence
Unlimited storage
Easy to maintain (no physical machines)
Perspective is a game changer