1. A M E M B E R O F T H E K E N D A L L G R O U P

2. T4 Network segmentation “methods and best practices”
Smart Networking Expo June 10th 2015
Derek Humphreys, Automation Engineer
T4 Network segmentation “methods and best practices”

3. Goals Understand Segmentation and why it makes sense
Identify the ways in which plant wide Ethernet Networks may be separated
Compare and contrast the methods of segregation

4. A Network History Legacy Networks were segmented by design
Examples : RIO, DH+, ControlNet, DH485, DeviceNet
Each network had a well defined maximum capacity which limited the number of devices to protect performance
By design, new devices did not typically appear on the network
Special adapters or bridges required to attach to the network

5. Typical Initial plant floor Ethernet implementations
Unmanaged Switches
Any network address structure works, at first
Even x.x , although it’s not the best idea
Islands of Automation
No communications beyond the cell

6. It’s just one new cell…
Change IP Addresses…

7. Issues Network continues to grow until the inevitable happens Virus
Broadcast Storm
Could be caused by redundant link between 2 or more switches
Network flooded with traffic, causing possible production loss
Difficult to find the issue on sprawling networks
Run out of IP addresses
Network slows down
Long time to connect
Run out of resources
Difficult to troubleshoot
You don’t know what is on the network
You may no know where it is on the network
Managed switches can help, but one of the best solutions is to make sure the network is properly segmented

8. Why Is This Important? Application Requirements
Function
Information Integration,
Slower Process Automation
Time-critical
Discrete Automation
Motion Control
Communication
Technology
.Net, DCOM, TCP/IP
Industrial Protocols - CIP
Hardware and Software solutions, e.g. CIP Motion, PTP
Period
10 ms to 1000 ms
1 ms to 100 ms
100 µs to 10 ms
Industries
Oil & gas, chemicals, energy, water
Auto, food & beverage, semiconductor,
metals, pharmaceutical
Subset of discrete automation
Applications
Pumps, compressors, mixers, instrumentation
Material handling, filling, labeling, palletizing, packaging
Printing presses, wire drawing, web making, pick & place
Latency and Jitter
Source: ARC Advisory Group

9. Why is this important? Industrial Automation & Control System Convergence
IACS Industrial Automation and Control Systems
Wgb workgroup bridge
Lwap light weight access point
Flat and Open
IACS Network Infrastructure
Flat and Open
Industrial Automation and Control System Network Infrastructure
Structured and Hardened
IACS Network Infrastructure

10. Networking Design Considerations Reference Architectures
Education, design considerations and guidance to help reduce network Latency and Jitter, to help increase the Availability, Integrity and Confidentiality of data, and to help design and deploy a Scalable, Robust, Secure and Future-Ready network infrastructure:
Single Industrial Network Technology
Robust Physical Layer
Segmentation / Structure (modular & scalable building blocks)
Prioritization - Quality of Service (QoS)
Redundant Path Topologies with Resiliency Protocols
Time Synchronization – PTP, CIP Sync, Integrated Motion on the EtherNet/IP network
Multicast Management
Convergence-ready Solutions
Security – Holistic Defense-in-Depth
Scalable Secure Remote Access
Wireless LAN – Autonomous, Unified

11. Industrial Network Design Methodology
Understand application and functional requirements
Devices to be connected – industrial and non-industrial
Data requirements for availability, integrity and confidentiality
Communication patterns, topology and resiliency requirements
Types of traffic – information, control, safety, time synchronization, drive control, voice, video
Develop a logical framework (roadmap)
Migrate from flat networks to structured and hardened networks
Define zones and segmentation based on functional requirements, place applications and devices in the logical framework
Develop a physical framework to align with and support the logical framework
Deploy a Holistic Defense-in-Depth Security Model
Reduce risk, simplify design, and speed deployment:
Use information technology (IT) standards
Follow industrial automation technology (IAT) standards
Utilize reference models and reference architectures
Avoiding Network Sprawl!!
Enabling OEM Convergence-Ready Solutions
MANAGE / MONITOR
IMPLEMENT
AUDIT
DESIGN/PLAN
ASSESS

12. OSI 7-Layer Reference Model

13. Terminology Cell Area Zone, where the production equipment is located
VLAN Virtual Local Area Network
NAT Network Address Translation
Layer 2 Switch
Layer 2 switching uses the (MAC address) from the host's network interface cards (NICs) to decide where to forward frames.
Layer 3 Switch
Layer 3 switching is solely based on (destination) IP address stored in the header of IP datagram 
Router The difference between a layer 3 switch and router is the way the device is making the routing decision
Subnet A subnet is a logical, visible subdivision of an IP network.

14. Structure and Hierarchy Logical Model
Level 5
Enterprise Network
Enterprise
Security Zone
Level 4
, Intranet, etc.
Site Business Planning and Logistics Network
Firewall
Remote Desktop Gateway Services
Patch Management
AV Server
Web
CIP
Industrial
DMZ
Application Mirror
Web Services Operations
Reverse Proxy
Firewall
FactoryTalk Application Server
FactoryTalk Directory
Engineering Workstation
Remote Access Server
Industrial
Security Zone(s)
Level 3
Site Operations
Area Supervisory Control
FactoryTalk Client
FactoryTalk Client
Level 2
Operator Interface
Engineering Workstation
Operator Interface
IACS Industrial Automation and Control Systems
Cell/Area Zones(s)
Basic Control
Level 1
Continuous Process Control
Batch Control
Discrete Control
Drive Control
Safety Control
Level 0
Sensors
Drives
Actuators
Robots
Process
Logical Model, Converged Multi-discipline IACS

15. Segmentation Structured and Hardened Network Infrastructure
Smaller modular building blocks to help 1) minimize network sprawl and 2) build scalable, robust and future-ready network infrastructure
Smaller broadcast domains
Smaller fault domains (e.g. Layer 2 loops)
Smaller domains of trust (security)
Multiple techniques to create smaller network building blocks (Layer 2 domains)
Structure and hierarchy
Logical model – geographical and functional organization of IACS devices
Campus network model - multi-tier switch model – Layer 2 and Layer 3
Logical framework
Segmentation
Multiple network interface cards (NICs) – e.g. CIP bridge
Network Address Translation (NAT) appliance
Virtual Local Area Networks (VLANs)
VLANs with NAT

16. First Segmentation principle, Subnetting
Unlike DeviceNet or ControlNet, there are no limits to the number of EtherNet/IP nodes which can be present on a single EtherNet/IP network.
In theory, this would imply that up to (2^24)-2 = 16,777,214 nodes could be placed on a single, flat network.
In practice, subnetting and VLAN's are used to break up an EtherNet/IP network into smaller, more meaningful segments.
There are no guidelines on how large or small a segment should be.

17. What should we consider ?
<machine number>.<node number> /16 ( )
This addressing scheme implies that both the machine number and the node can be identified immediately from the IP address. It also implies that for identical machines, the same devices can have the same node number.
So, for example, an ENBT card on
machine 1 might have the address
machine 2 might have the address
therefore all machines are on the same subnet.
The implications are as follows:
All nodes will see broadcast traffic, and EtherNet/IP I/O multicast traffic may be seen by all devices
An RSLinx browse will show all of the devices on the subnet - in other words, all machines are visible
Compare ETH to ETHIP driver
All types of EtherNet/IP communication is possible without additional hardware
A faulty device may impact all devices on the network

18. Change the subnet
An alternative strategy might use the same approach, but with a different subnet mask, for example:
<machine number>.<node number> /24 ( )
In this case, the numbering scheme remains the same but the machines are on different subnets. The implication of doing this is as follows:
Broadcast and multicast traffic for one machine is restricted to its own subnet
An RSLinx browse of a subnet will only show the devices for that machine
If used together with a VLAN, a fault on one subnet will not affect the operation of devices on another subnet or VLAN
Communication between machines requires a layer 3 switch or router
Not all types of EtherNet/IP communication is possible between machines

19. 2nd principle, CIP Bridge Multiple Network Interface Cards (NICs)
Isolated networks - two NICs for physical network segmentation
Converged networks - logical segmentation - two NICs for scalability, performance, capacity and flexibility
Plant Network
Level 3
Segmented (using VLANs), Layer 2 Network
Plant Network
Level 3
Layer 2 Network
VLAN 102
Converged
Network
Benefits
Clear network
ownership
demarcation line
Challenges
Limited visibility to control network devices for asset management
Limited future-ready capability
Smaller PACs may not support
Benefits
Plant-wide information
sharing for data collection and
asset management
Future-ready
Challenges
Blurred network ownership
demarcation line
Layer 2 Network
VLAN 103
Control Network
Levels 0-2
Control Network
Levels 0-2

20. One Address, No CIP Bridge
2 Ports,
1 NIC,
1 IP Address

21. 3rd Principle, Network Address Translation (NAT)
NAT is a service that allows the translation of a packet from one IP address to another
Functionality includes Layer 2 and Layer 3 implementation in multiple forms:
NAT One to Many (1:n) – also known as Port Address Translation and allows multiple devices to share one “public” IP address
Most common in consumer routers (in your home)
NAT One to One (1:1) – allows the assignment of a unique “public” IP address to an existing “private” IP address (end device)
The end device can communicate on both “public” and “private” networks by using an “alias” of the IP address physically programmed on the end device
NAT allows a single device, commonly a router, to act as an agent between the Internet (public network) and the private network. For example, this means that only a single, unique IP address is required to represent an entire group of computers.
NAT is a service that allows the translation of a packet from one IP address to another. It can take a number of different forms and work in several different ways, but mapping and lookup tables are the basic tools behind NAT.
The focus of this lab is NAT one to one (which is currently supported on some Rockwell Automation devices) which allows the assignment of a unique “public” IP address to an existing “private” IP address (belonging to an end device). The end device can thus communicate on both the “public” and “private” networks by using an “alias” of the IP address physically programmed on the end device.
Current Rockwell Automation® products support 1:1 NAT

22. NAT Use Cases
Integrating duplicate machines – no changes to machine code, easy to integrate and maintain
Redeploying machines in new location – easy conversion
Support for redundant architectures
IT ready solution for OEM machines – differentiated value
Solution for integration of devices with single network connection
updated

23. Segmentation Network Address Translation (NAT)
Layer 2 NAT Device Key Points
Hardware-based implementation
NAT device does not act as a router and uses 2 translation tables (Inside to Outside and Outside to Inside)
Performance is at wire speed without impacting the CPU
Broadcast traffic in a VLAN can propagate through the NAT boundary
Untranslated traffic, including multicast, can be permitted through the NAT boundary
Layer 3 NAT Device Key Points
Typically a software-based implementation
NAT device acts as the default gateway (router) for the devices on the inside network
NAT device will intercept traffic, perform translation, and route traffic
Translations are handled by the NAT device CPU
Performance of translation directly tied to the loading of the NAT device CPU
Broadcast traffic is stopped at the NAT boundary
Untranslated traffic is not permitted through the NAT device

24. NAT Device Selection Guide
60 Mbp/s
128*          
40 Mbp/s
N/A           
10 Mbp/s 
128   
This table shows the capabilities of the 4 NAT capable Rockwell Automation devices – the Stratix 5700, Stratix 5900, 9300-ENA and the 1783-NATR.
The Stratix 5700 Layer 2 switch is the best option for overall NAT performance and feature set.
The Stratix 5900 Services Router provides many features including software-based NAT translations. Keep overall network requirements and Stratix 5900 resources balanced for optimal performance. For reduced CPU loading, do not exceed 40 Mb/s of network traffic.
The 9300-ENA device is ideal for Studio 5000 controller access, or HMI monitoring of small machines and can handle 10 Mb/s of traffic through the NAT boundary with minimal delay.
The 1783-NATR is a new NAT device with embedded switch technology for DLR (no need for ETAP and separate NAT device on DLR) and Linear Support. It can handle traffic rates of up to 20 Mb/s through the NAT boundary.
20 Mbp/s 32     
* 128 individual NAT Entries per NAT table. An entry can be an entire subnet.

25. Segmentation Network Address Translation (NAT) Appliance
Segmented Networks - Layer 2 (e.g. VLAN) and Layer 3 (e.g. subnet)
Smaller Layer 2 building blocks
Layer 2 Network
Line
Subnet /24
Layer 2 Network
Layer 2 Network
Machine 1
Subnet /24
Machine 2
Subnet /24

26. VLAN
virtual local area network
In computer networking, a single layer-2 network may be partitioned to create multiple distinct broadcast domains, which are mutually isolated so that packets can only pass between them via one or more routers; such a domain is referred to as a virtual local area network, virtual LAN or VLAN.
VLANs address issues such as scalability, security, and network management
Assign different traffic types to a unique VLAN, other than VLAN 1

27. Segmentation Virtual Local Area Networks (VLANs)
Layer 2 network service, VLANs segment a network logically without being restricted by physical connections
VLAN established within or across switches
Data is only forwarded to ports within the same VLAN
Devices within each VLAN can only communicate with other devices on the same VLAN
Segments traffic to restrict unwanted broadcast and multicast traffic
Software configurable using managed switches
Benefits
Ease network changes – minimize network cabling
Simplifies network security management - domains of trust
Increase efficiency
Drive
Controller
= VLAN EtherNet/IP Device
= VLAN 10 - VoIP
= VLAN 42 - Scanners/Cameras

28. Segmentation Virtual Local Area Networks (VLANs)
Layer 2 VLAN Trunking
Independent of physical switch location
Logically group assets by type, role, logical area, physical area or a hybrid of these
Devices communicate as if they are on the same physical segment – no re- cabling required
Software configurable using managed switches
A Layer 3 device (Router or Layer 3 switch) is required to forward traffic between different VLANs
Inter-VLAN routing
VLAN 10
VLAN 102
VLAN 42

29. Segmentation Virtual Local Area Networks (VLANs)
Multi-Layer Switch
Layer 2 VLAN Trunking
Layer 3 Inter-VLAN routing
Layer 3
Switch
Layer 2 Network
Multiple VLANs
Layer 2 Network
Multiple VLANs
Drive
Drive
HMI
HMI
Controller
Controller
= VLAN 102 – EtherNet/IP Device
= VLAN 102 – EtherNet/IP Device
= VLAN 10 - VoIP
= VLAN 10 - VoIP
= VLAN 42 – Scanners/Cameras
= VLAN 42 – Scanners/Cameras

30. Segmentation Virtual Local Area Networks (VLANs)
Smaller Layer 2 building blocks
Single Cell/Area Zone, Single Line, Multiple Machines (vendors)
Line
VLAN17
Subnet /24
I/O
I/O
I/O
VFD
Drive
Controller
Servo
Drive
VFD
Drive
HMI
HMI
Machine 1
VLAN10
Subnet /24
Controller
I/O
Machine 2
VLAN20
Subnet /24
Servo
Drive

31. Segmentation Methods Summary
Subnetting
Multiple NIC
NAT
VLAN
And combinations of multiple techniques

32. No Segmentation (not recommended) Plant-wide / Site-wide Network
Enterprise-wide
Business Systems
Levels 4 & 5 – Data Center
Enterprise Zone
Level IDMZ
Level 3 - Site Operations
Industrial Zone
Levels 0-3
Plant-wide
Site-wide
Operation Systems
Physical or Virtualized Servers
Application Servers & Services Platform
Network Services – e.g. DNS, AD, DHCP, AAA
Remote Access Server (RAS)
Storage Array
Plant LAN – VLAN17 - Layer 2 Domain
Plant IP - Subnet /24, every device requires a unique IP address
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zone #1
Subnet /24
Cell/Area Zone #2
Subnet /24
Cell/Area Zone #3
Subnet /24 32

33. Multiple NIC Segmentation Plant-wide / Site-wide Network
Enterprise-wide
Business Systems
Levels 4 & 5 – Data Center
Enterprise Zone
Level IDMZ
Level 3 - Site Operations
Industrial Zone
Levels 0-3
Plant-wide
Site-wide
Operation Systems
Physical or Virtualized Servers
Application Servers & Services Platform
Network Services – e.g. DNS, AD, DHCP, AAA
Remote Access Server (RAS)
Storage Array
Plant LAN – VLAN17 - Layer 2 Domain
Plant IP - Subnet /24
Line/Area Controller
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zone #1
Subnet /24
Cell/Area Zone #2
Subnet /24
Cell/Area Zone #3
Subnet /24 33

34. NAT Appliance Segmentation Plant-wide / Site-wide Network
Enterprise-wide
Business Systems
Levels 4 & 5 – Data Center
Enterprise Zone
Level IDMZ
Level 3 - Site Operations
Industrial Zone
Levels 0-3
Plant-wide
Site-wide
Operation Systems
Physical or Virtualized Servers
Application Servers & Services Platform
Network Services – e.g. DNS, AD, DHCP, AAA
Remote Access Server (RAS)
Storage Array
Plant LAN – VLAN17 - Layer 2 Domain
Plant IP - Subnet /24
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zone #1
Subnet /24
Cell/Area Zone #2
Subnet /24
Cell/Area Zone #3
Subnet /24 34

35. VLAN Segmentation Plant-wide / Site-wide Network
Enterprise-wide
Business Systems
Levels 4 & 5 – Data Center
Enterprise Zone
Level IDMZ
Level 3 - Site Operations
Industrial Zone
Levels 0-3
Plant-wide
Site-wide
Operation Systems
Physical or Virtualized Servers
Application Servers & Services Platform
Network Services – e.g. DNS, AD, DHCP, AAA
Remote Access Server (RAS)
Storage Array
Plant LAN – VLAN17 - Layer 2 Domain
Plant IP - Subnet /24, every device requires a unique IP address
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zone #1
VLAN10
Subnet /24
Cell/Area Zone #2
VLAN20
Subnet /24
Cell/Area Zone #3
VLAN30
Subnet /24 35

36. VLAN Segmentation with NAT Plant-wide / Site-wide Network
Enterprise-wide
Business Systems
Levels 4 & 5 – Data Center
Enterprise Zone
Level IDMZ
Level 3 - Site Operations
Industrial Zone
Levels 0-3
Plant-wide
Site-wide
Operation Systems
Physical or Virtualized Servers
Application Servers & Services Platform
Network Services – e.g. DNS, AD, DHCP, AAA
Remote Access Server (RAS)
Storage Array
Plant LAN – VLAN17 - Layer 2 Domain
Plant IP - Subnet /24
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zones
Levels 0-2
Cell/Area Zone #1
VLAN10
Subnet /24
Cell/Area Zone #2
VLAN20
Subnet /24
Cell/Area Zone #3
VLAN30
Subnet /24 36

37. Network Segmentation Design and Implementation Considerations
Design smaller modular building blocks to help
1) minimize network sprawl and
2) build scalable, robust and future-ready network infrastructure
Smaller fault domains (e.g. Layer 2 loops)
Smaller broadcast domains
Smaller domains of trust (security)
Multiple techniques to create smaller network building blocks (Layer 2 domains)
Structure and hierarchy
Segmentation

38. Resource