1. September 2016 TechConnect Genius Webinar
ControlLogix Redundancy Troubleshooting
Brian Widman
Technical Support Engineer
September, 2016

2. PowerPoint is an Interactive Reference
Side note
PowerPoint is an Interactive Reference
Many portions will have the same information
Focus/scope is on enhanced redundancy (version or higher) though techniques will apply to standard as well
Last Slide Button throughout
IAB View of Redundancy

3. Agenda
Introduction to Redundancy
Setup
Verify
Troubleshooting
Demo

4. Introduction to Redundancy
ControlLogix®® Redundancy
Dual Chassis Design
Full Redundancy
Both Chassis match
Transparent
Primary/Secondary Chassis
IP Address Swap
ControlNet Node Swap
Easy to use
No special code
Automatic Crossloading of data and logic
Treat as one chassis
Uses Standard 1756 hardware
IAB View of Redundancy
Up To:
2 Controllers
7 Communications Modules
1 Redundancy Module

5. Introduction to Redundancy
Operation Basics
Application in Primary is automatically crossloaded into the Secondary processor
Data changes are sent to secondary at the end of each program.
On-Line application changes are also sent from the primary to the secondary.
This keeps Primary and Secondary always synchronized
System is “Floating Master” type. Each chassis is capable of being Primary
Primary Chassis
1756-RM2
Secondary Chassis

6. Introduction to Redundancy
ControlLogix Redundancy is design for High Availability
Upgrades can be made while the process is running
A locked update switchover allows ControlLogix Redundancy to switchover to a chassis with upgraded firmware and/or hardware.
Hardware can be changed and upgraded while the system is running
Firmware updates can be made while the system is running
Primary and Secondary controllers are always automatically synchronized
No data is lost during a switchover
I/O stays connected during a switchover (bumpless switchover)
Up until version 24 this is outlined in the release notes.
Version 24 and above will use Chapter 1 of ControlLogix Redundancy Update and Module Replacement Guidelines.

7. Agenda
Introduction to Redundancy
Setup
Verify
Troubleshooting
Demo

8. Setup Hardware Catalog numbers need to match in each chassis
Slot by slot comparison
No module requires a specific slot
Max modules per rack
1 Redundancy module
2 controllers
7 communication modules (sum of Ethernet and ControlNet modules)
Family of processors need to match in the chassis
1756-L7x vs 1756-L6x
Series of all modules do not need to match
Chassis sizes need to match
Redundancy modules are connected together via a single fiber
no switches/converters in between

9. Setup
Firmware
All firmware needs to be supported in the same bundle (lower/higher is not ok)
Ex RM2 ships with (Boot FW)
All modules firmware need to match their partners
This can cause series restrictions
Ex EN2T series C vs D
Bundles, hardware, and firmware requirements are outlined in technote

10. Setup Firmware (continued)
Always get the firmware from the bundle on the Product Compatibility and Download Center (PCDC)
Firmware is key!

11. Setup Redundancy Module Configuration Tool (RMCT)
Powerful tool used with redundancy
Not required but heavily recommended
Helps with troubleshooting
Accessed from RSLinx (Not standalone)
Listed with bundle firmware
Not included with the bundle
Note: or higher versions of the tool have tools back to installed with it (version shouldn’t be a problem)
Compatibility on the RMCT is outlined in technote 29511

12. Setup Network - Ethernet
IP/Subnet/Gateway (everything) needs to be set exactly the same
Port Configuration tabs within RSLinx must be identical for partner modules
Partners need to be on the same network
IP address assigned + 1 needs to be reserved
Control of I/O over Ethernet requires version
If secondary is set to +1 address you can SYNC but it will not switchover correctly; I/O will show 16#000C. See technote

13. Setup Network - ControlNet Set the same node for partner modules
Do not use the lowest node numbers
Recommendation: Set the node number to the highest in the network
There must be 2 additional nodes on the network
Cannot have lonely keeper during switchover
Node address assigned + 1 needs to be reserved
Partners need to be on the same network

14. Setup Network – Additional Protocols
ControlNet and Ethernet are the only network modules supported in the redundancy chassis
Networks supported in remote racks (not exhaustive)
Remote I/O (blue hose)
1756-RIO
1756-DHRIO
DH+ (blue hose)
DeviceNet
1756-DNB
Guidelines for remote communication modules within technote (specific to DeviceNet but can be applied to all)

15. Setup Software Ensure time sync configuration matches in 2 areas*
Enable redundancy
1756-RM(2) is not added to the I/O tree
Note: Don’t download to the processor until the system is synchronized
*Time Sync discussed further in technote

16. Setup Not supported - Hardware
The following hardware is not permitted in a redundancy system
1747-AENTR
1747-ACNR15
1769-AENTR
Hardware restrictions listed within technote

17. Setup Not supported - Software
The following configurations are not permitted
Unicast connections
Remote modules or consumed tags
Produced tags can be unicast in version
Tip: Check with the L5k export
Motion
Safety (SIL3 – red modules)
Event/inhibited tasks
Firmware Supervisor
An option available with an SD card in standard systems
Outlined under “Features Not Supported” and “Restrictions” in Chapter 1 of the ControlLogix Redundance User Manual

18. Setup Interface with Redundancy
Synchronize the racks before downloading
Download to the primary processor only
The project will crossload to the secondary processor
From a project perspective it’s treated like 1 controller
Redundancy is transparent

19. Agenda
Introduction to Redundancy
Setup
Verify
Troubleshooting
Demo

20. Verify Extremely Abbreviated Checklist Ensure Hardware matches
Ensure Firmware matches
Local communication module partners
Are set exactly the same
Are on the same network and can see each other
Redundancy modules are connected together
A complete checklist is in Appendix C of the ControlLogix Redundance User Manual

21. Verify
Test
Before downloading to the processor verify the racks are SYNC’d
Redundancy module in Rack A will show PRIM and Rack B will show SYNC
Download to Rack A processor
Ensure PRIM and SYNC still showing on the redundancy modules
Go offline with RSLogix 5000
Power down rack A
If you don’t go offline first it will kick you offline
It may be required to close and reopen RSLogix/Studio 5000 to recover
Rack B should now show PRIM on the redundancy module
Power on Rack A
Rack A redundancy module should display DISQ -> QFNG -> SYNC
This could take a few minutes depending on the application size

22. Agenda Introduction to Redundancy How to Setup How to Verify
Troubleshooting
Demo

23. Troubleshooting Basic Idea First you want to verify the Hardware
Modules
Firmware
Network
Then you want to verify the Software
Best practices with programming
Restrictions are accounted for

24. Troubleshooting Program Checks: Best Practices Unique periods set
Unique task priorities
Logix 5000 task monitor
CPU utilization under 80%
Continuous task with System Overhead Time Slice (SOTS)
Check software restrictions
Obtain the Logix 5000 Task Monitor from the RSLogix 5000 download or search “Studio 5000 tools” on the PCDC Website.

25. Troubleshooting Quick Fix
Remove everything from the rack
Add the redundancy modules to the rack
Verify the modules SYNC
Add another pair of modules
Communication modules need to be connected over their respective network
Verify the rack SYNCs
Repeat steps 4 and 5 until the rack contains all modules
Investigate whenever the rack doesn’t SYNC
Note: Modules need to see Redundancy Modules on the backplane when powered on initially
The Redundancy Module LED display provides the current redundancy status.

26. Check hardware and Firmware Start Disqualified?
Yes
Fault on Processor? No
Check hardware and Firmware
Start
Disqualified?
2nd page No
Yes
Do Program Checks
Address the Fault
Check communication modules configuration
2nd page
Default the Secondary Processor No
Verify Communication modules on same network
Can we download/is it a new system?
Yes
Yes
Sync?
Default both Processors
Yes
Sync? No No
Gather logs and call in
Do Quick Fix

27. Troubleshooting Redundancy Logs Logs are accessed through the RMCT
Obtain in CSV format
Error codes usually not important
Interested in the cause, not the consequences
For more information look in technote 55721

28. Maintenance
Recommended to perform the following when there is planned downtime, though it can be done online as well.
Hardware (verify functionality)
Software (Perform “Defrag”)
Power down the primary rack and turn it back on
System should switchover and sync
Power down the new primary rack and turn back on
System should switchover and sync again
System can go down
Go online with primary and save the project
Go offline and download to the primary processor
System is running
Perform the steps for a Redundant System Update (RSU)*
*For more information look in technote 63409

29. Agenda Introduction to Redundancy How to Setup How to Verify
Troubleshooting
Demo

30. Questions?

31. Thank You For Attending!