September 2016 TechConnect Genius Webinar ControlLogix Redundancy Troubleshooting Brian Widman Technical Support Engineer September, 2016

PowerPoint is an Interactive Reference Side note PowerPoint is an Interactive Reference Many portions will have the same information Focus/scope is on enhanced redundancy (version 16.081 or higher) though techniques will apply to standard as well Last Slide Button throughout IAB View of Redundancy

Agenda Introduction to Redundancy Setup Verify Troubleshooting Demo

Introduction to Redundancy ControlLogix®® Redundancy Dual Chassis Design Full Redundancy Both Chassis match Transparent Primary/Secondary Chassis IP Address Swap ControlNet Node Swap Easy to use No special code Automatic Crossloading of data and logic Treat as one chassis Uses Standard 1756 hardware IAB View of Redundancy Up To: 2 Controllers 7 Communications Modules 1 Redundancy Module

Introduction to Redundancy Operation Basics Application in Primary is automatically crossloaded into the Secondary processor Data changes are sent to secondary at the end of each program. On-Line application changes are also sent from the primary to the secondary. This keeps Primary and Secondary always synchronized System is “Floating Master” type. Each chassis is capable of being Primary Primary Chassis 1756-RM2 Secondary Chassis

Introduction to Redundancy ControlLogix Redundancy is design for High Availability Upgrades can be made while the process is running A locked update switchover allows ControlLogix Redundancy to switchover to a chassis with upgraded firmware and/or hardware. Hardware can be changed and upgraded while the system is running Firmware updates can be made while the system is running Primary and Secondary controllers are always automatically synchronized No data is lost during a switchover I/O stays connected during a switchover (bumpless switchover) Up until version 24 this is outlined in the release notes. Version 24 and above will use Chapter 1 of ControlLogix Redundancy Update and Module Replacement Guidelines.

Agenda Introduction to Redundancy Setup Verify Troubleshooting Demo

Setup Hardware Catalog numbers need to match in each chassis Slot by slot comparison No module requires a specific slot Max modules per rack 1 Redundancy module 2 controllers 7 communication modules (sum of Ethernet and ControlNet modules) Family of processors need to match in the chassis 1756-L7x vs 1756-L6x Series of all modules do not need to match Chassis sizes need to match Redundancy modules are connected together via a single fiber no switches/converters in between

Setup Firmware All firmware needs to be supported in the same bundle (lower/higher is not ok) Ex. 1756-RM2 ships with 20.017 (Boot FW) All modules firmware need to match their partners This can cause series restrictions Ex. 1756-EN2T series C vs D Bundles, hardware, and firmware requirements are outlined in technote 165251

Setup Firmware (continued) Always get the firmware from the bundle on the Product Compatibility and Download Center (PCDC) Firmware is key!

Setup Redundancy Module Configuration Tool (RMCT) Powerful tool used with redundancy Not required but heavily recommended Helps with troubleshooting Accessed from RSLinx (Not standalone) Listed with bundle firmware Not included with the bundle Note: 7.2.7.0 or higher versions of the tool have tools back to 3.6.4.0 installed with it (version shouldn’t be a problem) Compatibility on the RMCT is outlined in technote 29511

Setup Network - Ethernet IP/Subnet/Gateway (everything) needs to be set exactly the same Port Configuration tabs within RSLinx must be identical for partner modules Partners need to be on the same network IP address assigned + 1 needs to be reserved Control of I/O over Ethernet requires version 19.52+ If secondary is set to +1 address you can SYNC but it will not switchover correctly; I/O will show 16#000C. See technote 449462.

Setup Network - ControlNet Set the same node for partner modules Do not use the lowest node numbers Recommendation: Set the node number to the highest in the network There must be 2 additional nodes on the network Cannot have lonely keeper during switchover Node address assigned + 1 needs to be reserved Partners need to be on the same network

Setup Network – Additional Protocols ControlNet and Ethernet are the only network modules supported in the redundancy chassis Networks supported in remote racks (not exhaustive) Remote I/O (blue hose) 1756-RIO 1756-DHRIO DH+ (blue hose) DeviceNet 1756-DNB Guidelines for remote communication modules within technote 43364 (specific to DeviceNet but can be applied to all)

Setup Software Ensure time sync configuration matches in 2 areas* Enable redundancy 1756-RM(2) is not added to the I/O tree Note: Don’t download to the processor until the system is synchronized *Time Sync discussed further in technote 117550

Setup Not supported - Hardware The following hardware is not permitted in a redundancy system 1747-AENTR 1747-ACNR15 1769-AENTR Hardware restrictions listed within technote 586109

Setup Not supported - Software The following configurations are not permitted Unicast connections Remote modules or consumed tags Produced tags can be unicast in version 19.50+ Tip: Check with the L5k export Motion Safety (SIL3 – red modules) Event/inhibited tasks Firmware Supervisor An option available with an SD card in standard systems Outlined under “Features Not Supported” and “Restrictions” in Chapter 1 of the ControlLogix Redundance User Manual

Setup Interface with Redundancy Synchronize the racks before downloading Download to the primary processor only The project will crossload to the secondary processor From a project perspective it’s treated like 1 controller Redundancy is transparent

Agenda Introduction to Redundancy Setup Verify Troubleshooting Demo

Verify Extremely Abbreviated Checklist Ensure Hardware matches Ensure Firmware matches Local communication module partners Are set exactly the same Are on the same network and can see each other Redundancy modules are connected together A complete checklist is in Appendix C of the ControlLogix Redundance User Manual

Verify Test Before downloading to the processor verify the racks are SYNC’d Redundancy module in Rack A will show PRIM and Rack B will show SYNC Download to Rack A processor Ensure PRIM and SYNC still showing on the redundancy modules Go offline with RSLogix 5000 Power down rack A If you don’t go offline first it will kick you offline It may be required to close and reopen RSLogix/Studio 5000 to recover Rack B should now show PRIM on the redundancy module Power on Rack A Rack A redundancy module should display DISQ -> QFNG -> SYNC This could take a few minutes depending on the application size

Agenda Introduction to Redundancy How to Setup How to Verify Troubleshooting Demo

Troubleshooting Basic Idea First you want to verify the Hardware Modules Firmware Network Then you want to verify the Software Best practices with programming Restrictions are accounted for

Troubleshooting Program Checks: Best Practices Unique periods set Unique task priorities Logix 5000 task monitor CPU utilization under 80% Continuous task with System Overhead Time Slice (SOTS) Check software restrictions Obtain the Logix 5000 Task Monitor from the RSLogix 5000 download or search “Studio 5000 tools” on the PCDC Website.

Troubleshooting Quick Fix Remove everything from the rack Add the redundancy modules to the rack Verify the modules SYNC Add another pair of modules Communication modules need to be connected over their respective network Verify the rack SYNCs Repeat steps 4 and 5 until the rack contains all modules Investigate whenever the rack doesn’t SYNC Note: Modules need to see Redundancy Modules on the backplane when powered on initially The Redundancy Module LED display provides the current redundancy status.

Check hardware and Firmware Start Disqualified? Yes Fault on Processor? No Check hardware and Firmware Start Disqualified? 2nd page No Yes Do Program Checks Address the Fault Check communication modules configuration 2nd page Default the Secondary Processor No Verify Communication modules on same network Can we download/is it a new system? Yes Yes Sync? Default both Processors Yes Sync? No No Gather logs and call in Do Quick Fix

Troubleshooting Redundancy Logs Logs are accessed through the RMCT Obtain in CSV format Error codes usually not important Interested in the cause, not the consequences For more information look in technote 55721

Maintenance Recommended to perform the following when there is planned downtime, though it can be done online as well. Hardware (verify functionality) Software (Perform “Defrag”) Power down the primary rack and turn it back on System should switchover and sync Power down the new primary rack and turn back on System should switchover and sync again System can go down Go online with primary and save the project Go offline and download to the primary processor System is running Perform the steps for a Redundant System Update (RSU)* *For more information look in technote 63409

Agenda Introduction to Redundancy How to Setup How to Verify Troubleshooting Demo

Questions?

Thank You For Attending!