1. Lessons Learned - SNS Linac PPS Phased Upgrade to Safety PLC
John Bombard
Protection Systems Engineer
Spallation Neutron Source

2. Overview Summary of SNS Linac Safety PLC Upgrade Design Lessons
Installation Lessons
Operational Lessons
Questions or Comments?

3. SNS Linac Safety PLC Upgrade
The SNS Linac PPS mitigates radiation hazards from beam or other radiation sources in the Front-End “Ion Source” as well as the linear accelerator tunnel sections
5 Monitored Entry/Exit Doors
15 Emergency Stop / Sweep Stations with Status Lights
43 Controlled Devices: 3 Ion Source, 15 RF Modulators, 25 RF Amplifiers
Upgrade effort converts radiation safety controls to SIL rated safety class equipment
Get ahead of obsolescence issues using non-safety controllers in a safety application
First phase is the conversion of the Front End Ion Source

4. SNS Linac Safety PLC Upgrade
Historically used Allen-Bradley ControlLogix programmable logic controllers
Two independent systems, we call “PLC-A” and “PLC-B”
Software code maintained by two engineer/programmers
Follows A-B recommendations for using ControlLogix in a SIL 2 configuration

5. SNS Linac Safety PLC Upgrade
Upgrade replaces ControlLogix with its safety-rated equivalent GuardLogix
Two GuardLogix systems, maintaining PLC-A/PLC-B overall architecture
Safety field devices in a “Category 2” (“SIL 1” or “PL b/c”) configuration
One Input Device and One Output Device per PLC, with diagnostic monitoring
The duplex SIL 1 systems can meet SIL 2 reliability requirements

6. SNS Linac Safety PLC Upgrade
Upgrade is a “phased install”
Processor modules and one of five large control racks was updated with GuardLogix safety hardware
Racks were a complete remove and replace project with all-new hardware
Remaining 4 racks still using all ControlLogix hardware to control safety devices and will be upgraded in a future effort
Upgrade required new network infrastructure
The GuardLogix safety hardware only works over Ethernet, but can still control non- safety ControlLogix hardware over other busses
Lesson: Complete remove/replace allows for extended off-line testing and verification, allowing more efficient use of accelerator downtime to install with most of the intrusive testing already done.

7. Design Lessons - Hybrid Network Types
New GuardLogix safety hardware requires Ethernet communications
Existing ControlLogix hardware installed using ControlNet communications
Original design had Ethernet between new GuardLogix processor and the new field rack, but kept ControlNet between upgraded field rack and the remaining 4 non-upgraded racks
Source: Rockwell Automation Knowledge Base

8. Design Lessons - Hybrid Network Types
Later identified this topology would not work at all in practice due to incompatibilities
Caught early during design process
Decision made to convert all racks, including those not being upgraded, to Ethernet
Added an additional ~$70k in unexpected equipment cost
Lesson: Validate new network architecture with manufacturer support engineering or with internal testing

9. Design Lessons – Integrating Pulse Testing
Safety field devices in a “Category 2” configuration for GuardLogix requires the use of “Pulse Testing”
The voltage supply feeding a switch or actuator is replaced with a pulse train to constantly verify the input’s ability to sense transitions
In an ideal configuration, each device should have a different pulse

10. Design Lessons – Integrating Pulse Testing
Most existing field devices at SNS used intermediate junction boxes to fan out DC power for actuators
Example: 4 conductor cable was pulled for 3 signals, however proper pulse test implementation requires 6 conductors
Replacing these cables and boxes was outside the scope of this phase
Lesson: Proper implementation of diagnostic pulse testing will require design changes to field devices, and likely require new cabling

11. Design Lessons – Software Coding Methods
GuardLogix at a basic level uses two user programs, a “Standard Task” and a “Safety Task”
The “Safety Task” is logic run on the redundant high reliability hardware
This task can only use designated safety memory and safety I/O directly.
Standard I/O may be used with a ”mapping” procedure
The “Standard Task” is logic run like a standard ControlLogix
At SNS, the PPS logic was already a standard ControlLogix program

12. Design Lessons – Software Coding Methods
Two different programmers are used for the “PLC-A” and “PLC- B Systems
The PLC-A programmer chose to re-use the existing ControlLogix program running entirely in the Standard Task, only using the Safety Task to perform the mapping to the Safety I/O that is only accessible from the Safety Task
The PLC-B programmer chose to totally re-code the logic from the ground up, correctly isolating Standard and Safety functions to their respective tasks
However, similar mapping had to occur in reverse, such that the 4 un-upgraded control racks still controlling safety devices with standard I/O could have safety logic control standard I/O

13. Design Lessons – Software Coding Methods
Lesson: Either way, a hybrid system will never fully meet the intended use of safety/standard tasks; so neither implementation is correct or incorrect.
Which way it should be done depends on how much time you have during a particular upgrade
PLC-A worked exactly as it previously did with little debugging; but will be harder to make compliant with the manufacturer's “correct” implementation of the Safety PLC when all the racks have been upgraded
PLC-B required a lot of debugging and tweaking in the field during this upgrade since the code was all-new, but will be easier to maintain and integrate with additional future rack upgrades
Additionally, rewriting one program at a time minimized risk of introducing systemic errors to both programs.

14. Installation Issues – Operability of Other Systems
One of the refurbished cabinets contained a component that was part of a separate Oxygen Deficiency Hazard monitoring/alarm system for cryogenic equipment in the Linac tunnel
This system is a required Credited Control when the Linac tunnel is occupied by workers
The refurbished cabinets each had their own electrical feed, so all power would need to be under lockout/tagout to safely work in these racks
Upgrade work was done during a maintenance outage, locking workers out of the Linac tunnel for several weeks was not possible if the ODH system was down

15. Installation Issues – Operability of Other Systems
A small control box was constructed to house this equipment, and was swapped over to during a shorter outage
This temporary box needed safety-class redundant backup power such that it was equivalent to what was removed, making it rather expensive and complex for something that was only needed for a few weeks
Lesson: Systems that operate independently should have their own independent enclosures and power systems, such that one may be removed from service without affecting the other

16. Installation Issues – Field Cable Lengths
When the upgraded racks were designed they were laid out to separate standard and safety wiring/power, in a different fashion than the existing racks
The I/O was moved one cabinet closer to the field, intended to mitigate any cable length problems pulling the existing cables to the new I/O
This wasn’t enough, and many of the “PLC-A” cables were too short to reach their intended termination point
Additional rows of intermediate terminal blocks had to be added to extend these connections to their proper locations
Lesson: During construction, leave adequate service loops for 1-2 major rack changes during the life of a facility. When doing an upgrade, take the existing layout and cable length into account when designing a new rack.

17. Operational Issues – Default Configurations
Since the GuardLogix hardware requires creating a new software development project for the logic, there are many underlying settings that will be reset to defaults
One of these values is a “watchdog timer” that detects when the program gets stuck in a loop or otherwise hangs up or crashes
If the program doesn’t finish its scan in the specified time, the controller faults and shuts everything down
The default is set to a conservative 20ms, but it needs to be tweaked depending on how much logic needs to execute, how long it takes to communicate with the I/O, etc
Shortly after commissioning, one of the systems faulted due to a watchdog over-run, since this value was not changed from the default to something better suited for the communications speed and amount of logic being evaluated
Lesson: Ensure that you are familiar with all of the underlying configuration settings that need to be updated before releasing a system to operations.

18. Final Questions or Comments?