Reliability and Performance of the SNS Machine Protection System Doug Curry 2013

2Managed by UT-Battelle for the U.S. Department of Energy Presentation_name Overview Significant Reliability Issues at SNS – Report a fault in the absence of a fault False Trips – Report a real fault with some added delay > requirements Excessive system delays – Completely fail to deliver an input fault status Reporting false positive MPS status Minor Reliability Setbacks at SNS Methods for Improving System Performance Addressing Obsolescence – 2 nd Generation System Highlights

3Managed by UT-Battelle for the U.S. Department of Energy Presentation_name False Trip Reports “Noise” Trips – Beam image currents – Power supply switching noise – Other miscellaneous sources RF, transformers, long parallel cable runs, etc… MPS Signal MPS Return I   I

4Managed by UT-Battelle for the U.S. Department of Energy Presentation_name False Trip Reports Excessive Common Mode currents on MPS inputs Poor input isolation at the MPS interface Current source from device is below the required 10 – 20 mA Mitigation techniques – Redesigned interface module with integrated ferrites that cover broad frequency range – Replaced Power Supply opto-couples with lower dynamic range but higher CMRR parts

5Managed by UT-Battelle for the U.S. Department of Energy Presentation_name Excessive System Delays RF in Sub-link node Main-link node Beam Position Monitor Sub- link Main- link Copper link Fiber links Top

6Managed by UT-Battelle for the U.S. Department of Energy Presentation_name Excessive Delays How did we get here? – Schedule pressure to meet DOE milestones – Did not follow a formal configuration change method – No verification checks were performed Impacts – Lack of confidence in system – Damaged Cryo-cavities How do we prevent the similar mistakes from occurring again? – Utilize a formal design change process Document changes (DCN), require Configuration Control Committee approval prior to implementation, perform verification checks

7Managed by UT-Battelle for the U.S. Department of Energy Presentation_name Reporting false MPS status Output enable stuck high – Provides a false MPS status to the top level node and permits continued beam production Firmware implementation did not match the architecture approved by the review committee – Single clock source generated at the lowest node in the “chain” – Each node simply passes or prevents the passing of the permissive signal – Passed local clock source instead of original clock source Introduced 2 failure points

8Managed by UT-Battelle for the U.S. Department of Energy Presentation_name Minor Reliability Setbacks at SNS Low current drivers – Increased complexity Non-standard configuration High density connectors – Bent pins “stuck high” status shorted to power pin PLC interfaces Operating System errors – Software interlocks Hardware Assembly inconsistencies Vs Source Current Load  Vs   ??

9Managed by UT-Battelle for the U.S. Department of Energy Presentation_name Methods for Improving System Performance 100% link checks performed on every maintenance day Power measurement checks performed every 3 months – Plot the output power of fiber transmitters over time to identify degrading components 100% system verification checks performed after every extended outage Automated standardized testing procedures for improved consistency Equipment status tags – Red, yellow and green tags

10Managed by UT-Battelle for the U.S. Department of Energy Presentation_name SNS 2 nd Generation System Improve reliability by decreasing complexity – Reduce existing hardware components by 80% – Reduce existing interconnects by 82% Existing ConfigurationProposed Configuration Total HW Components32460 Total Interconnects840154

11Managed by UT-Battelle for the U.S. Department of Energy Presentation_name SNS 2 nd Generation System

12Managed by UT-Battelle for the U.S. Department of Energy Presentation_name SNS 2 nd Generation System Integrate Timing System Interface to provide improved post mortem analysis events – Currently provide global MPS status only Reports Auto-Reset or Latch type fault System integrity checks prior to the production of each beam pulse – Verify one channel per machine cycle during our idle period – Full MPS system verification checks competed every 30 seconds for 1800 input channels Standardized modules – Hot swappable for increased interface flexibility and maintainability

13Managed by UT-Battelle for the U.S. Department of Energy Presentation_name Conclusion Overcoming reliability problems SNS – Formalize and verify all design changes – Perform regularly schedule PM tasks – Standardized testing procedures Decreased complexity with increase functionality and reliability – > 80 reduction in hardware components and interconnects – Perform system diagnostic checks during beam idle periods