Accelerator Reliability and Software A Controls Engineer’s perspective

INTRODUCTION A bit about me Controls system overview Controls faults statistics What we did well What we might to better Acknowledgments Questions

MACHINE CONTROLS SYSTEM OVERVIEW Uses EPICS (3.14.8.2 though to 3.14.12-2) 175 IOCs 174 Linux (CentOS, Debian, Libera), 1 Windows 12 OPI boxes (8 recent replacements) Delphi, EDM, MEDM, Alarm Handler, MatLab 8 PLCs 1 Archiver (27000+ PVs) 1 File server 28 MOXA terminal servers 4 Gateways boxes – 27 EPICS gateways Network separation – separate sub nets for accelerator, accelerator EPS, each beamline, build network, integration lab, office, wireless, etc.

CONTROLS FAULTS YEARLY ANALYSIS 2007 2008 2009 2010 2011 2012 Number of Faults 17 7 3 8 5 Total Downtime (h:mm) 14:34 15:24 6:47 4:18 9:05 6:46 MTBF (h:mm) 157:40 278:00 699:42 1619:45 612:15 987:00 MDT (h:mm) 0:51 0:54 0:58 1:26 1:08 1:21 % of Total Downtime (10.3%) 9.50% 13.00% 7.41% 5.79% 11.37% 14.68%

PERCENTAGE DOWNTIME – CONTROLS FAULTS Controls Fault Break Down

GOOD STUFF 1 Selection of EPICS as control system (2004) Good community support (teck-talk) Collaboration meeting twice a year. Access control lists (Top-up related IOCs only so far). A good team for initial development Good interface with accelerator/operations team for on going maintenance and development. Good response times for issues New functionality integrated in timely manor Consistent OPI interface helps here.

USED SYSTEMS ENGINEERING PROCESS 1 Captured functional requirements for the Storage Ring Injection System, RF delivered by 3rd party vendors. Creates a System Breakdown Structure (SBS) Produced and reviewed design documentation for major sub-systems e.g. Vacuum, Magnets, Front End Control, Diagnostics etc. Produced system block diagrams Produced and reviewed interface specifications. Produced system test requirement specifications

USED SYSTEMS ENGINEERING PROCESS 2

USED SYSTEMS ENGINEERING PROCESS 3

CONFIG MANAGED BUILD SYSTEM Automated Uniform build and deploy process. Quarantined build boxes (avoids side effects). All source code extracted from source code repository (perforce) Only source code changes linked to bug tracking system (bugzilla) available for build. Completed build committed to build repository But not locked down so as prevent patching and/or development builds. Trust system.

ACCELERAOR OPERATOR INTERFACE Reliable User Interface Operators like it Attributes: Simple, intuitive, comprehensive, consistent. Not so good: Essentially only I have the in depth of knowledge. Also not embraced by the EPICS community. Use of confirmation dialogs on some actions – but we don’t over do it. Cosmetically pleasing – not bland Make training easier Provides a PV value configuration management capability E.g. magnet power supply and timing system settings.

OPI EXAMPLE 1

OPI EXAMPLE 2

NOT SO GOOD STUFF / RISKS 1 Turnkey systems. Quick to get going but… Lack hands-on experience with some items (e.g. PLCs) Extra work to maintain and modify. Needs some reverse engineering. This impact time to fix. Often we mitigate a problem rather than doing a proper fix (band-aid solutions) Still work, but not always ideal. Always in catch up mode for h/w and s/w to provided advanced functionality.

NOT SO GOOD STUFF / RISKS 2 Accelerator file server failure Highlighted a hole is the IT backup process – need to be vigilant. Some IOC now 7+ years old (e.g. 30 uIOCs) Relatively small maintenance team Utter dependence on the network. Recent network upgrade caused havoc, not all implications considered (duplicate routing tables). Last year, a security audit involving full port scan swamped the 10MHz link to EPS which then didn’t know state of shutters which caused a beam dump.

ACKNOWLEDGEMENTS Joel Trewhella (statistics and associated graphics) Don McGilvery.