1. Accelerator Reliability and Software
A Controls Engineer’s perspective

2. INTRODUCTION A bit about me Controls system overview
Controls faults statistics
What we did well
What we might to better
Acknowledgments
Questions

3. MACHINE CONTROLS SYSTEM OVERVIEW
Uses EPICS ( though to )
175 IOCs
174 Linux (CentOS, Debian, Libera), 1 Windows
12 OPI boxes (8 recent replacements)
Delphi, EDM, MEDM, Alarm Handler, MatLab
8 PLCs
1 Archiver ( PVs)
1 File server
28 MOXA terminal servers
4 Gateways boxes – 27 EPICS gateways
Network separation – separate sub nets for accelerator, accelerator EPS, each beamline, build network, integration lab, office, wireless, etc.

4. CONTROLS FAULTS YEARLY ANALYSIS
2007
2008
2009
2010
2011
2012
Number of Faults 17 7 3 8 5
Total Downtime (h:mm)
14:34
15:24
6:47
4:18
9:05
6:46
MTBF (h:mm)
157:40
278:00
699:42
1619:45
612:15
987:00
MDT (h:mm)
0:51
0:54
0:58
1:26
1:08
1:21
% of Total Downtime
(10.3%)
9.50%
13.00%
7.41%
5.79%
11.37%
14.68%

5. PERCENTAGE DOWNTIME – CONTROLS FAULTS
Controls Fault Break Down

6. GOOD STUFF 1 Selection of EPICS as control system (2004)
Good community support (teck-talk)
Collaboration meeting twice a year.
Access control lists (Top-up related IOCs only so far).
A good team for initial development
Good interface with accelerator/operations team for on going maintenance and development.
Good response times for issues
New functionality integrated in timely manor
Consistent OPI interface helps here.

7. USED SYSTEMS ENGINEERING PROCESS 1
Captured functional requirements for the Storage Ring
Injection System, RF delivered by 3rd party vendors.
Creates a System Breakdown Structure (SBS)
Produced and reviewed design documentation for major sub-systems
e.g. Vacuum, Magnets, Front End Control, Diagnostics etc.
Produced system block diagrams
Produced and reviewed interface specifications.
Produced system test requirement specifications

8. USED SYSTEMS ENGINEERING PROCESS 2

9. USED SYSTEMS ENGINEERING PROCESS 3

10. CONFIG MANAGED BUILD SYSTEM
Automated
Uniform build and deploy process.
Quarantined build boxes (avoids side effects).
All source code extracted from source code repository (perforce)
Only source code changes linked to bug tracking system (bugzilla) available for build.
Completed build committed to build repository
But not locked down so as prevent patching and/or development builds.
Trust system.

11. ACCELERAOR OPERATOR INTERFACE
Reliable User Interface
Operators like it
Attributes: Simple, intuitive, comprehensive, consistent.
Not so good: Essentially only I have the in depth of knowledge.
Also not embraced by the EPICS community.
Use of confirmation dialogs on some actions – but we don’t over do it.
Cosmetically pleasing – not bland
Make training easier
Provides a PV value configuration management capability
E.g. magnet power supply and timing system settings.

12. OPI EXAMPLE 1

13. OPI EXAMPLE 2

14. NOT SO GOOD STUFF / RISKS 1
Turnkey systems. Quick to get going but…
Lack hands-on experience with some items (e.g. PLCs)
Extra work to maintain and modify.
Needs some reverse engineering.
This impact time to fix.
Often we mitigate a problem rather than doing a proper fix (band-aid solutions)
Still work, but not always ideal.
Always in catch up mode for h/w and s/w to provided advanced functionality.

15. NOT SO GOOD STUFF / RISKS 2
Accelerator file server failure
Highlighted a hole is the IT backup process – need to be vigilant.
Some IOC now 7+ years old (e.g. 30 uIOCs)
Relatively small maintenance team
Utter dependence on the network.
Recent network upgrade caused havoc, not all implications considered (duplicate routing tables).
Last year, a security audit involving full port scan swamped the 10MHz link to EPS which then didn’t know state of shutters which caused a beam dump.

16. ACKNOWLEDGEMENTS Joel Trewhella (statistics and associated graphics)
Don McGilvery.