Operations WBS Dictionary Integrated Control Systems Henrik Carling ESS/ICS Date:

WBS Dictionary - Integrated Control Systems Operations phase Functional description – Competence mapping Possible organizational implementation – Infrastructure – Hardware and Integration – Management – Safety and protection – Software Organizational model – Cost – Staff/temporary – Recurring costs – Capital investments 2

Operations phase functional description The ICS division has the following tasks during operations – Normal operation tasks ensuring that ESS machine availability criteria are met Maintaining the Integrated control system Problem solving for control system issues Preventive maintenance Planned updates, enhancements and upgrades Tuning and refining machine protection systems Obsolescence and version management of running systems – For safety related systems Ensuring safe operation through the PSS Preventive maintenance Assisting calibration of critical monitors (ODH, radiation etc) Organizing and driving recurrent certifications/validations of the PSS Planning and assisting cyber security assessments – Extension and optimization tasks Extending the Integrated control and safety systems to accommodate instrument expansion Facilitating tuning and optimizing machine performance, availability and cost-effectiveness of operation Simplifying and improving operator interfaces to the machine Executing controlled enhancements to the integrated control system software 3

Functional -> Competence mapping The following competencies are needed to implement the above functions – Analog and digital electronics engineering – FPGA Firmware – Electronics manufacturing (PCB) and testing – Operating systems - kernel level, real-time systems – Software development (C/C++/Java, UI, Web) – Software development environments (continuous delivery platforms etc) – EPICs family of software (EPICS, CSS etc) – Databases, database administration, storage and throughput engineering – Software verification and validation – Accelerator/beam physics modelling – Industrial automation hardware and software, field-busses – Electrical engineering, electrical and mechanical installation – Computing and networking infrastructure, digital communication – Data network performance/security administration, cyber security – Safety engineering according to IEC61508 and other standards – Management and project management skills – Knowledge of the existing facility, laboratory facilities etc 4

Possible organizational implementation Management Control system Infrastructure Control system Software Hardware and integration Safety and protection 5

WBS Dictionary Management 6 What are the systems the group owns – Management and administration of the Integrated control system team – Technical responsibility and oversight of the Integrated control system – Project planning, management and execution of activities Description of the groups recurrent activities during operation – Ensuring competence and capacity availability to meet ESS requirements on availability and safety – Managing organization development, budget, administration and project activities Benchmark with a similar function at another facility – References: Diamond Light Source management functions: 7 FTE (total team size 72) SNS: difficult to compare: 3 FTE Justification of labor and operation costs – The Integrated control system is needed for ESS operation - management and administration needs to be taken care of Foreseen capital/investment costs – None Organization breakdown – FTE: 4 – Manager: 2 – Engineer: 1 – Assistant: 1

WBS Dictionary Control system infrastructure What are the systems the group owns – Control room – Control system network including timing distribution system – Control system data centre Description of the groups recurrent activities during operation – Maintenance and operation of control system network for 24/7 uptime – Optimization of network capacity, throughput and performance (yes, they are different parameters) – Responding to tickets in the control system infrastructure helpdesk backlog – Organizing and facilitating cyber security assessment/mitigation Benchmark with a similar function at another facility – References: Diamond Light Source: Computing systems team - 13 FTE SNS: difficult to compare. Estimated team size: 5 FTE Justification of labor and operation costs – The control system infrastructure is needed for any type of machine operation Foreseen capital/investment costs – Normal hardware obsolesence management computing/networking equipment: 650 k€/year Organization breakdown – FTE: 6 – Manager: 1 – Engineer: 2 – Technicican: 3 – Consultants: 0.25 (recurring cybersecurity assessments) 7

WBS Dictionary Hardware and integration 8 What are the systems the group owns – All Integrated control system hardware except saftety/protection and infrastructure hardware – The overall technical responsibility for the integrated control systems except for systems explicitly owned by other groups – All IOC:s (I/O Controllers) in the Integrated control system including EPICS integration of such IOC:s with their subsystems Description of the groups recurrent activities during operation – Maintenance and operation of the integrated control system for 24/7 uptime – Responding to tickets in the integrated control system helpdesk backlog – Extending, customizing and optimizing the integrated control system – Planning and execution of preventive maintenance of ICS hardware – Supporting ESS Users at the interface between the integrated control system and the instrument control system – Supporting operators with troubleshooting the machine Benchmark with a similar function at another facility – References: Diamond Light Source: Hardware support, Systems engineering, Electronic design, Beanline control/motion teams: 47 FTE SNS: difficult to compare. Estimated team size: FTE Justification of labor and operation costs – The control system is needed for any type of machine operation and needs to be properly maintained to reach availability goals Foreseen capital/investment costs – Hardware upgrade/update/preventive maintenance costs 980 k€/year Organization breakdown – FTE: 13 – Manager: 1 – Engineer: 9 – Technician: 3 – Consultants: 2

WBS Dictionary Safety and protection 9 What are the systems the group owns – The Personnel Safety System, including access control systems, radiation monitoring, ODH monitoring and safety interlock systems – The Machine Protection systems including various critical interlock systems and the target protection system Description of the groups recurrent activities during operation – Maintenance and operation of the personnel safety and machine protection systems for 24/7 uptime – Responding to tickets in the safety and protection helpdesk backlog – Extending, customizing and optimizing the safety and protection systems including continous refinement of MPS functionality – Follow enhancements/extensions to ESS to provide personnel safety and machine safety – Supporting ESS Users and operators the with safety and protection related topics – Organizing and executing yearly validation/certification of all personnel safety and machine protection systems Benchmark with a similar function at another facility – References: Diamond Light Source: Personnel safety system: 5 FTE not inluding technicians. MPS is distributed on other teams SNS: difficult to compare. Estimated team size: 9 FTE Justification of labor and operation costs – ESS is not allowed to run without a functioning and validated/certified personnel safety system – Availability goals will be very hard to reach without a well managed machine protection system Foreseen capital/investment costs – Preventive maintenance and upgrade costs (as required by SIL calculations): 660 k€/year Organization breakdown – FTE: 11 – Manager: 1 – Engineer: 7 – Technician: 3 – PhD students: 2 – Consultants: 1

WBS Dictionary Control system software 10 What are the systems the group owns – Control system software development environment – Control system software applications such as CCDB, Scripting environment, parts of the EPICS family (EPICS core, CSS) – Control systems physics modeling software Description of the groups recurrent activities during operation – Maintenance and operation of control system software applications and development environment for 24/7 uptime – Responding to tickets in the control system software helpdesk backlog – Extending, customizing and optimizing control system software, plannig and executing upgrades/updates – Ensuring coordination and keeping abreast with development in control systems software communities (EPICS, CSS etc) – Supporting operators with creation of opertor screens and managing the integrated control system Benchmark with a similar function at another facility – References: Diamond Light Source: Application + Toolkit development teams - 5 FTE SNS: difficult to compare. Estimated team size: 10 FTE Justification of labor and operation costs – The control system software is needed for any type of machine operation – It is also a flexible and easily extensible component/system to fit requirements and customization needs Foreseen capital/investment costs – Licensing costs < 100 k€ year Organization breakdown – FTE: 11 – Manager: 1 – Engineer: 10 – PhD Student: 1 – Consultants: 3

Cost modelling - ICS at operations Preliminary costs so far Work on investment costs 11