Machine All Hands meeting #4 WoW-Ways of Working Peter Rådahl Carlo Bocchetta Henrik Carling 2019-05-15

Vision and Mission This presentation Introduction to WoW at ESS ESS Forums for a coordinated and agreed WoW across ESS Practical examples

ESS 12 Way of Working Areas WoW Area Owner/Directorate Managers Manage Requirements DG Head of Q Division Execute, govern & control Deputy Director of PS&A Develop Excellence/ESSMS Manage Safety & Licensing ESH&Q Head of ESH Division Manage Emergency & crisis Develop & maintain Facility MD Engineering Manager Operate Facility Deputy Director of MD Enable Science SD Deputy Science Director Manage Staffing & competence PS&A Head of HR Division Manage Supply Head of SPL Division Manage Documented info. Manage Configuration Owners and Managers to coordinate the work are appointed Three WoW areas are owned by the Technical Director Develop and Maintain Facility Operate Facility Manage configuration Management Core Support

Scope -Way of Working Develop & Maintain Facility, Operate Facility and Manage configuration Link to ESS Management System page Develop & Maintain Facility Way of working when developing (design, installation and test/commissioning) and changing the facility from requirements to acquired verification records of the changed facility. Assurance of configuration control (applying WoW Area Manage Configuration content), traceability requirements-design- V&V records and adequate requirements compliance. The analysis to identify and plan needs for maintenance, decision making of maintenance measures and the follow-up on implementation of the maintenance measures. Operate Facility Operating procedures for all systems of the neutron source and technical infrastructure Develop, maintain and implement training for shift leaders and operators. Verify operations procedures for control room staff together with the system owners Provide support during maintenance periods to the system groups and gain experience to ensure high availability Additionally, be involved in installation, testing and commissioning work of all equipment, to gain knowledge on systems Manage configuration Configuration control Identification Baselining Change Management

Forums for coordinating and agreeing on WoW Develop & Maintain Facility, Operate Facility and Manage configuration EEB Members Engineering Manager (Chair) Installation Manager Operations manager Deputy science director Engineering representatives John Weisend (AD) Rikard Linander (TD) Hector Novella (ICS) Lisa Arthursson (CF) Anton Lundmark (AD Infra) Ralf Trant (Head of ES&H) Mattias Skafar (HoD Quality) Johan Waldeck (SSM Licensing project manager) ESS Engineering Board (EEB) Builds a shared ESS way of working on Develop and maintain facility Manage configuration Manages Configuration and Changes to the facility Link to Engineering Management page OCT Members For operation Operations Manager (Chair) Operations section leader Deputy science director For baseline activities Installation Manager Engineering Manager AD/ICS/TD commissioning managers For systems and Services Head of ICS Associate Director for ESH&Q A representative for Systems (Target, Radio Frequency Systems, Vacuum, Cryogenic Plants, etc…) Additional representatives as needed Operation Coordination Team (OCT) Builds a shared ESS way of working on operations Assure operation of the neutron facility and technical infrastructure for science Link to Operations Management page

System Ownership A “system” can be a single object or a composite set of objects forming a system. Physical objects, software and services are “assets”. 1) The ”Owner" owns the requirements of an asset. 2) The ”Provider" has the responsibility to deliver the asset, meeting the requirements, to the Owner. 3) The ”Operator" has the responsibility to operate the system and to deliver a service to the Client. 4) The ”Client” has needs which are provided by the owner. The asset may only be owned by one of the four roles above at a specific time. An owner can also be a provider.

System Ownership Interactions Owner requests Provider to fulfil requirements. Provider supplies system meeting requirements and is defined as baseline Asset. Handover to Operator. Operations only controls operation, cannot change system requirements. On handover to Operations the Configuration &/or Requirements cannot be changed. Provider requests from Owner the ability to make changes. Workorders in EAM, changes to assets logged, configuration & maintenance is managed. Handback to Owner For fixes, maintenance and improvements/upgrades. Asset config & history is managed. Owner of System Requirements Operator Operational Asset/Service Requirements Provider(s) Asset/Service Maintenance Client Needs & Payment Service

System Ownership for ESS Core WoW Interactions Global requirements Owner of System Requirements WoW Develop & Maintain Provider(s) Operator WoW Manage Configuration Client WoW Operate Facility WoW Develop & Maintain WoW Enable Science

16 years ago…

Long shutdown - Many installations (new RF cavity, removal and upgrade of SC wiggler, complete ring re-alignment, relocation of disgnostics ...) 6400 Hours of Operation Unable to store beam - 8 to 12 machine turns Radiation losses distributed, consistent with mm thick metal obstruction Many days of trouble-shooting & unable to solve problem Mobilised entire division, people called back from vacation (one was in USA) A few hours before decision to cancel run and open machine... ... an operator spotted a light on a power supply controller that was normally a different colour.

Lessons Learnt Many stakeholders involved push needs for a common and proper WoW People performing the work today might not be here tomorrow The methods of “Develop and maintain” have a reason, the paper plant must reflect the reality Hand-over procedures including validation & verification and Ownership for all equipment Asset management to track installed components Baselining and Configuration management is key In this case, luck and dedicated operators saved the situation! Power supply to a corrector magnet, installed several years before, that was: Tested with local control Tested by physicists only Never used Forgotten, left ON, set to zero Not in control system! Errored, flipped a bit from 0 to –FFFF

Ownership and configuration management of the control system for the ion source and LEBT Background The delivery, installation and commissioning of the ion source and LEBT control systems is an example from which ICS has learned a lot It is a four-party project: Accelerator, ICS, CEA and INFN Situation Collaboration between in-kind partners was sometimes difficult Documentation was sometimes not delivered to the expected standards The testing conducted during the FAT could have been more comprehensive. The configuration used for testing during the Factory Acceptance Test (FAT) was not fully documented Verification of control systems before commissioning was cumbersome - too many unexpected problems Experiences Establish roles, responsibilities, interfaces and ownership very early Ensure close interaction with in-kind partner to ensure fully and appropriately documented configuration Work with in-kind contributors the reinforce verification and documentation of deliverables Comprehensive well-documented factory acceptance tests are important Avoid underestimating time and effort required for commissioning Do verification in controlled environments before commissioning