Review of Hardware Commissioning May 11-13, 2005 Hardware Commissioning Co-ordination Organisation Roberto Saban the logic day to day organization safety decision making failure scenarios/fall-back solutions my worries

Roberto Saban TS/HDO HC Review subsystems ready the logic all systems ready for machine check out individual system tests hardware commissioning Injection System RFCollimator System Superconducting Circuits Short Circuit Tests we assume everything is installed before the Hardware Commissioning starts two types of documents : as designed as tested impact on equipment safety impact on equipment/system performance impact on machine performance

Roberto Saban TS/HDO HC Review preparation the scope all the accelerator systems in the LHC tunnel with particular emphasis to superconducting circuits documentation the program the schedule the manpower HCC AT/MELAT/ACRAT/VACAB/ATBTS/CSETS/CVTS/ELTS/LEAAB/BTAB/PO...AB/CO individual system tests commissioning failure scenarios x 48

Roberto Saban TS/HDO HC Review preparation : software requirements prototypes procedurescommissioning scenario x 10 sacec : software applications for the commissioning of the electrical circuits AB/CO AT/MELAT/ACRHCCAB/COAB/PO  QPS  Power Converters  Powering Interlock

Roberto Saban TS/HDO HC Review subsystems ready the as designed documentation all systems ready for machine check out defines the pre-requisites to start the tests in terms of conditions and availability of systems describes in what conditions the system should be delivered to the next step specifies the objective of the test and either describes each step or summarizes the test procedure... although special meetings were devoted by the HC WG, many documents describing the individual system tests are missing

Roberto Saban TS/HDO HC Review preparation : essential time savers  user friendly application suite  tool to aid the interpretation of the test results  programmed procedures for battery tests  but also all the classical tools like the logging system, the alarm system, the post-mortem system, the electronic logbook, etc.  optimization of the MTF configuration for hardware commissioning  exchange of data between the central database and the specialist owned databases support from the Controls Group support from the MTF and Database Team

Roberto Saban TS/HDO HC Review day to day organization: the field minor control centres field control room arc and even matching section inner triplets and odd matching section warm magnets ~210cold circuits 190 orbit corrector circuits ~10 warm circuits ~210cold circuits 190 orbit corrector circuits ~10 warm circuits

Roberto Saban TS/HDO HC Review day to day organization: the field control room DFBs field control room UA 83 – Left of Point 8 Power Converters Energy Extraction Systems + wi-fi coverage of the UA + printer + social corner + water fountain + desks + meeting table + lockable closets cryogenicspowering testsinterlockspost-mortem

Roberto Saban TS/HDO HC Review day to day organization: the tunnel and the odd extremities + wi-fi coverage of the UJ and RR + desk + lockable closets 106 m mobile wi-fi coverage of the tunnel will it work with the magnet assemblies in place ?

Roberto Saban TS/HDO HC Review the sequence of tests around an even point Powering Tests

Roberto Saban TS/HDO HC Review organisation coordination presence in the field commissioning of sector A commissioning of sector B individual system tests and cool down of sector A’ individual system tests and cool down of sector B’ commissioning of warm components analysis follow up of data entry communication

Roberto Saban TS/HDO HC Review day to day... coordination : the human resources x 6 x 9 x 1 operation support to operation data recording reporting Field Control Room Sector A 1 Engineer 1 Technician 1 AB Operator Sector B 1 Engineer 1 Technician 5 people during two shifts Progress Tracking Planning Re-scheduling Meetings Crisis Management MTF Database Analysis Information processing Publications Analysis Project Management 1 Project Leader 1 Mr.Circuit & DPL 1 Expert Data Analysis 1 Database/Web Expert 1 Administrative Assistant 5 people during one shift hardware oriented removal of FCR small repair missing details access/balisage Field Preparation on request LHC Safety Coordination advise on safety matters enforce safety in the field

Roberto Saban TS/HDO HC Review Job Descriptions

Roberto Saban TS/HDO HC Review resource loading : Hardware Commissioning Coordination team 1 team 2 team 1 team 2 team 1 team 2 team 1 team 2 team [teams]

Roberto Saban TS/HDO HC Review personnel safety: assumptions 1.the safety systems (fire and smoke detectors, ODH system, evacuation sirens, emergency lighting, arrêts d’urgence, etc.) are operational 2.posts, signs, flashing lights are in place to announce the ongoing hardware commissioning activity in the tunnel sector or sub sector 3.the access system at the surface allows in only personnel involved in hardware commissioning 4.the panels at the entrance indicating tests in progress will not be installed on time 5.the tunnel and underground areas are ventilated in nominal conditions

Roberto Saban TS/HDO HC Review personnel safety: the field control room  the issue of the danger related to electricity was discussed on two occasions at the HC WG meetings, but failed to obtain clear directives on the training level of the people present in the FCR; many followed an ad hoc course with dissatisfaction  other issues related to helium venting in the tunnel, fire, number of people present undeground, etc. are known to worry some who are expected to spend a significant amount of their time in the UAs, UJ and RRs.

Roberto Saban TS/HDO HC Review safety: zonage & balisage        + riot barriers + red & white plastic chains HC Coordination delimits the areas where particular safety conditions must be enforced during the tests in conjunction with LHC Safety Coordination

Roberto Saban TS/HDO HC Review safety : personnel underground Front 1Front 2Front 1Front 2 LeftRight 6.6 km in the field on call 2t + 2o1e+1ocryogenics operation 1e+1tcryogenics instrumentation 1x+2e+2tquench protection 2x+2t1x+1t+...machine interlocks 1x+2t2e+1t+...power converters 2e+2tcoordination 1t1econtrols 1ooperation 4x+5e+10t+3o1x+4e+2t+1o ooperator xexpert eengineer ttechnician 22 present in the field Front 1 arc and the matching section on the even side Front 2 arc, the matching section on the odd side and the inner triplets on both sides two shifts = 15 hour days resources deployed for the powering tests

Roberto Saban TS/HDO HC Review safety : access during hardware commissioning 1/2 assumptions 1.the sectorization (into access sectors) of the machine is not possible 2.installation is ongoing in other sectors 3.the access system will not be operational states1.access is not allowed 2.access is regulated the activity is stopped during an access - if required... for the passage of a convoy, the equipment for the tests is folded so that it is not in the way regulated

Roberto Saban TS/HDO HC Review safety: access during hardware commissioning 2/2 activity access restrictions area concerned first cool down of a sector from room temperature to 80K no accesswhole sector electrical quality assurance (hi-pot) regulatedarc, subsector short circuits testsno accesssubsector (RA, RR, UJ) powering tests 1 no accesswhole sector instrument checksregulatedarc, subsector equipment checksregulatedarc, subsector validation of systemsregulatedarc, subsector ( 1 ) above minimum current as during normal operation of the collider

Roberto Saban TS/HDO HC Review decisions the commissioning must be carried-out in safety for the personnel and the hardware, be according to the approved procedures, be carried-out in the time expected, be traceable, remain within budget.  scheduling impact on commissioning time impact on other systems budget, resources  go/no-go rules personnel safety, procedures, traceability equipment integrity circuit integrity HCC HCC + HC Team owners HCC  operation & organisation impact on commissioning time  HCC

Roberto Saban TS/HDO HC Review communication  electronic media web pages progress reports (logbook, MTF, etc.) s for major milestones/events  regular meetings daily operations meeting in the FCR where every system is represented including the experiments via LEA weekly reports to GLMs, Team Meeting weekly meetings of HC monthly TCC  extraordinary meetings in case of crisis with concerned Groups, the Project and Department Leaders

Roberto Saban TS/HDO HC Review failure scenarios  following the suggestion of the MAC in December 2003 and the endorsement of MARIC, failure scenarios during the commissioning - both for Individual System Tests and Hardware Commissioning - were studied  the objective was to define strategies for at least a few case studies per major system in order to gain in reactivity at the time of failures  evaluate the impact on the program and  suggest alternatives to carry-on  six sessions of the HCWG were devoted to this study:  Superconducting Electrical Circuits  Cooling and Ventilation System  Cryogenic System  Vacuum System & Leak Tests  Electrical Power Distribution  Power Interlock Controllers can it be circumvented do it according to defined procedures clear diagnostic apply diagnostic protocols decision crash programme failure yes decision apply diagnostic protocols

Roberto Saban TS/HDO HC Review failure scenarios  Superconducting Electrical Circuits earth fault diagnostic short circuit configuration demineralised water circuit air cooling of the 600 A cables in the ducts loss of quench detection instrumentation bad splices start-up with missing circuits  Cooling and Ventilation System redundancy of components blowing of water cooled cables  Cryogenic System redundancy of cooling through the five feedpoints higher heat loads redundant instrumentation but no redundant cabling contamination of the helium inventory helium loss in the tunnel via safety valves impact of a utility (water, electricity) failure  Vacuum System & Leak Tests overhead for leaks detected after the pressure test  Electrical Power Distribution over-voltages (40 to 50% amplitude) for some tens of ms short spikes  Power Interlock Controllers electro-magnetic compatibility the download of a wrong configuration file to a controller or a software bug swapping cables failures at the level of SCADA failure scenario document in preparation * * * *

Roberto Saban TS/HDO HC Review impact of a fault requiring a repair  any failure requiring a repair of a cold part of the collider takes between 10 and 35 days [1]  qualification tests (e.g. elqa) will have to be repeated  commissioning program will have to be rerun: how much depends on where we were when the fault was encountered  during most of this time, the commissioning of the warm systems can continue 5 weeks warm-up repair cool-down qualify between 15 and 40 days out of ~72 [1] LHC-PM-ES-0002 General Parameters for Equipment installed in the LHC, April 1999

Roberto Saban TS/HDO HC Review consequence of a failure

Roberto Saban TS/HDO HC Review and then there is an exception !  a core team made of a few specialists/experts will meet on short notice to study the problem recommend additional diagnostics take a decision  go  no-go & crisis meeting with Project and Department Management  something the specialists or the hardware commissioning team do not understand.

Roberto Saban TS/HDO HC Review can it be circumvented/can it be accepted as is ?  what is the impact of a non-conformity on the performance of  the equipment ?  the system ?  the machine ? what is the budget ? have we reached the limit of the budget ? AB-ABP LHCOP Equipment Groups Hardware Commissioning Core Team

Roberto Saban TS/HDO HC Review my worries  coordination missing documentation transport & logistics access restrictions not respected safety of tunnel, of field control room, of the other areas  availability of components major components is not the problem of HC sub- components like instrumentation is  operation thefts noisy environment system testing infrastructure  hardware (e.g. field busses)  software (e.g. post-mortem system) applications utilities (e.g. e-logbook)  LSS8L tests not given the importance it deserves  accelerator component failure during the commissioning