LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: LHC MAINTENANCE POLICY AND REQUIREMENTS

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Contents Introduction – LHC & TQM Cost of down-time Availability indicators Designed for availability – the Power Converter approach Cryogenics – maintenance management now! Quench protection – Energy Extraction RF Technical infrastructure maintenance Common LHC Maintenance policy? Tools available to help you… Conclusion

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: LHC and TQM The reliability and availability of the LHC will depend not only on the quality of its components but also on the quality of the maintenance of these components… This maintenance quality involves everyone and all the activities related to LHC. Quality also means conformance to requirements (you should get what you ask for…). Quality can and must be managed. This is known in industry as Total Quality Management (TQM) and it implies: –A process for managing quality is defined; a philosophy of perpetual improvement of how we execute our activities.

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: The Principles of TQM Quality can and must be managed. Everyone in the LHC has a customer and is a supplier of something. Processes, not people constitute the problems to solve. Everybody is responsible for the quality achieved – not only the QAWG! Problems must ideally be prevented, not just fixed after they have happened. Quality can and must be measured. Quality improvements may be small but must be continuous. The quality standard is defect free. Goals are based on requirements, not negotiated. Life cycle costs should be taken into account, not only the initial costs… Management must be involved and lead the process. Plan and organize for a permanent quality improvement.

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Cost of down-time LHC CtC is ~ 3.5 BCHF LHC experiments ~ 1.5 BCHF Expected yearly operation duration ~ days Expected life-time ~ 20 years Average yearly maintenance cost of a technical installation is ~ 2% of capital cost (industry). Physicists of course, worry much more about integrated luminosity…

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Global availability indicator - MTTR The MTTR indicator can be split into nine parts: 1 – Production reactivity 2 – Maintenance reactivity 3 – Availability of man power 4 – Diagnostic Delay 5 – Preparation Delay 6 – Stock Logistics Delay 7 – Purchase Logistics Delay 8 – Work duration 9 – Administrative Delay

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Some examples of approaches taken Orbit corrector power converters (AB/PO) Cryogenics (AT/ACR) Quench protection and Energy extraction system (AT/MEL) Radiofrequency (AB/RF) Technical infrastructure (TS dept).

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Designed for availability 752 Power Converter units for orbit correction. Expected MTBF ~ hours AB /PO expects to lose on average 1 orbit corrector power converter unit every 4:th day (100+ hours)! Sufficient power in the remaining corrector power supplies to permit operation for days without stoppage. Implies that when the machine stops – everything has to be ready for rapid intervention including logistics in the tunnel to permit rapid exchange of any failing supply (25-65 kg). (MTTR factors) Faulty equipment is repaired on the surface (“off-line”).

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Cryogenics – Maintenance Management Now! Maintenance plans, tasks and spare parts policy are being defined taking into account manufacturers information, operational experience and detailed process knowledge. Plans and spares lists are based on industrial state-of-the- art approach and methodology are prepared using the CERN asset tracking and maintenance management system (CAMMS) Maintenance plan will be validated on the initial cryogenic installations going into operation for subsystem commissioning. Experience will be gathered, stock levels of spares adjusted, maintenance routines (preventive and corrective) optimised. Outsourcing of maintenance requires extensive preparations of contracts to provide clear objectives and to get reasonable performance from the contractor.

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Cryogenics – general approach 1. Corrective maintenance as needed during the operational periods 2. Preventive maintenance: - every winter shutdown on rotating machines, oil levels, filters, inspections, etc. - instrumentation and actuators calibration every two years - safety devices verification and validation every two years – tbc 3.Major overhaul of pumps (every 20'000 hours) and compressors (every 40'000 hours). Issues: 1.How to deal with maintenance of equipments still under warranty and/or in commissioning but already critical for subsystem commissioning? 2.Definition of component criticality with no operational experience and lack of support from manufacturers? 3.Missing resources to establish a complete maintenance plan, spare parts lists, component criticality, maintenance indicator to improve and optimize spare, preventive and corrective maintenance… MTTR3!

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Maintenance of the Quench Protection System The Quench Protection and Energy Extraction systems must be both reliable and available! This is reached by: –Systematic repair of faults –Analysis of data, acquired during quenches, and action –Diode voltages –Heater voltage recharging etc. –Beam Loss Pattern –Other information in Post Mortem –Analysis of data, taken during test periods, and action –Monthly ≥1 day maintenance with access No LHC run may start without a check of the QP and EE Systems

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Requirements for the repair/maintenance Material –Spares are available and can be repaired/ replaced Manpower issue (MTTR3) –At HERA there are 12 persons with experience available when the need arises (24h/24h, 7d/7d) –In AT/ MEL 4 persons are available today (expecting 2 more later…) –No detailed plans have been made today, MEL is waiting for experience from the HW commissioning before commitments are made.

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Radio Frequency System Detailed work on maintenance management issues by the time the RF hardware is being commissioned, i.e. towards end The RF group has a vast experience from running RF systems in other accelerators for guidance. Spares for critical equipment are acquired as required already now.

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Technical Infrastructure Maintenance The old ST division was an early adapter of CAMMS and much work has been done on: –Standardised terminology! –Naming conventions…. –The Top Ten of non-disposable equipment –Maintenance cost par equipment or family –Number of on-failure (corrective) maintenance per month –Costs of Spare Parts per month –Mean Time Between Failure (MTBF) –Mean Time To Repair per equipment (MTTR) Yearly maintenance cost ~ 1.9% of installed value for technical equipment.

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Outsourced maintenance Major issues seen: –Contract specifications – needs have to be defined very carefully. –Performance – you get (sometimes) what you pay for… –Planning of activities – has to concur with the accelerator schedule. –Tracking of interventions – work orders traced in CAMMS –Inventory management.

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Maintenance policy? Objective is clear – no LHC downtime except scheduled! Present status, location, stocks, maintenance procedures and interventions on all equipment installed must be managed - Create dashboards for all equipment families that clearly show both costs incurred and services effectively supplied. Identify the real processes along with potential bottlenecks, split them into sub-processes. Document the procedures and alleviate the bottlenecks where possible. Time measurement of the sub-processes must be done automatically using a computer application – an asset tracking and maintenance management system. Each intervention must originate an Work Order – no action without a trace! Analyse, evaluate and compare the performance indicators (MTTRj) – otherwise they are of no use.

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Tools available to help you… A computerized asset tracking and maintenance management system is used to do this – we have MP5 ( new release is called D7i). LHC equipment tracked by the MTF application will automatically be accessible from MP5/D7i. But maintenance procedures must still be created for each family of equipment. The technical infrastructure is already mostly managed with MP5/D7i as is parts of the cryogenics installations. Much experience exists and is available for you to profit from!

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Tools available to help you…dashboard with maintenance performance indicators….

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Conclusion Maintenance management of LHC equipment has already started! Keeping LHC available will be taxing resources in the various accelerator and technical departments. This has been expressed by ALL groups that have contributed! A strict policy of keeping track of all equipment in the complex, their maintenance procedures and the interventions on any equipment related to the availability of LHC is required. Use of a tool is mandatory but manpower resources must also be made available. Key indicators can be created to measure how well the quality aspects of the maintenance activities are managed, i.e. they will indicated how the LHC availability is being managed. LHC QAWG will be asked to prepare general rules and advise on implementation issues for each equipment group.

LHC Maintenance Management Policy Chamonix XIV January 2005 T.Pettersson - EDMS: Acknowledgements Thanks to F. Bordry, L. Serio, K-H Mess, T. Linnecar and I.Ruehl who shared their information about their aproaches to this subject.