ARIES WP 6 / Task 1.3 - September 2017

ARIES WP 6 / Task 1.3 - September 2017 TE/EPC Methods Maintenance Service Power converter maintenance at CERN Ch. Mugnier https://edms6.cern.ch/document/1846156/1 EDMS: 1846156 ARIES WP 6 / Task 1.3 - September 2017

ARIES WP 6 / Task 1.3 - September 2017 Summary Electrical Power Converter Group – Maintenance organization Data collections Management of maintenance activities: processes Report, feedback and analysis Conclusions Ch. Mugnier - EDMS: 1846156 Maintenance*: “All actions to maintain or restore an item in a specified state or able to provide a specific service” (AFNOR NFX 60-010) ARIES WP 6 / Task 1.3 - September 2017

Electrical Power Converters Group maintenance organization Ch. Mugnier - EDMS: 1846156 ARIES WP 6 / Task 1.3 - September 2017

ARIES WP 6 / Task 1.3 - September 2017 In the CERN operational context More than 4500 power converters: large spectrum of technologies few kW to hundreds of MW 700 different types of PC Large geographical distribution of equipment, Limited and difficult access to equipment, On-call 24/24 service for 1st line intervention is requested, Maintenance cycle based on Injectors & LHC schedules. TS Commissioning YETS Operation Feb. Mi-Mars December Ch. Mugnier - EDMS: 1846156 ARIES WP 6 / Task 1.3 - September 2017 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Group structure Power electronics TE/EPC Group Control systems 8 Sections - ~ 70 staff In charge of the design, development, procurement, construction, installation, operation and maintenance of electrical power systems for all accelerators, transfer lines, experimental areas and tests facilities at CERN Control systems Analog and digital electronics Power electronics High-precision current and voltage measurements Maintenance management DC AC Iref Reg. I Ch. Mugnier - EDMS: 1846156 Magnet ARIES WP 6 / Task 1.3 - September 2017 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Power Converter design Maintenance Management Support Group structure TE/EPC Group 8 Sections - ~ 70 staff In charge of the design, development, procurement, construction, installation, operation and maintenance of electrical power systems for all accelerators, transfer lines, experimental areas and tests facilities at CERN Measurement design Section HPM High Precision Measurement - 5 staff Software development Section CCS Converter Control Software - 5 staff Power Converter design (few kVA to hundreds MVA) Sections: FPC: Fast Pulsed PC - 8 staff LPC: Low PC - 10 staff MPC: Medium PC - 10 staff HPC: High PC - 10 staff Electronic Section CCE Converter Control Electronics - 5 staff Maintenance Management Support Section OMS Operation Maintenance Support - 15 staff Control systems Analog and digital electronics Power electronics High-precision current and voltage measurements Maintenance management DC AC Iref Reg. I Ch. Mugnier - EDMS: 1846156 Magnet ARIES WP 6 / Task 1.3 - September 2017 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Organization for maintenance activities PH Section CCE HPC FPC MPC LPC HPM CCS Each section is responsible for the maintenance and the repair of its equipment. Each designer is also: 1st line interventions Standby service Section OMS responsible for the maintenance of some equipment, involved in the Standby service. Maintenance information and documentation management, Asset management, Maintenances processes, standards and rules definition, Data management tools development The methods service, part of OMS Section, provides a support for the management of maintenance activities: is also responsible for the: Standby Service of TE-EPC, Electronic Repair LABoratory, in charge of the repair of faulty electronic cards and crates. Ch. Mugnier - EDMS: 1846156 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Standby service Three 1st line intervention teams to face to the large geographical dispersion and to the large spectrum of technology of equipment, LHC and LHC exp. 1st line interventions (2x8 staff) EA’s + SPS 1st line interventions (FSU TE-11: 12 techn. + 3 second line engineers) Complex PS Injectors 1st line interventions (2x8 staff) Two additional teams on standby to support the 1st line teams Control support (8 staff and fellow from CCS and CCE sections) Major event coordination (Group and Section Leaders) Ch. Mugnier - EDMS: 1846156 Experts on call Organization Data collection Processes Feedback, KPI’s Demo & Questions

Corrective maintenance Preventive maintenance Type of maintenance performed Corrective maintenance Preventive maintenance Corrective Maintenance (Planned) Emergency Maintenance (Unplanned) 1st line Interventions Intervention during TS Maintenance activities PS and SPS Complexes LHC Accelerators in operation PS and SPS Complexes LHC Calendar based Maintenance Condition Based Maintenance Shutdowns (YETS) LHC PS and SPS Complexes Design Improvement Redundancy Equipment replacement Punctual actions after specific tests: visual inspections, infra-red measurements, oil of transformers…… Ch. Mugnier - EDMS: 1846156 Organization Data collection Processes Feedback, KPI’s Demo & Questions

How do we manage the transition between project phase and operation for new equipment? Data collection Ch. Mugnier - EDMS: 1846156 Organization Data collection Processes Feedback, KPI’s Demo & Questions

The transition phase During the transition phase, the methods service in collaboration with designers: Constitute the operational documentation Identify parts, organize the spare parts, Collect data on the environment, Define the training for maintenance teams Propagate data into the relevant databases… Transition phase   Ch. Mugnier - EDMS: 1846156 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Data structures Tree structure created in MTF&EDMS follows standard rules Ch. Mugnier - EDMS: 1846156 Documentation of each component is stored in its own Item in EDMS Organization Data collection Processes Feedback, KPI’s Demo & Questions

Coding Quality Assurance: Naming and Layout RPHGB.RR13.RD2.L1 HCRPHGB___-KP000012 Following the Quality Assurance Plans (QAP) of: LHC SPS PS Complex Working in close collaboration with the Naming Service: https://cern.ch/service-acc-naming The GCO* is responsible for the overall coding Ch. Mugnier - EDMS: 1846156 GCO*: Group Coding Officer Organization Data collection Processes Feedback, KPI’s Demo & Questions

Parts inventory Parts are: registered in the data-base labelled with an equipment code following the LHC convention and validated by “Naming-Service”, Pictures of each part are systematically added to the DB, Recent cards contain a serial number chip (Dallas) for remote readout Ch. Mugnier - EDMS: 1846156 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Operational spare parts storage e-Catalogue Local user interface One central spare part storage and 80 satellite storages containing 12’000 references and 110’000 components Ch. Mugnier - EDMS: 1846156 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Information and data sharing TE/EPC database: centralizes all data relating to the management of maintenance activities; is connected to other CERN databases; constitutes a CMMS with specific users interfaces; provides a support to maintenance teams and all those involved in the operation MTF LAYOUT DB AIS APPL. INFOR EAM EDMS GIS External users Ch. Mugnier - EDMS: 1846156 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Who does what, when and how? Maintenance* task Who does what, when and how? Processes Ch. Mugnier - EDMS: 1846156 Maintenance*: “All actions to maintain or restore an item in a specified state or able to provide a specific service” (AFNOR NFX 60-010) 18 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Equipment groups or Contractors Information sharing tools Flow diagrams Processes of the Group are defined with flow diagrams CCC Improvement loop Equipment groups or Contractors Interface management Information sharing tools Roles Standards Tasks Ch. Mugnier - EDMS: 1846156 19 19 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Example : 1st line intervention An equipment is faulty, CCC calls TE-EPC Piquet Beam is recovered New parts Piquet intervenes to fix the problem Operation test of equipment with CCC Consolidations Corrective actions Maintenance activities Improvement Logbook, recommendations KPI’s statistics Feedback Means documentation, procedures, spare parts, instrumentation, tools Databases Operation data Equipment characteristics Deming Wheel The methods service is in charge of steering and controlling the process All group members are involved Ch. Mugnier - EDMS: 1846156 20 Organization Data collection Processes Feedback, KPI’s Demo & Questions

TE/EPC roles Methods service Equipment responsible GIO: Group Information Officer GSPO: Group Spare Parts Officer GCO: Group Coding Officer Database administrator Operation coordinator Standby service: 1st line intervention teams Control support Major event coordinator Electro-mechanical support Equipment responsible Technical responsible Machine coordinator Ch. Mugnier - EDMS: 1846156 21 Organization Data collection Processes Feedback, KPI’s Demo & Questions

CMMS System TE/EPC Group’s CMMS TE-EPC Databases: INFOR EAM External users AIS APPL. EDMS LAYOUT DB MTF Repair Management …. Equipment Management CMMS System Document Management Maintenance plans Intervention Management Activity Planning/Tracking Equipment traceability Human ressources Stock Management Budget management tracking expenses Supply Management - Procurement KPI dashboard and statistics TE-EPC Databases: supports the teams that are involved in the operation of the converters, is a CMMS* composed of Oracle database and asp.net (users interfaces) is open to other CERN systems, many exchanges are already in place (EN / EL, layout and naming DB, GIS, MTF ...) Ch. Mugnier - EDMS: 1846156 (*) Computerized maintenance management system 22 Organization Data collection Processes Feedback, KPI’s Demo & Questions

TE/EPC Databases Organization Data collection Processes Ch. Mugnier - EDMS: 1846156 https://te-dep-epc-databases.web.cern.ch/ 23 Organization Data collection Processes Feedback, KPI’s Demo & Questions

TE/EPC Databases: E-catalog Ch. Mugnier - EDMS: 1846156 24 Organization Data collection Processes Feedback, KPI’s Demo & Questions

TE/EPC Databases: Reports Tracking of interventions Tracking of exchanged parts Tracking of repaired parts Equipment Functional position Fault reported - Causes Machine – Model - Location Date - Time (Intervention Duration) Repair Tracking and Fault Identification Parts Changed Equipment history, intervention reports and failure forecasts are used to plan maintenance activities in order to increase the availability of the machine. Ch. Mugnier - EDMS: 1846156 25 Organization Data collection Processes Feedback, KPI’s Demo & Questions

TE/EPC Databases: Statistics During working hours 1st line interventions by week in 2012 Outside working hours During working hours Outside working hours 189 176 -7% Comparison of year 2012 versus 2011 Ch. Mugnier - EDMS: 1846156 26 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Report, feedback and analysis Ch. Mugnier - EDMS: 1846156 27 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Analysis of interventions records From model down to individual component Top Ten in terms of intervention count: Top Ten of exchanged parts Per Model Per Equipment (Slot)) Part changed during repair in laboratory RPJAP: Auxiliary Power Supply Ch. Mugnier - EDMS: 1846156 28 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Equipment Failure mode Failure analysis : to identify failure modes What are the different failure modes detected during repair process? For each failure mode, what are the stress factors? Are the faults depending of the location, current …? Ch. Mugnier - EDMS: 1846156 29 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Machine Availability Indicator Two Key Performance Indicators used to monitor power converter performance regarding machine availability: Uptime proportional to reliability Downtime proportional to maintainability LHC Machine MTBF ‘Time between Failures’ [Days] 2010 2011 2012 Global 1.3 1.4 1.8 w/o external faults 1.7 2.8 3.4 LHC Machine MTTR ‘Average Intervention Time’ [Min.] 2010 2011 2012 Global 128 94 Ch. Mugnier - EDMS: 1846156 30 Organization Data collection Processes Feedback, KPI’s Demo & Questions

To be developped Equipment Reliability: Weibull Analysis Weibull analysis models the failure rate and gives risk prediction using ‘age to failure’ data. Weibull distribution gives 2 main parameters: η = characteristic life or scale parameter β = slope or shape parameter More information in Yves Thurel talk Weibull Analysis based on failure data useful for Preventive Maintenance Strategy β = 1, failure rate is constant, Failures are random Preventive maintenance of the equipment will have no effect on the equipment's failure occurrences β > 1, failure rate increases Preventive maintenance not waiting the failure has a benefit regarding the failure rate of the equipment To be developped Early Life Failures β < 1 Wear-Out Failures β > 1 Random Failures Ch. Mugnier - EDMS: 1846156 Weibull need good quality of data 31 Organization Data collection Processes Feedback, KPI’s Demo & Questions

Demo and questions https://te-dep-epc-databases.web.cern.ch/default.aspx Ch. Mugnier - EDMS: 1846156 32 Organization Data collection Processes Feedback, KPI’s Demo & Questions