1. ARIES WP 6 / Task 1.3 - September 2017

2. ARIES WP 6 / Task 1.3 - September 2017
TE/EPC
Methods Maintenance
Service
Power converter maintenance at CERN Ch. Mugnier
EDMS:
ARIES WP 6 / Task September 2017

3. ARIES WP 6 / Task 1.3 - September 2017
Summary
Electrical Power Converter Group – Maintenance organization
Data collections
Management of maintenance activities: processes
Report, feedback and analysis
Conclusions
Ch. Mugnier - EDMS:
Maintenance*: “All actions to maintain or restore an item in a specified state or able to provide a specific service” (AFNOR NFX )
ARIES WP 6 / Task September 2017

4. Electrical Power Converters Group maintenance organization
Ch. Mugnier - EDMS:
ARIES WP 6 / Task September 2017

5. ARIES WP 6 / Task 1.3 - September 2017
In the CERN operational context
More than 4500 power converters:
large spectrum of technologies
few kW to hundreds of MW
700 different types of PC
Large geographical distribution of equipment,
Limited and difficult access to equipment,
On-call 24/24 service for 1st line intervention is requested,
Maintenance cycle based on Injectors & LHC schedules. TS
Commissioning
YETS
Operation
Feb.
Mi-Mars
December
Ch. Mugnier - EDMS:
ARIES WP 6 / Task September 2017
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

6. Group structure Power electronics TE/EPC Group Control systems
8 Sections - ~ 70 staff
In charge of the design, development, procurement, construction, installation, operation and maintenance of electrical power systems for all accelerators, transfer lines, experimental areas and tests facilities at CERN
Control systems
Analog and digital electronics
Power electronics
High-precision current and voltage measurements
Maintenance management DC AC
Iref
Reg. I
Ch. Mugnier - EDMS:
Magnet
ARIES WP 6 / Task September 2017
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

7. Power Converter design Maintenance Management Support
Group structure
TE/EPC Group
8 Sections - ~ 70 staff
In charge of the design, development, procurement, construction, installation, operation and maintenance of electrical power systems for all accelerators, transfer lines, experimental areas and tests facilities at CERN
Measurement
design
Section HPM
High Precision Measurement
- 5 staff
Software development
Section CCS
Converter Control Software - 5 staff
Power Converter design
(few kVA to hundreds MVA)
Sections:
FPC: Fast Pulsed PC - 8 staff
LPC: Low PC staff
MPC: Medium PC staff
HPC: High PC staff
Electronic
Section CCE
Converter Control Electronics - 5 staff
Maintenance Management Support
Section OMS
Operation Maintenance Support - 15 staff
Control systems
Analog and digital electronics
Power electronics
High-precision current and voltage measurements
Maintenance management DC AC
Iref
Reg. I
Ch. Mugnier - EDMS:
Magnet
ARIES WP 6 / Task September 2017
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

8. Organization for maintenance activitiesPH
Section
CCE
HPC
FPC
MPC
LPC
HPM
CCS
Each section is responsible for the maintenance and the repair of its equipment.
Each designer is also:
1st line
interventions
Standby service
Section
OMS
responsible for the maintenance of some equipment,
involved in the Standby service.
Maintenance information and documentation management,
Asset management,
Maintenances processes, standards and rules definition,
Data management tools development
The methods service, part of OMS Section,
provides a support for the management of maintenance activities:
is also responsible for the:
Standby Service of TE-EPC,
Electronic Repair LABoratory, in charge of the repair of faulty electronic cards and crates.
Ch. Mugnier - EDMS:
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

9. Standby service
Three 1st line intervention teams to face to the large geographical dispersion and to the large spectrum of technology of equipment,
LHC and LHC exp. 1st line interventions
(2x8 staff)
EA’s + SPS 1st line interventions
(FSU TE-11: 12 techn. + 3 second line engineers)
Complex PS Injectors 1st line interventions
(2x8 staff)
Two additional teams on standby to support the 1st line teams
Control support
(8 staff and fellow from CCS and CCE sections)
Major event coordination
(Group and Section Leaders)
Ch. Mugnier - EDMS:
Experts on call
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

10. Corrective maintenance Preventive maintenance
Type of maintenance performed
Corrective maintenance
Preventive maintenance
Corrective Maintenance
(Planned)
Emergency Maintenance
(Unplanned)
1st line Interventions
Intervention during TS
Maintenance activities
PS and SPS Complexes
LHC
Accelerators in operation
PS and SPS
Complexes
LHC
Calendar based
Maintenance
Condition Based Maintenance
Shutdowns (YETS)
LHC
PS and SPS
Complexes
Design Improvement
Redundancy
Equipment replacement
Punctual actions after specific tests: visual inspections, infra-red measurements, oil of transformers……
Ch. Mugnier - EDMS:
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

11. How do we manage the transition between project phase and operation for new equipment?
Data collection
Ch. Mugnier - EDMS:
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

12. The transition phase
During the transition phase, the methods service in collaboration with designers:
Constitute the operational documentation
Identify parts, organize the spare parts,
Collect data on the environment,
Define the training for maintenance teams
Propagate data into the relevant databases…
Transition phase
Ch. Mugnier - EDMS:
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

13. Data structures
Tree structure created in MTF&EDMS follows standard rules
Ch. Mugnier - EDMS:
Documentation of each component is stored in its own Item in EDMS
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

14. Coding Quality Assurance: Naming and Layout
RPHGB.RR13.RD2.L1
HCRPHGB___-KP000012
Following the Quality Assurance Plans (QAP) of:
LHC
SPS
PS Complex
Working in close collaboration with the Naming Service:
The GCO* is responsible for the overall coding
Ch. Mugnier - EDMS:
GCO*: Group Coding Officer
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

15. Parts inventory Parts are:
registered in the data-base
labelled with an equipment code following the LHC convention and validated by “Naming-Service”,
Pictures of each part are systematically added to the DB,
Recent cards contain a serial number chip (Dallas) for remote readout
Ch. Mugnier - EDMS:
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

16. Operational spare parts storage
e-Catalogue
Local user interface
One central spare part storage and 80 satellite storages
containing 12’000 references and 110’000 components
Ch. Mugnier - EDMS:
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

17. Information and data sharing
TE/EPC database:
centralizes all data relating to the management of maintenance activities;
is connected to other CERN databases;
constitutes a CMMS with specific users interfaces;
provides a support to maintenance teams and all those involved in the operation
MTF
LAYOUT DB
AIS APPL.
INFOR EAM
EDMS
GIS
External users
Ch. Mugnier - EDMS:
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

18. Who does what, when and how?
Maintenance* task
Who does what, when and how?
Processes
Ch. Mugnier - EDMS:
Maintenance*: “All actions to maintain or restore an item in a specified state or able to provide a specific service” (AFNOR NFX ) 18
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

19. Equipment groups or Contractors Information sharing tools
Flow diagrams
Processes of the Group are defined with flow diagrams
CCC
Improvement loop
Equipment groups or Contractors
Interface management
Information sharing tools
Roles
Standards
Tasks
Ch. Mugnier - EDMS: 19 19
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

20. Example : 1st line intervention
An equipment is faulty,
CCC calls TE-EPC Piquet
Beam is recovered
New parts
Piquet intervenes to fix the problem
Operation test of equipment with CCC
Consolidations
Corrective actions
Maintenance activities
Improvement
Logbook,
recommendations
KPI’s
statistics
Feedback
Means
documentation, procedures,
spare parts,
instrumentation, tools
Databases
Operation data
Equipment characteristics
Deming Wheel
The methods service is in charge of steering and controlling the process
All group members are involved
Ch. Mugnier - EDMS: 20
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

21. TE/EPC roles Methods service Equipment responsible
GIO: Group Information Officer
GSPO: Group Spare Parts Officer
GCO: Group Coding Officer
Database administrator
Operation coordinator
Standby service:
1st line intervention teams
Control support
Major event coordinator
Electro-mechanical support
Equipment responsible
Technical responsible
Machine coordinator
Ch. Mugnier - EDMS: 21
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

22. CMMS System TE/EPC Group’s CMMS TE-EPC Databases:
INFOR EAM
External users
AIS APPL.
EDMS
LAYOUT DB
MTF
Repair Management ….
Equipment Management
CMMS
System
Document Management
Maintenance plans
Intervention Management
Activity Planning/Tracking
Equipment traceability
Human ressources
Stock Management
Budget management tracking expenses
Supply Management - Procurement
KPI dashboard and statistics
TE-EPC Databases:
supports the teams that are involved in the operation of the converters,
is a CMMS* composed of Oracle database and asp.net (users interfaces)
is open to other CERN systems, many exchanges are already in place (EN / EL, layout and naming DB, GIS, MTF ...)
Ch. Mugnier - EDMS:
(*) Computerized maintenance management system 22
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

23. TE/EPC Databases Organization Data collection Processes
Ch. Mugnier - EDMS: 23
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

24. TE/EPC Databases: E-catalog
Ch. Mugnier - EDMS: 24
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

25. TE/EPC Databases: Reports
Tracking of interventions
Tracking of exchanged parts
Tracking of repaired parts
Equipment Functional position
Fault reported - Causes
Machine – Model - Location
Date - Time (Intervention Duration)
Repair Tracking and Fault Identification
Parts Changed
Equipment history, intervention reports and failure forecasts are used to plan maintenance activities in order to increase the availability of the machine.
Ch. Mugnier - EDMS: 25
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

26. TE/EPC Databases: Statistics
During working hours
1st line interventions by week in 2012
Outside working hours
During working hours
Outside working hours
189
176
-7%
Comparison of year 2012 versus 2011
Ch. Mugnier - EDMS: 26
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

27. Report, feedback and analysis
Ch. Mugnier - EDMS: 27
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

28. Analysis of interventions records
From model down to individual component
Top Ten in terms of intervention count:
Top Ten of exchanged parts
Per Model
Per Equipment (Slot))
Part changed during repair in laboratory
RPJAP: Auxiliary Power Supply
Ch. Mugnier - EDMS: 28
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

29. Equipment Failure mode
Failure analysis : to identify failure modes
What are the different failure modes detected during repair process?
For each failure mode, what are the stress factors? Are the faults depending of the location, current …?
Ch. Mugnier - EDMS: 29
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

30. Machine Availability Indicator
Two Key Performance Indicators used to monitor power converter performance regarding machine availability:
Uptime proportional to reliability
Downtime proportional to maintainability
LHC Machine
MTBF ‘Time between Failures’ [Days]
2010
2011
2012
Global
1.3
1.4
1.8
w/o external faults
1.7
2.8
3.4
LHC Machine
MTTR ‘Average Intervention Time’ [Min.]
2010
2011
2012
Global
128 94
Ch. Mugnier - EDMS: 30
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

31. To be developped Equipment Reliability: Weibull Analysis
Weibull analysis models the failure rate and gives risk prediction using ‘age to failure’ data.
Weibull distribution gives 2 main parameters:
η = characteristic life or scale parameter
β = slope or shape parameter
More information in Yves Thurel talk
Weibull Analysis based on failure data useful for Preventive Maintenance Strategy
β = 1, failure rate is constant, Failures are random
Preventive maintenance of the equipment will have no effect on the equipment's failure occurrences
β > 1, failure rate increases
Preventive maintenance not waiting the failure has a benefit regarding the failure rate of the equipment
To be developped
Early Life Failures β < 1
Wear-Out Failures β > 1
Random Failures
Ch. Mugnier - EDMS:
Weibull need good quality of data 31
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions

32. Demo and questions
Ch. Mugnier - EDMS: 32
Organization
Data collection
Processes
Feedback, KPI’s
Demo & Questions