1. prepared by R.A.Barlow (May 2014) TE-ABT-EC Maintenance methods and policy Edms nb. TE-ABT-EC Maintenance methods and policy in EC section

2. Outline  EC equipment location  Equipment failures  EC environment  Hardware  Software  Maintenance and procedures  Repairs  Documentation  Storage of components  Quality control  CONCLUSION TE-ABT-EC Maintenance methods and policy 2

3. EC equipment location (expert/piquet equipment) TE-ABT-EC Maintenance methods and policy 3 Equipment Responsible and Deputy

4. TE-ABT-EC Maintenance methods and policy 4 HV GENERATOR IGBT’s/GTO’s/Ignitrons, power triggers Thyratrons + Heaters CONTROLS PLC’s, Fast Controls, Electronic cards, Software, INTL ACTUATORS and SENSORS: Electrovalves, Sensors, Gauges, Temp measurements, etc... VACUUM Failure of vac gauges, Vac levels HYDRAULICS Water cooling systems, oil systems POWER SUPPLY Large power supplies, Small and Medium type,

5. TE-ABT-EC Maintenance methods and policy 5 BB ABT Equipment PS Complex, SPS & LHC Kickers & Septa TE/ABT/EC Electronics & Controls BE/CO & EN/ICE Control Infrastructure BE/OP Operation TE/ABT Specialist Electronic Modules Power electronics HV Generator Elements Controls PLC’s & Pxi systems Power distribution Software Actuators & Sensors Power supplies Control chassis Cabling & associated hardware

6. The EC environment TE-ABT-EC Maintenance, methods & policy Hardware modules Software Maintenance & procedures StorageDocumentationQuality control TE-ABT-EC Maintenance methods and policy 6

7. Electronic design data  Electronic design is done in-house with the schematics  PCB routing with exterior companies or TE-MPE-EM  Manufacturing produced outside or by TE-MPE-EM + assembly  AED/EDA numbering system for card versioning  If done by TE-MPE-EM then all documentation goes into EDMS under the EDA numbering scheme  If done by EC section then documentation is stored in EDMS and numbered with AED marking  Sometimes these are mirrored ! (WE wish to keep OUR ‘AED’ numbering system)  Each working manufacturing portfolio contains: Schematic, layout, gerber files, mechanical drawings, drillings file and serigraphy, BOM’s, test procedures, photo’s... 7 TE-ABT-EC Maintenance methods and policy

8. Equipment coding of modules 8 TE-ABT-EC Maintenance methods and policy Electronic cards labeling with bar code system MTF (Manufacturing and Test Folder) + AED/EDA number + Tested label The complete MTF system (judged not best suited for our equipment), with LIMITATIONS but...on-going

9. Programmable Logic Controllers  EC has a vast array of industrial automation in its controls park  PLC distributed systems are based around H/W design with associated software  Some chassis design requires custom made electronics coupled to PLC’s H/W  Each PLC chassis is created with technical drawing, cabling scheme, connector scheme (See_electrical) and software code (Step7)  In recent years industrial PXi systems (N.I) are becoming an additional resource to our supervisory control systems 9 TE-ABT-EC Maintenance methods and policy

10. Power elements/HV elements  Many semi-conductor switches type (IGBT, GTO, Behlke)  Trigger systems in the medium power range  HV discrete components  HV generators (EC/FPS/SE)  Electro Mechanical elements 10 TE-ABT-EC Maintenance methods and policy

11. Power distribution systems  EC responsible for the design of system power distribution  These are documented systems and undergo approval from HSE unit with outside companies such a DEKRA who come along for conformity testing and place approval seals on the systems. This includes Electrical circuit and Electrical protection checks.  We are currently looking at how to produce these elements on the outside using rigorous procedures and guidelines compatible with expected industry standards. 11 TE-ABT-EC Maintenance methods and policy

12. Software maintainability  PLC codes (Ladder, SCL, STL, Graph7) + WinCC (scada) Stored onto DFS volume, currently looking at other systems for software maintainability (Autosave product)  FPGA ‘s codes (mainly VHDL, sim files, chipscope, etc....)/SVN  C++, Java, other/SVN  Hardware drivers/SVN  PXI systems software/SVN  Fesa equipment classes/SVN 12 TE-ABT-EC Maintenance methods and policy

13. Preventive maintenance  Annual recalibration  Annual rotation of DCPS  Capacitor exchange (ex: Heater crates)  System consolidation  Timing system  Firmware upgrades  Interlock checks  Software release  Piquet weekly inspections of equipment where possible (ex: oil leaks, HV cable degradation, flasher not functioning, etc...) And many more! Preventive maintenance is difficult in areas with difficult access, no clear evaluation of possible cyclic maintenance, lack of data and analysis for overall systems. Also MTBF and MTTR are unknown quantities at the present (only LBDS LHC has undergone reliability studies) 13 TE-ABT-EC Maintenance methods and policy

14. Procedures  Magnet conditioning procedures (SPS MKE4 & 6, MKP, MKD and LHC MKI)  ‘Consignation’ procedures  Electronic module installation procedures  LHC LBDS generator testing + energy scan  Test procedures for electronics card  Procedures for loading BETS energy tables et al... 14 TE-ABT-EC Maintenance methods and policy

15. Procedures – movies ? 15 http://www.youtube.com/wat ch?v=AwugbEjSxAI The requirements are: ‘To create a multimedia support that shows and demonstrate some typical intervention carried during piquet time and outside of piquet time on TE/ABT equipment (Kickers & Septa) TE-ABT-EC Maintenance methods and policy

16. 16 Procedures – JIRA “ Plan, track, work – smarter and faster ” ABT-EC has started to use JIRA as an accomplished work and planning management tool for the following actions: TE-ABT-EC Maintenance methods and policy Creating and tracking FSU jobs Creating and tracking Hardware production jobs Notification of repairs Notification of troubleshooting Notification of general maintenance of equipment related issues Intervention tracking

17. Interventions/Repairs  Daytime requires -> Experts  Outside working hours -> Piquet/Experts These events can last 5mins (ex: Remote solving) up to extreme cases such as 48 hours or more (intervention in-situ involving longsome trouble shooting, special equipment handling, heavy maneuvering. Lots of specialists, etc.,,,) And occasionally a new spare part will be required !!! TE-ABT-EC Maintenance methods and policy 17

18. Anatomy of a typical intervention TE-ABT-EC Maintenance methods and policy 18 Log intervention CCC calls ‘online’ monitoring of faulty system Go to system location Solve problem Call CCC Intervene remotely Problem solved Identify Fault Stay around Leave premises Log intervention Controls, Hardware, Procedural ? Time Materials Pressure Human factor Unknowns Technical expertise sharing Etc....

19. Surface interventions vs tunnel interventions Surface equipment interventions  Items and spares are located in-situ (storage cupboards, racks, Faraday cages)  Missing items have to be fetched from 865 in EC section AND other storage areas Tunnel equipment interventions  Items are invariably located in 865 or other storage areas, no spares in UA’s for LHC kicker equipment (except certain power elements) TE-ABT-EC Maintenance methods and policy 19

20. Intervention examples  LHC LBDS fault: The ratio between DVs1 [v] and DVs2 [v] was out of specification. A possible GTO failure inside the GTO stack. Decision taken to mask the interlock due to built in redundancy. Clearly a procedural intervention requiring permission from various sources and ultimately the chief LHC coordinator on that day  SPS MKE4 heater problem: after analysis of a thyratron related problem a decision was taken to replace the suspected damaged heater module This required locating a spare module, and the removal of the faulty module for future repairs  SPS MKD thyratron switch replacement: heavy maneuvering, specialists (not an evident one man job). This operation is a complex one, requiring transport, crane maneuvering, specialists, potentially dangerous and costly, re-conditioning of system (procedure) and follow up of faulty thyratron switch ‘The technical diversity of the systems under EC responsibility presents a challenge and requires high levels of expertise from individuals. All members of the team cannot always intervene on equal competence levels’ TE-ABT-EC Maintenance methods and policy 20

21. Written follow up of intervention  Usually carried out in the logbook system, operation and EC logbook  Discussed at weekly ‘Piquet’ meetings  Actions follow-up are required TE-ABT-EC Maintenance methods and policy 21

22. Storage of spare equipment  PLC spare parts critical stock (en-ice-plc, b.864)  Electronic modules/cards (b.865)  Power supplies (b.865) + building  HV elements spares (b.865, b.867, in-situ)  Power distribution components (b.865)  Spares in-situ for surface controls (BA’s) (Electronic modules, power electronics, thyratron switches)  Spare part lists (Excel file) with components and spare location for each installation TE-ABT-EC Maintenance methods and policy 22

23. 23 TE-ABT-EC Maintenance methods and policy

24. Storage of spares at 865 TE-ABT-EC Maintenance methods and policy 24 865 2-A50

25. Documentation types  Electronic designs schematics, layouts, PCB masks, drilling drawings, mechanical features  Functional specifications and Engineering specifications, internal notes, papers  Minutes of meetings, summaries  Mechanical designs  System layouts, timing schemes, cabling schemes, procedures, stock lists  Many more.... ! 25 TE-ABT-EC Maintenance methods and policy

26. Documentation Storage TE-ABT-EC Maintenance methods and policy 26  Much of previous list still stored on DFS volume (should be avoided), some in file holders/no electronic versions.  Ongoing guidelines still to be finalised for the EDMS framework.

27. Pre-installation quality control  Electronic diagrams reviewed by third party  Surface and visual checks for H/W  Weekly rounds on surface points of equipment controls  Visual inspection of produced PCB’s  Electronic hardware test procedures TE-ABT-EC Maintenance methods and policy 27

28. Recapitulative summary HardwareSoftware Maintenance OK Procedures OK Storage OKGOOD Documentation GOOD Quality check OK Module tracking POORn.a Returned hardware for repairs OKn.a 28 TE-ABT-EC Maintenance methods and policy POOR, OK, GOOD, EXCELLENT

29. To take away today  Maintenance and methods is a vast complex subject !  There are no all ‘encompassing’ solution  This leaves room for bigger ideas and concepts  What is OK today will not suffice in the future  There are maintenance methods systems, they are long and tedious (massive legacy of data entry) to put in place and require resources 29 TE-ABT-EC Maintenance methods and policy