1. bellomo@slac.stanford.eduOctober 31, 2006 1 LCLS Maintenance Personnel and Procedures Paul Bellomo

2. bellomo@slac.stanford.eduOctober 31, 2006 2 Topics Scope and Perspective Why we know we are ready Department descriptions ARTEMIS The maintenance process Examples

3. bellomo@slac.stanford.eduOctober 31, 2006 3 Scope and Perspective Accelerator equipment New electrical, mechanical, electro-mechanical and electronic equipment from the injector gun through the end of the tune-up dump TD-11. New control room in Building 5 New substations K10B, S520, MCC S20, panel-boards Operations Directorate 12 Departments, only 5 directly involved Controls, CEF, Klystron/Microwave, Mechanical Fabrication, Power Conversion Perspective Maintenance departments

4. bellomo@slac.stanford.eduOctober 31, 2006 4 Why We Know We Are Ready All departments LCLS equipment possesses some new features, but is not totally unlike the equipment that SLAC currently services and maintains Designers are the persons/groups that will maintain the equipment Maintenance personnel are qualified and have been involved in the LCLS design from the start. Training requirements are modest and provided by LCLS, Department, or outside agency Staff quantity is adequate to handle increased LCLS workload. The documentation (detailed later) needed to support the systems exist Procedures (detailed later) needed to maintain the systems exist

5. bellomo@slac.stanford.eduOctober 31, 2006 5 Department Descriptions Department Head – Hamid Shoaee, staff of 75 Software and hardware sections Software Section Write low-level applications, EPICS drivers, displays Maintain MATLAB infrastructure (licenses), ensure compatibility with all control room platforms Provide network hardware and software Provide control room infrastructure (alarms, displays, servers and workstations) Development and maintenance are the same people

6. bellomo@slac.stanford.eduOctober 31, 2006 6 Department Description - Controls (Continued) Hardware Section Beam Containment System (BCS) Prevents radiation from 'escaping' the shielding enclosure BSOICs, BTMs, LIONS, toroids and current monitors Machine Protection System (MPS) Protect equipment from beam damage VME processor-based protection ion chambers (PICs), RTDs, digital monitors Personnel Protection System (PPS) Protect personnel from radiation exposure, laser and electrical hazards PLC driven systems Vacuum - PLC based gages, valves, pumps and interlocks Movers - Tables, mirrors, cameras, wire scanners

7. bellomo@slac.stanford.eduOctober 31, 2006 7 Department Head – John Weisend, staff > 150 CEF Operations - Bernie Romero, staff of 28 Electrical maintenance of substations and electrical distribution equipment Beam-line manifolds and plumbing Documentation One-line diagrams Panel-board schedules Manufacturer drawings and instruction manuals Beam line assembly and layout drawings

8. bellomo@slac.stanford.eduOctober 31, 2006 8 Klystron/Microwave Department Department Head - Chris Pearson, staff of 80 Microwave Engineering and Maintenance/AMRF, staff of 15 Positron Source, Damping Rings, LINAC, LCLS LINAC and SLC RF systems and 2856MHz pulsed klystrons PEP (HER and LER) and SPEAR 3 RF systems and 1.2 MW, 476MHz CW klystrons Documentation Rack front views and Captar rack profile drawings Electrical interconnect diagrams, cable database Electronic chassis diagrams Manufacturer drawings and instruction manuals ELPs, EWPs, bench test, hi-pot procedures

9. bellomo@slac.stanford.eduOctober 31, 2006 9 Mechanical Fabrication Department (MFD) Department Head – Karen Fant, staff of 80 Mechanical fabrication/maintenance – no design Vacuum gages and pumps on the vacuum side, pneumatic valves, beam pipe and other components designed by others (LCLS staff) Magnet cooling water hoses, disassemble magnets to find magnet problems and ground faults On-call support of Operations Documentation Beam line assembly and layout drawings Vacuum pump-down procedures

10. bellomo@slac.stanford.eduOctober 31, 2006 10 Department Head – Paul Bellomo, staff of 75 DC magnet power systems and RF modulators Cables, cable tray and raceways Dedicated maintenance staff of 31 people, 24/7, 3-shift operation Participate in power system commissioning Documentation Rack front views and Captar rack profile drawings Electrical interconnect diagrams, cable database Electronic chassis diagrams Manufacturer drawings and instruction manuals ELPs, EWPs, bench test, hi-pot procedures

11. bellomo@slac.stanford.eduOctober 31, 2006 11 ARTEMIS Accelerator REMEDY Trouble Entry & Maintenance Information System Web-based problem reporting, work scheduling and work status program Report and track both hardware and software problems Incorporates ISMS core functions Five states New Entry – report problems or schedule maintenance In Progress – Jobs in progress or have more work to do Scheduled Jobs – Input from the New Entry state Review To Close – All known work completed, ready for management review Closed – trouble report entered into archive

12. bellomo@slac.stanford.eduOctober 31, 2006 12 ARTEMIS

13. bellomo@slac.stanford.eduOctober 31, 2006 13 The Process

14. bellomo@slac.stanford.eduOctober 31, 2006 14 Example S20 Power Distribution Layout

15. bellomo@slac.stanford.eduOctober 31, 2006 15 Example One-Line Diagram Excerpt from ID-950-075-03 Substation K10B 13.8kV:480V to S20 MCC to 6 Power Supply Racks Portion of S20 MCC shown

16. bellomo@slac.stanford.eduOctober 31, 2006 16 Example Rack/Equipment Profile

17. bellomo@slac.stanford.eduOctober 31, 2006 17 Example Electrical Interconnect (EI/ELP) Diagram

18. bellomo@slac.stanford.eduOctober 31, 2006 18 The PCD Process – Example 1 LCLS bend magnet system fails unexpectedly at 3AM. The indications are that a power supply has failed. LCLS Operations files an ARTEMIS and also notifies the PCD owl shift supervisor/maintenance manager (MM). Ops classifies the problem as critical, because the machine is down without this particular magnet. MM evaluates the system, reviews past ARTEMIS reports and chooses appropriate technicians to troubleshoot the problem. The MM, the technicians then assess the situation, from the technical, logistics and safety standpoints. The system is 480VAC input, 40VDC output. Appropriate test equipment and PPE (Category 0 with Class 0 gloves) The group notifies Ops that it is responding to the call. Ops turns the system over to PCD. PCD applies LOTO. ARTEMIS upgraded to “in process”

19. bellomo@slac.stanford.eduOctober 31, 2006 19 The PCD Process – Example 1 Continued Work progresses power system will not turn on. After several hours, the solution is not readily apparent. Ops escalates the problem to the designated system expert or Department Head. After 3 hours without resolution the Department Head and LCLS manager are notified. Work continues The technical expert finds and solves a short in the power supply rectifier. Parts are field-replaced. Ops is notified that the system is running. The system is returned to Ops. Technicians file “review to close” ARTEMIS report, file e-log detailing the problem, the solution and parts used. The PCD MM checks the parts inventory and if below a pre-determined critical level, orders replacement parts. PCD closes the job internally

20. bellomo@slac.stanford.eduOctober 31, 2006 20 The PCD/MFD Process – Example 2 A ground fault is discovered in a magnet system. Ops files ARTEMIS and notifies PCD. The fault keeps the system off. It must be repaired immediately PCD locks out the system. They troubleshoot the system and determine that the ground fault is in the magnet. PCD asks Ops to notify MFD. The ARTEMIS is annotated and responsibility re-assigned. PCD replaces personal LOTO with group LOTO. The PCD MM is now responsible for the system. MFD is called. MFD responds and confers with PCD. PCD demonstrates that the power supply cannot be turned on. MFD apply personal locks over the PCD group locks. MFD apply other LOTO, verify magnet ground fault existence and perform work necessary to remove the ground fault.

21. bellomo@slac.stanford.eduOctober 31, 2006 21 The PCD/MFD Process - Example 2 Continued The ground fault is found and repaired. The magnet is re-assembled and hi- potted. MFD personnel remove their locks. PCD is re-called. PCD replaces Group with personal locks. PCD hi-pots the system, remove their locks, turn the system on and release it to Ops. PCD technicians file a “review to close” the ARTEMIS and file e-log entries PCD MM conducts safety de-briefing and closes the job

22. bellomo@slac.stanford.eduOctober 31, 2006 22 Last Slide The SLAC maintenance groups are ready and eager to safely and professionally support the LCLS project