1. Summary of Session 6 ATC/ABOC days
Steve Myers

2. ATC/ABOC Days AB Groups
“MTTR & Spare Parts Policy for the LHC injectors, experimental areas and other facilities: AB Groups”
Richard Scrivens (ABP) session 5
Roberto Losito (ATB)
Jocelyn Tan (BI)
Jan Borburgh (BT)
Marc Vanden Eynden (CO)
Christophe Mugnier (PO)
Carlo Rossi (RF) 2

3. ABP Proton/Ion Sources
Proton Source
Source reliability >99%.
Possibility of 12hr Time to Repair if fault occurs in vacuum (less than once per year).
Most spare parts exist, but no database.
Fault rates kept low by continuous study of operational performance, facilitated by source access possibility during operation. (Will not be the case for LINAC4)
Ion Source
Only one hardware failure on source in 2007.
But, source requires a lot of tuning during operation (>1 time per day). Remote control needs upgrading to allow tuning from CCC.
Oven filling also required for LHC (2 fills per LHC ion run).
>250kchf invested in spare parts in Missing spare parts can be financed from operation budget in 2008/9 (except spare klystron).

4. ABP Linac 2 / 3 drift tubes REX TRAP+EBIS
Failure of a magnet in the DTL of Linac 2 is a serious concern (but has not happened in 30 years).
Tooling is missing for opening and replacing drift tube. Could lead to a very long stop of the entire proton programme of CERN
 Study required to estimate length of stop and how to reduce it.
Linac 4 estimates a few weeks to change a failed permanent magnet in the DTL.
REX TRAP+EBIS
Faults can cause entire runs (~1 week long) to be cancelled, which has happened a few times in the last years.
Equipment still being transferred to AB standards. Then need to assess how to improve the reliability of this “transferred” system.

5. ATB

6. North Area Obstacles
With the exception of XTAX, Microcollimators and XTDV’s, spares are available and can be installed in 1 day. Extra time for cool-down required if in TCC2, affecting the whole North Area.

7. North Area Beam Obstacles
Mechanics of beam obstacles not in bad shape
The most critical items, XTAXs recently renovated
Spare elements exist or can be made in reasonable delays
Failures and interventions typically stop the corresponding beam line.
Can be absorbed with re-scheduling and good will from the experiments
Failures in elements located in the upstream part of the beam lines or in TCC2 imply switching off the whole North Area for the time of the intervention
Plus the cool-down time if in TCC2

8. ATB General Conclusions
Situation under control in SPS & transfer lines but:
TIDVG to be modified or re-designed
Situation not under control in PS complex:
PS Dumps: no more spare?
Sieve: spare recommended by APC, under discussion for consolidation program.
Safety elements: consolidation budget will (?may) be made available in 2008.
Situation not under control in Experimental Areas:
Controls obsolete, only 1 person knows how to keep them alive
Budget to renew hardware too high for consolidation (>1.5 MCHF, <5kCHF/axis) 8

9. BI Critical Elements Instr Tot Problem Beam abort Action BCT
53 + a few specials
Vacuum leakage at ceramics
YES
Replacement if spare available,
MTTR 2days
Watchdog : Electronics + cables
MTTR = 2 h.
Linacs + rings : good shape
Transfer lines : about to be consolidated towards LHC standards
Special BCTs No
No spare ceramics nor toroids.
BLM
400
Linked to machine protection
MTTR =2h. 5% in storage
Electronics
MTTR = 2 h. PM electronics for SPS needs repair. Consolidation waiting approval
Cables are replaced during shutdown
Phase PU 3
No acceleration
No spare monitor. Good shape
Electronics : MTTR =2h, spare OK
Intercepting devices
360
Vacuum leakage + mechanical in/out
Electronic MTTR = 2h
SEM grids : MTTR 1 week
Wire scanners : MTTR = 1 day
LEIR electron cooler 1
no phase space cooling
Gun, cathode, collector MTTR = 2 weeks
Magnetic elements : MTTR 2 months
Spare : OK

10. BI Unavailable Diagnostics
Instr
Tot
Problem
Beam abort
Action
Orbit + Trajectory
NO FAILURE OF SENSORS
430
SPS : interlock linked to orbit extraction bump
YES
Electronics is ageing, MTTR =2h.
Consolidation requested for MOPOS
Others : no orbit or trajectory No
90% spare monitors available
Electronics : MTTR = 2h
LEIR circular BPM
MTTR = 1day + baking
Electronics MTTR = 2h
Ionisation profile monitor 2
SPS : no profile
If failure, then replacement during shutdown
LEIR : no profile
Spare monitor OK. MTTR = 1 week due to baking
Tune 3
No tune
All BBQ + FESA based.
Electronics : MTTR is short due to standardization
WCM
No tomoscope
No spare monitor.
PS monitor suffer from radiation damage. Consolidation foreseen this year.

11. BI in AD (problem area) Deceleration Electron cooling
Longitudinal Schottky monitors : 1 low freq unit + 1 high freq unit, no spares
if sensor USY4105 is damaged, AD STOPS for one week
Electronics : Head amplifier : spare OK, MTTR = 8h
Electron cooling
gun, collector and cathode : one spare of each, AD STOPS for 2 weeks
magnetic elements : unique spare for the drift solenoid.
other magnetic elements : no spare. if damage = AD STOPS UNTIL REPAIR
Intensity measurements
Longitudinal Schottky monitors : TFA7049, no spare
Tune measurements
BTF with Transverse Schottky monitors : no spare
Electronics : Head amplifier : no spare
Orphan system ?????

12. BI Conclusions Low probability of sensor failure
Sensors fail less often than electronics
Real concerns about the lack of spare ceramic chambers for BCTs
slow BCTs : Lin2, Lin3, LEIR
DCCTs : SPS, LEIR
Fast BCTs : F16, TT2, TT10, Booster to PS. In short : any transfer lines
AD injection
Wall Current Monitor : PSB
Strong consolidation effort still needed particularly in AD
BI group provides CCC a list of specialists on call.

13. AB/BT equipment in PS complex BT equipment in the PS Complex
ISOLDE
East Hall AD
PSB PS cv
CTF3
Injection systems
BI.DIS booster distributor
BI.SMV vertical deflection septa
BI.SMH horizontal injection septa
Injection kicker BI.KSW
Extraction systems
Extraction kickers BE.KFA
Extraction bumpers BE.BSW
Extraction septa BE.SMH
Recombination kickers BT.KFA
Recombination septa BT.SMV10
Recombination septa BT.SMV20
LINACs
magnetic Septa
electrostatic Septa
kicker systems
PS Complex
LEIR 13

14. SPS and LHC involvement
Injection system
Injection kickers MKP
Extraction systems
Extraction kickers for CNGS and LHC beam 2: MKE4
Extraction kickers for LHC beam 1: MKE6
Extraction septa North Area, ZS, MST, MSE; LSS2
Extraction septa for CNGS and LHC beam 2: MSE; LSS4
Extraction septa for LHC beam 1: MST, MSE; LSS6
Beam extraction protection elements
TPSN
TPSG4
TPSG6
Beam dumping system
Vertical Extraction kickers MKDV
Horizontal Extraction kickers MKDH
Tune kickers MKQ H/V
Kicker
Electrostatic septa
Magnetic septa
SPS

15. MTE will make PE.SEH31 obsolete
PS Electrostatic septa
SEH23, only partial spare
All PS septa used for LHC beams as well as the electrostatic septa benefit from a preventative exchange program to reduce the likelihood of a failure during the run, based on past experience.
MTE will make PE.SEH31 obsolete
Name
Septa
Recovery from major fault
Failure likelihood
Spares
PE.SEH23 1
Septum: week
Vacuum: days
Cable: hours
HV generation: hours
Medium
Low
Even years: 2nd choice spare
Odd years: full spare
1 spare set of cables available, but old
Spare generator online
PE.SEH31
Cable: hours/days
High
1 full spare septum+ 2nd choice spare septum
1 spare set of cable available, but with short life expectancy
Failure prone cable runs from HV resistor to feedthrough over beam. Short life time observed during 2007 high intensity runs.
23/1/2008 ATC/ABOC days 15

16. BT Conclusions
NO ‘shocking’ or ‘striking’ situations identified (no high risk without spares)
Septa HV CABLE situation (delicate components) LESS COMFORTABLE, but PE.SEH31 will be phased out after MTE comes online, and for PE.SEH23 and ZS the situation is under control. (see slide)
PS SEPTA situation starts to profit from consolidation, and PREVENTIVE EXCHANGE is undertaken for all PSB and PS septa part of the LHC injector chain from this SD onwards.
For PS KICKERS only PARTIAL SPARE PARTS inventory is defensible because of their modularity, some redundancy, combined with low risk and high capital investment to achieve full spares for (sometimes) unique equipment.
SPS KICKERS spare situation generally OK. Weak points will be addressed in the coming years.
MTTR strongly dependent on radiation levels (can partly be influenced by operation).
Balanced solution between expert lists and stand-by service .

17. SPS Beam Dumping kickers MKDH & MKDV
No spare MKDH magnet nor vacuum tank
(failure likelihood: medium, MTTR: weeks)
1 spare MKDV magnet and tank being refurbished. Pulse generator to be re-designed in future.
(failure likelihood: medium, MTTR: weeks 17

18. CO TODAY’S REALITY PS complex SPS In general for all injectors
CO group is still in charge of almost all front-end hardware and software systems
CO « piquet service » as first line support to OP
SPS
Lighter HW and SW involvement of CO compared to PS
CO list of experts, best effort
In general for all injectors
Hardware obsolescence is a concern and must be addressed
Poor layout and asset management compared to LHC
Important Initiatives will be taken by CO already in 2008 in order to deploy new FE solutions from 2009 onwards

19. CO Major Components Front-ends Specific Risks
Several components are reaching their end of life time
VME processors and crates (Wes) > 10 years old
Out of stock for some Obsolete HW modules (GFAs, MIL-1553, …)
CO has taken the following actions
Annual preventive maintenance of FE systems
Re-engineering of Obsolete core hardware modules such as GFAs, MIL-1553 (mid-2008)
Upgrade of some operational systems with new HW modules in order to re-establish spare parts for obsolete modules
Bulk orders for new VME crates (mid-2008)
Major on-going market surveys for
New generation of VME CPUs running Linux (mid-2008)
New generation of Industrial PCs (PICMG 1.3) as a more cost-effective platform (mid-2008) that will replace VME when appropriate

20. CO First Line Support (PS)
Organization
Team of 5 to 6 technicians
Each member on service during one week
Applies to ‘standard CO’ controls (hardware/software),
PS Complex (Lin2, PSB, CPS, Lin3, LEIR, Isolde, AD)
CTF3
Manages spare parts
Tracing: E-logbook, Follow-ups
For SPS and North area, OP SPS team uses expert call list

21. CO General Conclusions
Spare Policy
Spares available today for all CO hardware components, with sufficient margins
CERN wide spare policy for PLCs due for Q1 2008
Hardware Obsolescence
On-going program for the re-engineering of obsolete hardware modules
New contracts for AB (Q2 2008) for the medium term procurement of new Front-end platforms (VME, Industrial PCs)
Front-End (GM) and Application (X-Motif) Software
Will remain as is, with CO corrective actions when required, no new developments

22. Statistics 2007: AB-PO diagnostic tools
Diagnostic tools for the MTTR/MTBF monitoring have been put in place. For 2007, MTTR: ~1h, MTBF depending on the CERN complex area.
Weeks of exploitation
1740: Power converters
350: Interventions:
- 25,5% (07/06 ratio)
250 WH & 100 NWH
52.3 min: MT/Intervention:
305 hours / 350 Interv.
29: Weeks of exploitation
738: Power converters
251: Interventions:
+7 % (07/06 ratio)
151 WH &100 NWH
60 min: MT/Intervention:
251 hours / 251 Interv.
E-Logbook Interface
Ch. Mugnier, ATC-ABOC days, 23 January 2008

23. Conclusions
Long MTTR Risks: LHC Injectors
Vast consolidation plan has been launched to mitigate long MTTR’s, but until 2010, the PS rotating Machine represents the highest MTTR risk.

24. PO Conclusions
Diagnostic tools for the MTTR/MTBF monitoring have been put in place (MTTR: ~1h, MTBF depending on the CERN complex area)
Consolidation plan has been launched to mitigate long MTTR’s, but until 2010, the PS rotating Machine represents the highest MTTR risk.
According to the large number of interventions by year, Stand by service (“piquet”) was a choice. PIPO is organized by 3 teams, sharing the different areas of the CERN complex;
Spare parts are centralized and managed across the different machines and converter technologies.
Maintenance method service is responsible for the tool developments and operation organization.

25. RF Conclusions
The general strategy relies on the availability of hot spares, whenever possible, or at least on a reasonable stock of spare parts.
RF Vacuum Tubes
Long term contract with RF tube manufacturers, agreement on our policy for spare stock management.
Few critical areas:
SPS TWC200 Siemens amplifiers – the tube RS2004 could be soon out of production, spare tubes partly not new  diacrodes;
SPS TWC800 – minimum stock of old klystrons  IOTs;
Linac2 – tube TH170R limited production still possible, stock sufficient until 2015;
Rex – tube TH391 limited set of spares available.
Linac2 and Linac3
Critical areas:
Linac3 100 MHz amplifiers (2 x 350 kW) – no spares for the HV anode capacitor  production of few spares;
Linac3 ramping cavity 100 MHz amplifier – partial stock of spare parts (controls and electronics). 25

26. RF Conclusions PS Booster – HLRF (C02, C04, C16) Critical areas:
One spare HV anode power supply for 12 operational power supplies.
PS – HLRF (C10, C20, C40, C80, C02)
Critical areas:
Some components in the C10 system are old and becoming obsolete, including cavity servos  general consolidation in progress.
AD & AD Stochastic cooling
Critical areas:
Some spares available for the high level RF, but general weakness due to component aging.
LLRF Schottky pick-ups have no spare.
Critical areas for Stochastic cooling:
No drawings for the pick-up and kicker movement electronics and motors.
Control system still based on old, modified CAMAC controller. 26
Only one specialist available for maintaining the system !

27. RF Conclusions PS Booster – LLRF Critical areas:
No hot spares for the PSB damper pick-ups and kickers.
PS – LLRF
Critical areas:
40 / 550 modules have no working spare  repair/production of working spares in progress;
The one-turn delay feedback is not installed on cavity C10 11 (spare system)  degraded operation for AD beam;
Obsolete electronics in the Hereward damping system;
10 MHz matrix (special chassis, modules but no complete spare available);
The whole system availability relies on the operation of the room air conditioning system (system monitoring);
Extremely old electrical power distribution from the switchboard to the racks. 27

28. RF Conclusions SPS – HLRF (TWC200, TWC800) Critical areas:
Spare couplers (TWC200) and RF windows (TWC800) could not be tested because of lack of manpower;
2 out of 3 transformers broken during 2007 on TWC800  new transformers ordered.
SPS RF Controls
Critical areas:
The TWC200 cavity controller is equipped with Simatic PLC dating from 1982, with limited amount of spares;
The programming machine for that system is obsolete and only few remain (probably) at CERN.
No hot spares available for the transverse dampers.
As a general remark, the reduction of resources, in particular the reduced number of specialists available on the different systems, is the most important source of danger for the RF availability. 28

29. RF Conclusions STAND-BY POLICY
HLRF is providing a piquet service for SPS equipment and PS 200MHz system
LLRF is providing a piquet service for PSB, PS and SPS.
The HLRF intervention in the PS area (Linacs, LEIR, AD, Rex, PSB, PS) and the LLRF intervention in some of the machines of the PS area (LEIR, AD) is guaranteed by means of a list of specialists available on call.
In the PS area the HLRF piquet is not possible due to lack of manpower; however the on-call list has assured an effective intervention until now. The presence of hot spares helps.
Critical areas:
Small number of specialists available for many different systems in the PS area;
Limited standardization of modules and equipments (basically due to their different age and to limited renovation).
Variable MTTR depending on specialist availability when the “piquet” is not provided or cannot solve the problem. 29