Eddy Jans 0 Vertex2013 Operational aspects of the VELO cooling system of LHCb Eddy Jans (Nikhef) on behalf of the LHCb VELO group Introduction Main components and operation principle of the system Issues: how to prevent and to tackle them Keep the detectors cold 24/7 Summary & outlook 19 September 2013

Eddy Jans 1 Introduction VELO-module is double-sided (300  m, oxygenated, n + -on-n) and operated in vacuum, Strip closest to the beams is at 8.2 mm, Per double-sided module: 2x16 frontend chips that together dissipate ~20 W, 4 NTCs give temperature readings Two movable detector halves with 21 VELO + 2 PileUp modules each (400 W/side) Vertex2013 NTCs 19 September 2013

Vacuum tank One detector half Detectors are operated in a secondary vacuum  absolutely no leaks allowed. Orbital welding or vacuum brazing of the pipes. All tested at 170 bars. cooling block frontend chips module base is kept at +20 o C CO 2 connections Eddy Jans 2 Vertex September mm thick RF-box

in 1999 it was proposed to use CO 2 as refrigerant for the vertex detector of LHCb [ LHCb note ], CO 2 is radiation hard, CO 2 has a high latent heat value  can use small diameter capillaries  small amount of dead material in the acceptance, stainless steel capillaries:  inner =1 mm, wall thickness  0.25 mm system uses bi-phase CO 2 via the accumulator controlled method. Some cooling considerations cooling blocks Eddy Jans 3 Vertex September 2013

Pressure x Enthalpy diagram of CO 2 At -30 o C: 300 J/g mass flow: 10 g/s per module: 0.43 g/s full evaporation  130 W liquid CO 2 -speed: 28 cm/s all-gas speed: 240 cm/s bi-phase area pressure [MPa] enthalpy [kJ/kg] bi-phase area -30 o C gas phase liquid phase pressure [bar] critical point vapour quality 0 1 isothermal cooling Eddy Jans 4 Vertex September 2013

liquid vapor bi-phase Enthalpy Pressure P7P7 The accumulator controlled cooling cycle 1  2: pump increases the pressure of the sub-cooled liquid 2  3: heat exchange in the transfer line brings evaporator pre-expansion per definition right above saturation point, since (E 2 -E 3 ) = -(E 5 -E 6 ) 3  4: pressure drop in restriction and expansion in capillary brings CO 2 in cooling blocks in bi-phase state, 4  5: isothermal cooling via evaporation 5  6: warming up of incoming sub-cooled liquid 6  1: condensation and cooling of the CO 2 R507a chillers Condenser pump evaporator restriction accumulator Heat in Heat out P7P7 55 m transfer line insulation  =80 mm sub-cooled liquid in bi-phase return only passive components in the radiation zone Eddy Jans 5 Vertex September 2013

Main components of the system evaporative CO 2 cooling system “independent” system for either side PLC-controlled 2.5 kW water-cooled chiller at –40 o C 1 kW air-cooled backup chiller at -25 o C 55 m CO 2 transfer lines 10 heat exchangers 8 actuators 9 heaters 31 pressure sensors 192 temperature sensors 350 parameters monitored in PVSS only passive components at VELO 2*400 W heat load of detectors 2*12 kg CO 2 detector main chiller: -28 o C operational CO 2 temp. LV on: -7 o C backup chiller: LV -8 o C stability < 0.1 o C Eddy Jans 6 Vertex September 2013

Design considerations and operational experience redundancy of crucial components insulation clogging filters superheated CO 2 dependence on electrical power dependence on chilled water safety measures to prevent overheated detectors keeping the system 24/7 cold Eddy Jans 7 Vertex September 2013

Redundancy in the design To minimise down time the system has a few redundant crucial components: 3 CO 2 pumps, where 2 are needed, 2 chillers, water-cooled and an air-cooled as backup, for controls crucial temperature and pressure sensors are two-fold implemented, possibility to interconnect the two sides by hand, PLC is on a 1500 VA/1000 W UPS, PLC, backup chiller and CO 2 pumps are connected to a diesel generator. Eddy Jans 8 Vertex September 2013

Insulation Liquid pumped system  cold transfer lines  good thermal insulation required. This seems trivial, but turned out not to be so in practice. Originally CERN safety regulations forced us to use Armaflex NH. Glued surfaces started to delaminate after 2 years. Renewed insulation of the transfer lines and most of the cooling plant during Winter shutdown ‘10-’11. Now foamglass covered by an Aluminium protection shield and Armaflex AF, respectively. Eddy Jans 9 Vertex September 2013

Filters piece of Armaflex thermal insulation once completely blocked a restriction valve Throughout the system eleven 15  m filters are installed. (5 (CO 2 -plant), 2(manifold), 1(main chiller), 1(backup chiller)). In one detector half we have experienced a few times clogging filters. Replacement procedure is tricky and risk of additional dirt in the system due to difficult accessible filter houses. 2 months pressure [bar] Eddy Jans 10 Vertex September 2013

Post mortem analysis of the filters Many >15 µm orange objects have been observed inside the filter. They mainly contain Fe and O. Before, particles containing Cl had been observed. Possibly due to connections soldered with flux for a testbeam experiment  risk in terms of corrosion. Work extremely clean from construction to installation. Scanning Electron Microscope image Energy Dispersive Spectrometer analysis of an orange particle found inside the filter 75  m Eddy Jans 11 Vertex September 2013

Superheated CO 2 after startup After startup we occasionally observe in a varying number (a few  all) of cooling blocks the phenomenon of superheated CO 2. Issue: cooling performance is very bad because liquid cooling has much less cooling power than evaporative cooling. Eddy Jans 12 Vertex September 2013

Superheated CO 2 after warmup and cooldown  T=3 o C time  T silicon [ o C] LV off warmup of the cooling plant cooldown 30 minutes Eddy Jans 13 Vertex September 2013

Superheated CO 2 after warmup and cooldown  T=3 o C time  T silicon [ o C] start adding heat 30 minutes Remedy: add heat by means of a dedicated heater to bring the incoming CO 2 in the liquid+gas state. LV off liquid vapor bi-phase Enthalpy  Eddy Jans 14 Vertex September 2013 Pressure 

Not all cooling blocks behave the same way: - not all show superheating - when adding heat they don’t start boiling at the same moment Some more superheated CO 2 start adding heat silicon temperatures of 4 modules  T  4 o C Eddy Jans 15 Vertex September 2013

Power cuts PLC and backup chiller are connected to the power of a diesel generator of LHCb and the PLC also to its own UPS (1500 VA/1000 W) When the power gets cut the switch-over from main to backup chiller is handled automatically by the PLC. After switching back to the main chiller the system is stable after ~20 minutes. After switching on the LV the sensors are at their operational temperatures after 10 minutes.  half an hour recovery time sensor temperature 10 minutes LV on Eddy Jans 16 Vertex September 2013

Failure of chilled water supply Chilled water supply, that cools the main chiller, sometimes gets interrupted. If so, the PLC switches on the air-cooled backup chiller. However, this causes the LV to be switched off also. Eddy Jans 17 Vertex September 2013

Safety 1. HW-based: interlock system Beam Conditions Monitor Beam Conditions Monitor Cooling-PLC Vacuum-PLC Module temperatures Temp-boards Conditions LV Off HV Off Retract VELO Cooling Off Actions 2. SW-based: warning and interlock system 132 cooling parameters monitored 3 levels each: warning, error and fatal Emergency button Combined information of 4 NTCs per module are input to the FPGA, which can interlock the LV. Operation in vacuum requires immediate reacting safety systems. Three levels. Eddy Jans 18 Vertex September 2013

3. Human-based Emergency button in the LHCb control room to power off the VELO. Eddy Jans 19 Vertex September 2013

Keep the detectors cold 24/7 At the tip the received fluence is 2 x n eq /cm 2 and type inversion has taken place, so the sensors should always be kept cold, (below -8 o C), in order to prevent the V depletion to increase due to reverse annealing. The beneficial annealing budget amounts to a handful of weeks at room temperature. We try to save it till we really need it. But the conditions of the LongShutdown1 period at LHCb make it hard to do so. V depl  N C +  N eff (=effective space charge density) Short term: “Beneficial annealing (N A )” Long term: “Reverse annealing (N Y )” - time constant depends on temperature: ~ 500 years(-10°C) ~ 500 days( 20°C) ~ 21 hours( 60°C) Eddy Jans 20 Vertex September 2013

The challenge is to keep the system operational 24/7. Under normal conditions the PLC deals with common problems. Goal: minimize the warm time due to scheduled maintenance of crucial components, repair of malfunctioning components, unexpected problems during LS1 and shutdown periods. Eddy Jans 21 Vertex September 2013

Regular maintenance Yearly maintenance of the R507a chillers is performed by a specialized external company. Downtime ~0.5 day / chiller. Yearly maintenance of the 3 CO 2 pumps is done by Nikhef-technicians. Pump is unavailable for >24 hours. Effective downtime of the system: 2 hours / pump. Repair of failing components So far no component had to be replaced, although a (redundant) pressure gauge stopped working in 2012, but miraculously reincarnated after 6 months. Eddy Jans 22 Vertex September 2013

Unexpected problems How do you know a serious problem occured, causing the cooling to go off and the detector to warm up ? Especially during LS1. Can’t rely on a PVSS-script sending a mail or sms. A modem and sms-routine have been installed in the PLC. modem with Sunrise sim card, so works underground. antenna When a problem occurs every half hour a text message is sent to a list of phones numbers, until the cooling system is again in a proper state. Acted 6 times since Feb. ‘13 due to failing services. Eddy Jans 23 Vertex September 2013

Integrated warm time in 2012: ~1 day Eddy Jans 24 Vertex September 2013 year 2012 = 8 hours

Summary cooling system is continuously operational since >4 years, performance is stable and according to specs, redundancy of crucial components has shown to pay off, clogging filters are annoying, good thermal insulation is less simple than it seems, superheated CO 2 can be dealt with, the warning system that sends sms-es is a great tool, thus far the integrated warm time has been ~1 day/year, so ………. Eddy Jans 25 Vertex September 2013

Eddy Jans 26 Outlook Eddy Jans 26 Vertex September 2013 lets keep it cool till LS2, when the new VELO pixel detector goes in.