Outcome from the RPC DSS/DCS Action Matrix Review 2015 Anton Dimitrov RPC LV2 Coordination Meeting

Review Summary RPC DCS Action Matrix Review went perfectly fine. The questions raised were trivial to answer. No remarks to follow up. RPC DSS Action Matrix Review raised couple of questions leading to a fruitful and easily converging discussion. Special remarks were given about the persistency to kill the LV mainframe. Request to have rack S2G18 under UPS discussed and rejected.

DSS signal to RPC HV Mainframe 2 alarm conditions are sent by the RPC gas system in case of a – stop of the gas distribution or a single gas rack being in state “stop” – alert from the IR2 analyser However, only 1 action reaches the HV mainframe in case of a – stop of the gas distribution or a single gas rack being in state “stop” The hardwire DSS signal is connected to the interlock of the CAEN SY1527 Mainframe, therefore it kills the Mainframe whenever it arrives. On 05.Mar.2015 a new mechanism in the DSS action matrix was implemented which will – first, provide an immediate sms notification to the RPC DOC (165508) from the condition (Digital Input): “DI_Gas_SG5_RPC warning at …” – second, hold the action for N minutes, i.e. do nothing for N minutes, where N is to be defined at this meeting – third, at the end of the time interval of N minutes, checks again the condition (DI) and if the condition is still TRUE it will perform the action (kill HV mainframe), otherwise will do nothing (HV will remain ON). The hardwire DSS signal is connected to the interlock of the CAEN SY1527 Mainframe The reviewers asked why don’t we send an OFF or STDB signal instead of a kill. My answer that we would love to, but we miss the proper firmware was satisfactory. After some discussion of how dangerous the kill turns out to be to our system, they proposed to have a “software” DCS solution anticipating the “hardware” DSS signal. Conclusion: Despite previous statements that the only safety signal is the hardwired one and the DCS commands might be lost due to network congestion, the commission agreed/recommended to treat the hardware condition as a trigger and apply a safe software solution before we finally receive a DSS signal with big enough persistency to assure the entire mechanism working.

Few questions to answer before we decide how to treat the gas kill signal How does the gas system work? What is the system inertia? What can be the source of a gas kill signal? What persistency do we need before we kill the mainframe? How do we react to the warning condition? What can we do by software while the persistency is ticking away?

Mixer IR1IR2 5% iso-butane 5% iso-butane, Inertia ~ 15min USA Distribution HV System DSS no action sms2 sms1, sms3 Notifications: sms1 – EN_ICE gas software to gas piguet, gas group sms2 – CMS DSS to RPC DOC, gas piquiet, sms3 – RPC DCS to RPC DOC sms0 I nertia ~ 45min

Gas system: IR1 When there is IR1 alarm, the DSS cuts the mixer in the surface gas room (SGX5) to avoid propagating flammable mixture underground. However, the pump continues working, therefore there is still gas distribution between USC gas room and the cavern. In other words there is gas flux through the chambers, but without 10% of fresh mixture being added. System can run like this till the pressure in the gas lines drops below the minimum. Gas lines from the surface to the detector are quite long, so it takes about 45 minutes till the first rack goes in state “stop” or the distribution pump stops. Therefore, inertia ~ 45 minutes, no kill signal to the maniframe is sent and no need to worry at all about a kill, gas is always flushing through the detector, though without a fresh mixture.

Gas system: IR2 When there is IR2 alarm, the DSS cuts the electro valve of the isobutane bottle, i.e. there is no isobutane going to the mixer. The mixer itself continues working till there is enough pressure in the line between the bottle and the mixer. When the pressure goes below a predefined minimum the mixer stops. The inertia of the isobutane line is ~ 15min before the mixer stops. The electro valve is automatic which means that it switches on/off automatically according to the triggering condition (isobutane level > 5%). Therefore, the total inertia of the system in this condition is ~ 1h, no kill signal to the mainframe is sent and no need to worry at all about a kill, gas is always flashing through the detector, though without a fresh mixture.

Gas system: Gas Racks When a single gas rack is in stop we used to receive a gas kill signal immediately. There is no natural mechanism which can introduce any inertia, so this was the main reason we used to suffer kill signals. A single rack can stop when it goes to under pressure. Racks are working very close to 0 mbar relative pressure wrt to environmental (UXC) pressure. – Change of differential pressure between tunnel and UXC or UXC and USC can lead to a kill. – Change of cavern ventilation mode has always created problems, though we tried to impose to EN-CV a better procedure. – Single rack failure can also lead to a rack stop. To restart a rack in STOP is really a fast procedure. It takes about 1-2 minutes. This the ONLY place where we need to define the N minutes for persistency of the DSS signal. We have to take into account that: – Calling the gas piquet should be immediate, therefore a clear message should arrive to the RPC DOC phone and to the cDCS alert screen in order to avoid wasting time to power a computer and search and stare at the panel till they understand the problem. – I agreed with Roberto that all gas services should reach the cDCS alert screen and calls to the gas piquiet should be redundantly doubled: cDCS shifter and RPC DOC should call him independently when they receive the alarms. Sometimes sms arrive late, so really the cDCS alert screen is the most reliable. – Anyway, we have to foresee time for the gas piquet to react properly in case he sleeps or travels in the vicinity. I consider N=20min reasonable for a proper reaction.

Gas system: Pump When the gas distribution pump stops, obviously very soon all the racks will be in state stop. For the DSS signal logic there is no difference/urgency whether 1 or 10 racks are in stop. I have never seen the Pump is state Stop for so many years apart from the yesterday’s incident which was a clear mistake, not only in the communication, but also in the operation (they connected the bypass to a wrong contact which forced a power cut).

DCS action to HV mainframe Finally, after we know how the gas system behaves, which are the most probable conditions to cause a gas alarm and what is the inertia of the system to different conditions, we can define the proper persistency of the DSS signal. Moreover, we can design new software features in DCS to take action while waiting for the kill, which at the end may never come if the problem/cause is solved on the way. Which options do we have? – We still have the firmware option to ask CAEN to work on the V1sel implementation. Then we can connect the DSS signal not to the interlock but to the V1sel in which case we can set the detector automatically to Standby, even after a persistency, because all the HV channels Settings.V1 are already set to 7000V. – We can pick the DSS condition (DI) which triggers the alarm and use the persistency time to put the detector in Standby. Alternatively, we can also pick up a single rack in STOP state instead of a DSS trigger and perform the same action after some persistency, in which case we need to check ourselves whether the condition (rack in error) is still TRUE and of not we do not perform the action. In this case I would wait 15min with HV ON and then go to STDB if condition is still TRUE. In this case the persistency N should be ~20-25min. – Being able to read in DCS the mixer state, the pump state, the gas rack state, the main input flow and isobutane level ratio we can make our selves whatever condition we would like to trigger on. In this way we can go through the different possibilities discussed in the previous slides. – If we decide to completely avoid the kill while the detector is ON (no matter at what voltage) we shuold implement a way to switch OFF the HV automatically by DCS. For the moment this is not possible because LV and HV are coupled in the FSM states therefore an OFF command will switch OFF LV&HV which is not desirable. – We can even go further and decide to switch off the detector safely as I would do it by hand if I am in front of PC: Initial condition which triggers all actions appear Wait 15 minutes. If condition is still TRUE, go to STDB. Wait another 15 minutes. If condition is still TRUE, go to 4500V Wait another 15 minutes. If conditions is still TRUE, go to 1000V Wait another 10 minutes. If condition is still TRUE, switch HV OFF After total of N=60min, if initial condition is still TRUE, send the DSS signal to mainframe  kill. In all previous cases, if the initial condition is FALSE, break  no action. In this case, the RPC DOC should take care to power back the detector from the step it was left in. Obviously, this implies adding new FSM states such as HV_4500 and HV_1000 which might not be trivial if possible at all (to discuss with Alberto). – Optimally, the thing we can immediately implement is to use the existing FSM states – STANDBY and OFF. In this scenario Initial condition which triggers all actions appear Wait 15 minutes. If condition is still TRUE, go to STDB. Wait another 15 minutes. If condition is still TRUE, go OFF After total of N=45min, if initial condition is still TRUE, send the DSS signal to mainframe  kill. All this logic should imply not simply sending commands at a given time, but a new mechanism which checks every tot minutes the initial condition and performs the action according to the condition. Pretty much like the P-correction mechanism which checks the pressure every tot minutes and applies correction if needed. However, I would like to split the OFF command into HV_OFF and LV_OFF. There is no need to power cycle FEBs and LBB if there is a gas problem. Moreover, this splitting would be more needed to the other alarms coming to the LV mainframe.

DSS signal to RPC LV Mainframe 2 alarm conditions are sent by the CMS power system and the CMS magnet – UPS_on_batteries sends a kill signal to the LV mainframe with delay of 390s. – CMS magnet fast discharge with current > 10000A sends a kill to LV mainframe with delay 0s. The hardwire DSS signal is connected to the interlock of the CAEN SY1527 Mainframe, therefore it kills the Mainframe whenever it arrives. CMS requests to significantly decrease the delay discussion