R2E and Availability Workshop Interlock Systems I.Romera, M.Zerlauth, B.Todd, S.Gabourin, P.Dahlen, R.Mompo, Y.Bastian, C. von Siebenthal + many colleagues for discussions & contributions R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ2

Outline Beam and magnet interlock systems provide connectivity and diagnostics of > interlock conditions Systems designed to be fast and highly reliable (<1 false dump/year/system) Radiation constraints a priori considered during system design Availability not related to beam energy/intensity Run 1 experience and outlook to HL-LHC R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ3

Beam Interlock System R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ4

Beam Interlock System Example for LHC - Beam 2 (Duplicated for Beam 1) -17 BICs all over the LHC, 2 for each point (right and left), plus 1 in the CCR -Similar architecture for SPS, tree architecture in SPS-LHC TL + LINAC4 LHC BIC in UA63 R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ5

BIS system components CIBMD & CIBTD Partially located in Radiation environment R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ6

Dependability vs Configuration (FMECA) R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ7

BIC availability during Run1 (seen by the beam/OP) Communication lost with CCR BIC (bug in FESA class) Spurious dump from OP button (contact issue) Glitches on Optical Fibers (CIBO exchanged) R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ8 Failure of VME processor (RIO3) in SPS (x8) VME bus access failure in SPS (3x) CIBU failures (1x critical NC in UJ33)

BIC availability (seen by the equipment expert) not enough data… R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ9

BIC availability during Run1 (seen by the beam/OP) Communication lost with CCR BIC (bug in FESA class) Spurious dump from OP button (contact issue) Glitches on Optical Fibers (CIBO exchanged) R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ10 CIBU PS (redundant) Accelerator Fault Tracking Project (AFT) will be an asset also for equipment teams! Failure of VME processor (RIO3) in SPS (x8) CIBU failures (1x critical NC in UJ33, 4x test & monitor) VME PS (redundant) VME bus access failure in SPS (3x)

BIC Radiation BIS hardware not rad-tolerant and hence installed in protected areas (UAs, US,..), apart from client boxes (CIBUs) Equipment relocated from UJ56 to USC55 in LS1, expect no major gain with further relocation Some 10 oo 250 CIBUs remain in (low) radiation areas as installed in client racks CIBUs located in UX85, UX45, RR73, RR77 can be affected during Run3 and HL- LHC according to predictions Most critical component (CPLD XC95144) radiation tested Optical fibers for beam permit loop (non-intrusive monitoring system currently under test) Obsolescence Chassis VME (ELMA replacement?) CIBM (BIC Master) and CIBU: Xilinx (CPLD 95xx and FPGA Spartan 2 & 3) CIBO (optical boards): ELEDs R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ11

BIC Improvements / Maintenance System design is sound and highly dependable = available, exchange of RIO3 -> MENA20 expected to bring the 20% availability increase Maintenance to remain roughly as is, monitoring of fiber optics loop as preventive measure No increased radiation effects expected up to LS3 (SEU and TID), profit from further client relocation/new layout in e.g. IR1/5 Aging of components will result in more electrical failures of components (today well < expectations) Outlook to HL-LHC / new inventory BIC V2.0 foreseen for (latest) HL-LHC era to address new requirements and obsolescence of components, would profit from high-reliability FECs New installations foreseen in PSB (LINAC4 connection) and PS Low statistics! R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ12

Powering Interlock System Distributed system (2-6 controllers / insertion region) providing interlocking between power converters, QPS, cryogenics, UPS, AUG and Access Interfacing magnet powering with BIS Vital role in diagnostics of powering events R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ13

Powering Interlock System 36x SIEMENS PLC modules + redundant supplies 36x Remote IO modules 120x Profibus slaves, 36x CPLDs, 2000x Optocouplers 36x Power modules (5V/5V/24V) 220x AC/DC modules 150x Patch Panels + daughterboards 1800x Current sources, 1400x Optocouplers Partially located in Radiation environment R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ14

PIC availability during Run1 SEU in PLC (UJ14, UJ16, UJ56) PSU Optocoupler (damaged during TS) Current loop spurious triggers (not fully understood ?!) R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ15

Powering Interlocks Radiation None of the COTS components is considered rad-tolerant and hence installed in protected areas (UAs, US,..) or redundant (power supplies) During LS1 9 systems fully relocated as part of R2E efforts (UJ56, UJ14, UJ16) – PLCs already relocated during TS end 2011 Few remote I/O modules remain in low radiation areas (RRs in point 1/5/7) Components radiation tested in Louvaine la Neuve (60MeV, <5E8p/cm2s) for up to ~ 300 Gy, optocouplers & CPLDs to 150Gy Obsolescence CPLD Remote I/O modules SIEMENS PLCs (319 series) R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ16

Powering Interlocks Improvements / Maintenance Maintenance to remain roughly as is, might need at some point campaign to replace opto-couplers in most exposed areas Today no obvious handle to improve availability (no HW failure seen in 6 years)?! Spares might become an issue if we see increased electrical failures Outlook to HL-LHC / new inventory No (increased) radiation effects expected up to LS3, HL-LHC probably brings removal of last components in RRs (PC and QPS) No new installations planned (sc magnets), apart IR1 & IR5 for HL-LHC PIC V2.0 foreseen for (latest) HL-LHC era to address new requirements and obsolescence of components Low statistics! R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ17

Warm Magnet Interlock System Distributed system (8 controllers in LHC) providing interlocking between power converters, water cooling and magnets Interfacing magnet powering with BIS Vital role in diagnostics of powering events R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ18

~ 30 Controllers ~ 500 I/O modules ~ 2500 magnets LHC (2007) TT60 & Ti2 (2005) TT66 (2010) 4 systems TT40 & Ti8 (2004) 2 systems TT41 LEIR (2005) 1 system 8 systems LINAC3 (2009) (2005) WIC installations SPS (2014) 7 systems 1 system PSB (2013) 4 systems LINAC 4 (2014) 2 systems R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ19

SIEMENS PLC modules CPUs, memory card, Ethernet module, F 24DI Power Supply crate Redundant 24V/10A power supplies, redundancy module WIC renovation: Cost estimation HW Remote I/O crate Safety DI and redundant DO modules Magnet interlock boxes Mechanics, PCB, electronic components, connectors Partially located in Radiation environment R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ20

WIC availability during Run1 Network glitches (PC trip) SEU in TI8 remote I/O crate (beam shot on close-by collimator) R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ21

Warm Magnet Interlocks Radiation COTS components not considered rad-tolerant and hence installed in protected areas (UAs, US,..) Exception is remote I/Os installed in TI2/TI8 Radiation tests in TCC2 & TT60 (2002&2003) and PSI (2003) up to 250 Gy for chosen type with one single SEU, Magnet boxes to 1MGy One PLC relocated from US85 to UA83 during Run 1 Main concern is new deployments in injectors (PS, North Area…) Investigating new remote IOs to stay closer to clients and optimize cable lengths Obsolescence Entirely based on COTS, compatible replacements can be found (but need revalidation) R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ22

Radiation CHARM Preparing radiation test of low cost low density I/O module Robust, IP56 Would allow cost efficient architectures for PS, TT2,… R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ23

Warm Magnet Interlocks Improvements / Maintenance Maintenance to remain ~ as is Today no obvious handle to further improve availability Spares might become an issue for modules in ‘radiation areas’ Outlook to HL-LHC / new inventory No (increased) radiation effects expected up to LS3 in LHC >> new installations/renovations in the pipeline in injectors (PS, TL L4-PSB, TL PSB-PS, TT2, TT20, North Area, East Area, AD,..) New architectures possible if radiation tolerant I/O module can be found Low statistics! R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ24

Fast Magnet Current Change Monitors 12 devices installed in LHC, 14 in SPS- LHC Transfer lines to protect from beam excursion due to powering failures in nc magnets Dedicated electronic development in collaboration with DESY R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ25

Fast Magnet Current Change Monitors 26 FMCM modules FPGA, memory, analogue circuity, power supply 26 Voltage dividers & isolation amplifiers Isolation amplifier Partially located in Radiation environment R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ26

FMCM Defective earth connection in power converter RD34.LR7 (end 2011/ beg 2012) Network glitches (thresholds vs PC sensitivity) Isolation amplifier in SR3 R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ27

FMCM vs electrical perturbations Warm magnet trips EXP magnets, several sectors, RF,…tripped No beam, no powering (CRYO recovery) No beam in SPS/LHC, PS affected Majority of perturbations < 100ms/-20% Thyristor converters revealed poor rejection of network perturbations Used for powering normal conducting magnets in high beta regions of the LHC, dump septas,… (RD1, RD34, RQ4/5 in IR7, RMSD,…) 24 preventive beam dumps during 2012 operation, mainly during summer months RD1 and RD34 most affected due to network topology and rated power EPC started development of switched mode converter, deployment for RD1 envisaged 2015/16 R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ28

Fast Magnet Current Change Monitors Radiation Not considered radiation tolerant, installed in radiation free areas Sole (potentially) exposed unit relocated from UJ56 to USC55 during LS1 Obsolescence YES! RAMs, Shift registers,… R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ29

Fast Magnet Current Change Monitors Improvements / Maintenance Maintenance to remain roughly as is New switched mode power converters will greatly reduce number of FMCM dumps Spares might/will become an issue if we see increased electrical failures Outlook to HL-LHC / new inventory Will have to remain outside radiation areas Additional deployments possible or HL-LHC (Crab cavities?!), additional TL elements,… 2 nd production batch or new development foreseen for next year to address obsolescence of components R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ30

Summary Interlock Systems have performed up to (or better) than initial estimates, however statistic is still (too) low Few (unexpected) weak points affecting availability have been or are being addressed (R2E, MENA20, optical loops, power converter re-design,…) System performance expected to be stable up to LS3, after that new designs mainly due to component obsolescence (reaching end-of life of components) and partially new operational requirements Upgrades would profit from common developments towards radiation tolerant / high dependability FECs and data links Should maintain active exchange platform (between equipment groups, experiments, industry,..) concerning reliability engineering (AWG) / radiation tolerant designs (R2E) R2E & Availability Workshop 15 th October 2014 Interlock Systems – IRR & MZ31

We need to know the inventory of electronics that need to be considered for radiation effects… 1) Remaining Inventory: · What options are there for system relocation and what are the advantages/disadvantages? · What equipment will remain in affected environments? · Do you expect the amount of equipment exposed to radiation to be lower, equal, or higher than Run1? · Do you expect problems in the injectors? 2) New Inventory: · What new developments are planned? · What are they motivated by? [e.g. operational requirements, obsolescence requirements, …] We need to know whether cumulative effects due to radiation and electrical reliability are known… 3) Cumulative Effects · What are the predictions for cumulative radiation effects and electrical reliability? · How is your long term supply of system components ensured? [e.g. WorldFIP] We need to know if access to the machine can be expected to be lower, stay the same, or higher, and whether it will be due to radiation…. 4) Maintenance Expectations · Do you expect the maintenance of your system to be lower, stay the same, or higher? · Do you have a preventive maintenance strategy? · Do you expect the ratio of access : remote interventions to be lower, stay the same or higher? We need to know what engineering effort would be needed to improve the system level availability… 5) Improving Availability · What effort would be needed to make a 20% improvement in your system availability We need to know whether there is common ground to develop a new system, designed specifically for the LHC machine radiation environment… 6) Signaling and Communications · At the interface to the machine, what are the signals and bandwidth requirements for your system? · What is the communications mechanism between your system and controls? [Ethernet, WorldFip, …]