LHC Cryogenics Schneider PLCs: Radiation tests at H4IRRAD Mitigation measures M. Calviani, M. Brugger, G. Spiezia (EN/STI) E. Blanco, J.M. Beckers, D. Willeman, P. Durand, Ph. Gayet (EN/ICE)

Outline 2  Schneider PLCs issues on the LHC CRYO Control system  H4IRRAD test campaign results  Proposed mitigation measures LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department

LHC downtime due to the Cryo Schneider PLC issues  Is radiation the cause ?  Potential SEUs: 7 LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department3 All underground! PLC Schneider Premium LHC Cryo SurfaceCavern P1863 P264 P4114 P6116 P8104 Total

LHC refrigerators control architecture RM SL Sector L (3.3 Km) 4 LHCA QURA LHCCA QURCA QSCCA LHCCB QSCCB LHCB QSRB QSCB QUI QSDN QSAA Comp 4.5K Comp 1.8K Main Dryer Comp 1.8K Comp 4.5K QURCB Cold Box 4.5K LN2 Buffer CB 1.8K Connection Box UCB 4.5K QSRA QSKA QSCA QSAB Main Dryer Local & Central Control Rooms SCADA Data Servers RM SR Alcoves Sector R (3.3 Km) Tunnel Cavern Surface Shaft QSDN RM PA Profibus DP WorldFIP Return Module SL & SR UX85: ✓✓✓ UX45: ✓ US65: ✓ US45 : ✓ US85 : ✓ LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department4

Radiation tests  Objectives  Confirm PLC radiation sensitivity: as shown in the LHC cryogenics control system during 2011 operation  Not for testing PLC radiation hardness  Validate potential solutions  Architectures to test: 1. PLC Premium CRYO-like  LHC Cryogenics like sample but in two different configurations:  P575634M with memory card  P576634M without memory card 2. PLC Premium with remote I/O backplane  PLC CPU in safe area (UL) and a passive bus X connection to the remote I/O backplane located in radiation area. 3. PLC Quantum Redundant architecture  Possible solution to increase availability (should be seen decoupled of the radiation issue) LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department

H4IRRAD SPS beam Production target Internal irrad zone (LHC tunnel) External irrad zone (LHC shielded zones) Access is possible only from the top shielding (removal of 160 cm Fe blocks) LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department6  H4IRRAD can reach 1 year of nominal LHC in ~1 week (underground zones)  LHC RadMon for monitoring – agreement with FLUKA simulations within 20-30%

Radiation sensitivity  The PLCs have been tested during a period of low beam intensity due to their significant radiation sensitivity  Crashes observed after ~5-10 minutes at H4IRRAD nominal beam (~5*10 5 HEH/cm 2 /min)  Clear correlation between PLCs CPU crash and radiation  Cross-section estimated from a period with ~5*10 4 HEH/cm 2 /min (US85 ~ 3.5*10 2 HEH/cm 2 /min) ~5*10 4 HEH/cm 2 /min ~2*10 5 HEH/cm 2 /min ~4.5*10 5 HEH/cm 2 /min LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department7

Sensitivity results TypeHEH to failureStatus PLC Premium (low memory user application occupancy: ~ 7% exec code ) 5*10 6 HEH/cm 2 /failureNot responding + restarted PLC Premium (P8 QURCB-like application: ~ 18% exec code) 2*10 6 HEH/cm 2 /failureNot responding + restarted Remote I/O backplane (cabled I/Os) ≥5*10 9 HEH/cm 2 /failure OK! PLC Quantum redundant3*10 6 HEH/cm 2 /failure (increased availability) PLC malfunction  PLC Premium cross-section in agreement with LHC operation data  Radiation sensitivity appears to be dependent to the user application memory occupancy (~ 2.5 times)  Crash with no remote access possible with PLC Premium!  No failure observed or induced by the remote I/O backplane LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department

LHC radiation levels in P4/P6/P8  P8: LHCb luminosity  Values ≤4*10 7 HEH/cm 2 /2011 (in US85)  Evolution will directly depend on the cumulated LHCb lumi (1-2 fb /2) – same value expected for 2012  P4/P6: beam-gas due to pressure spikes in the cavities (P4) and in the extraction septa (P6)  Values ~10 6 HEH/cm 2 /2011 in UX  2012 levels should be similar as this year  After LS1 values will depend on the vacuum behaviour with 25ns operation (situation not yet clarified)  P2: ALICE luminosity  levels insignificant in US25 LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department9

Conclusions of the test campaign  What we confirmed:  High sensitivity to radiation (SEUs) of Schneider Premium PLCs  SEU induced failure cross-section in agreement with LHC operation data during 2011  What we observed:  Radiation sensitivity dependent on the user application memory occupancy (to be confirmed with Schneider)  Radiation robustness of the remote I/O backplane and I/O cards (including CPU reliability with that architecture) – no failure observed  Equal sensitivity with or without flash memory card  Redundancy solution offers limited radiation sensitivity robustness LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department10

Mitigation actions (winter shutdown) 1. Improve availability : The radiation campaign in H4IRRAD showed remote I/O low radiation sensitivity and, therefore, the PLC CPU relocation to a safe area (UL) is the solution retained.  Feasibility discussed already within the R2E activities (EN/ICE, EN/EL, TE/CRG)  Action prepared for the winter shutdown (including ordering, cabling, electricity)  Affecting P4, P6 and P8 (in that precise order). P2 not needed 2. Other improvements (once the PLCs are in a safe area) 1. Remote accessibility - Additional Ethernet card to be installed in the relocated PLCs. (21) - Allows remote access via IP reducing time and resources during interventions 2. Robustness - Eliminate flash memory cards in some PLCs (9, awaiting Schneider validation) - Already done in 2011 for the P8 QURA PLC 3. Diagnostics - Enrich the onboard PLC diagnostics (collaboration with Schneider Electric) LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department

Mitigation actions (Long Term)  Long-term actions (LS1)  Awaiting 2012 LHC operation to decide the next move (reliability of Schneider PLCs)  But in parallel: 1. Continue radiation testing on H4IRRAD facility on equipments like the ones remaining in radiation areas (PLC Quantum, Siemens on turbines,…) and also in other architectures (redundancy) 2. Study a possible full relocation of all active electronics in P4 and P6 as done in P8 3. Validate PLC redundant architectures for a possible migration of some equipments during LS1 This is independent of radiation issues and will increase the availability of the cryogenics control system in view of the large downtime induced by an issue in the control system (only if the new situation does not satisfy the required availability!) LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department12

Acknowledgments  Many thanks to:  EN/MEF for the precious support during construction and during each installation phase  The H4IRRAD team and EN/STI for the follow-up of the test area operation (radiation and beam monitoring)  EN/ICE for availability and reactivity  Thanks a lot for your attention LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department13

LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department14

H4IRRAD external zone p+ beam EN/EL EN/ICE PLCs LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department15

H4IRRAD configuration SPS beam Production target Internal irrad zone (LHC tunnel) External irrad zone (LHC shielded zones) Access is possible only from the top (removal of 160 cm Fe blocks) LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department16

Passive vs. active bus X solutions 17  The active bus X includes two electronic pieces in both ends of the cable that can be a source of problems under radiation environment. LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department

H4IRRAD test area overview  What is it?  Test area to test equipment in LHC-like particle spectra  H4IRRAD can reach 1 year of nominal LHC in ~1 week (tunnel and alcoves)  Radiation monitoring  Radiation monitoring based on the LHC RadMon system  FLUKA simulations employed to cross-check and understand the radiation field (agreement within 20-30%) LMC, 9/Nov/2011M. Calviani, E. Blanco - EN Department18