An overview of FPGA use in the LHC accelerator and the CERN experiments. Dr. Salvatore Danzeca EN-STI-ECE SEFUW: SpacE FPGA Users Workshop, 3rd Edition Thanks a lot to: F. Anghinolfi, K. Wyllie, E. Chesta, A. Masi, M. Brugger, S. Gilardoni, T. Grassi, E. Gousiou, M. Barros Marin, N. Trikoupis, S. Uznanski, P. Peronnard

Outline Radiation environment The challenge of using COTS FPGAs at CERN Accelerator equipment Cryogenics Power Control, Quench Protection, Beam instrumentation Experiments equipment ALICE TPC CMS HCAL The on-going research field and future solutions FPGA system level testing at CHARM CERN facility Conclusions 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments SEFU Workshop 2016

CERN accelerators complex 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Accelerators: Radiation Sources Direct beam Losses Collimators and collimator like objects injection, extraction, dump levels usually scale with beam intensity & energy Beam/Beam, Beam/Target Collisions around experimental areas scale with luminosity/p.o.t. & energy Beam-Residual-Gas circular machines: all areas along the ring scales with intensity, residual gas density & energy Synchrotron radiation (lepton machines) RF (e.g, during conditioning) Radiation sources (irradiators, calibration, tomography, etc.) 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Radiation Levels in the CERN accelerators Sea Level Avionic ISS Space & Deep Space R2E Single-Event Effects Material Damage R2E Cummulative Damage PIXEL PIXEL TRT Atlas MUON COTS Systems Hardened Electronics Electronics Custom Boards with COTS Damage 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

The challenge 𝑁 𝐹𝑎𝑖𝑙𝑢𝑟𝑒 = 𝑵∗σ ∗𝑓𝑙𝑢𝑒𝑛𝑐𝑒 The reliability is a main concern for the CERN equipment The criticality of the equipment can be very high 2 proton beams at 6.5 TeV of ~3×1014 p+ each Total stored energy of 0.7 GJ Sufficient to melt 1 ton of Cu Tiny fractions of the stored beam suffice to quench a superconducting LHC magnet or even to destroy parts of the accelerator The radiation effects on the FPGAs can lead to lose the beam (beam dump) Lost time for physics The radiation effects on the FPGAs can lead to a failure of the safety system of the LHC Part of the machine can be destroyed In the experiments the radiation effects should be minimized in order to have meaningful data for the physics analysis High number of devices (N): 𝑁 𝐹𝑎𝑖𝑙𝑢𝑟𝑒 = 𝑵∗σ ∗𝑓𝑙𝑢𝑒𝑛𝑐𝑒 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

FPGA vs ASIC use at CERN In the accelerator sector the use of ASIC is very limited due to the high development time and resources needed COTS FPGAs are the primary choice Benefits Accelerator COTS: Low cost (high volume required) low development costs re-programmable well developed & affordable platforms of tools In the experiments the high radiation levels close to the interaction points and the high performances required a custom ASIC development In environments with low/medium radiation and where power/integration issues are less critical FPGA are now an attractive alternative Experiments 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Accelerator equipment using COTS FPGAs in radiation areas Cryogenics Power Converter Quench protection system Beam Instrumentation Common architecture: Radiation environment 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Cryogenics FPGA use Card architecture includes: Anti-fuse FPGAs LHC: Installed below dipoles in all arcs from cell 8 and in shielded areas. Card architecture includes: Anti-fuse FPGAs Other critical components (regulators ADCs, DACs) Overall design The electronic cards use binary-data interfaces and perform simply math. Complex calculations performed at higher level PC (no radiation). FPGA/PCB design TMR (FFs) safe state-machines watchdogs memory data refreshing resets, power-cycles 800 WorldFIP electronic crates Active channels: 6500 Temperature 800 Pressure 500 Liquid He level gauges 1500 Cold Mass Heaters 500 Beam Screen Heaters 1100 Mechanical Switches (I/O) 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Why choosing an Antifuse? Reliability: No SEE after applying all the mitigation techniques Card malfunction can result in a beam-dump and might cause damage to the accelerator (e.g. heaters) Long life time Can withstand more than 800 Gy Drawbacks: Very small code possible (all the resources used in the project) Non-reprogrammability can lead to the necessity to change all the cards if an upgrade is needed (never happened) Higher cost Currently searching for a new FPGA candidate: Reprogrammable, non latching-up, tolerant >1kGy (100krad) 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Power converter, Quench protection system and Beam instrumentation Power converter radiation tolerant controller FGClite Radiation tolerant solution based on the ProAsic3 FPGA Will replace the senior FGC2 which led to 10 and 3 dumps during the 2012 and 2015 operation respectively (cross section of ~10-10cm2) 1600 units to be produced and replaced Quench protection system DQHSU is a rad-tol fast measurement board based on ProAsic3 to supervise quench heater discharges 1232 cards installed under the LHC dipole Main feature: Communication handling ADC and DACs controller digital notch filter design + processing Beam instrumentation Point to point architecture (different from the common fieldbus) Use of the ProAsic3 with RadHard high speed link (GBT/Versatile link) 350 units to be produced 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Why choosing an flash based FPGA (ProAsic3)? Reliability: No SEE after applying all the mitigation techniques Card malfunction can result in a beam-dump Reprogrammability Different features already added on-the-go during the operation Drawbacks: TID limit is less than the Antifuse ~500 Gy Reprogrammability feature not working above 200 Gy Limited resources in terms of logic gates (i.e DSP math) Currently searching for a new FPGA candidate: Reprogrammable, non latching-up, tolerant up to >500 Gy (50krad) with DSP capability 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Experiments architectures Many subsystems use or will use FPGAs in radiation areas (but not in extreme radiation areas like central trackers) ATLAS: TileCal, muons CMS: HCAL, muons, GEM, CP-PPS LHCb: RICH, calorimeters, muons ALICE: TDC In the HEP Front-End Electronics installed around 2005, FPGAs were mostly used for control and readout logic with external high speed links. Future users of SRAM FPGAs in the radiation zone will be ATLAS TileCal, ATLAS Lar and LHCb-RICH. They favour the Kintex7 as it has the possibility to implement auto-correction of SEUs in the config bits (using an IP provided by Xilinx.) Use from different groups (CMS HCAL, CMS Muon detectors, ALICE) of the flash based Igloo2 and Smartfusion2 for future upgrades A more detailed list of the experiment FPGA use is in the Backup 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

ALICE TPC Readout Electronics 216 Readout Control Units Read data from Front End Card Sends data over fiber for analysis and permanent storage 2 SRAM based FPGAs & 2 flash based FPGAs per system TMR is not possible due to resource limitation The reconfiguration networwk consists of: A radiation tolerant flash memory, a radiation tolerant flash based FPGA and the DCS board – an Embedded PC with Linux. Corrects SEUs in the configuration memory of the Xilinx Virtex-II pro Why it works: Active Partial Reconfiguration How it works: RCU support FPGA reads one frame at the time from the flash memory and Xilinx configuration memory. The frames are compared bit by bit. If a difference is found, the faulty frame is overwritten. Upgrade for the RCU2 Use a single FPGA based on the SmartFusion2: Linux on ARM+ FPGA J.Alme TWEPP 2013 J.Alme CERN FPGA Workshop 2014 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

FPGA CMS HCAL FPGAs both in the Front-End Electronics (FEE) mounted on the detector and in the Back-End electronics (BEE) located in the counting rooms. Existing system: produced in 2004 The new FPGAs system uses the SERDES at about 5 Gbps 450 boards produced and 1600 FPGAs to be used before the 2018 Upgraded system: in steps between 2014 and 2019. Existing Upgrade Expected TID on FEE 2 Gy 45 Gy (4.5 krad) Expected 1 MeV-equivalent neutron fluence on FEE 1 x 1011 / cm2 7 x 1011 / cm2 Front-End Actel antifuse (for control only) Microesemi flash-based (control and data) Igloo2 Back-End Xilinx and Altera Xilinx Number of FPGA types 2 (FEE) + ~8 (BEE) ~5 (FEE) + 5 (BEE) Number of developers 1 (FEE) + ~4 (BEE) ~6 (FEE) + 4 (BEE) T. Grassi ECFA 2014 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Current research from different CERN groups Igloo2- SmartFusion2 as future FPGA for new developments Several test campaigns carried out by EN-STI-ECE group on the full qualification of the igloo2 device family Fabric logic SEU sensitivity Embedded SRAM sensitivity Reprogrammability TID lifetime PLL (on going) ARM processor (on going) System level testing from the CMS HCAL group Test of the SERDES block – high speed communication link Xilinx Version7 FPGAs (Zynq/ Kintex7) ATLAS Liquid Argon (LAr) Calorimeter and LHCb RICH is testing the suitability of the Kintex7 devices EN-STI-ECE group evaluates the possibility of using SRAM based FPGAs (Zynq) in CERN accelerator equipment 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Testing FPGAs in the accelerator environment Typical process for quality assurance of COTS based electronics applies also to FPGAs (CERN guidelines) Radiation testing on FPGAs usually carried out in proton/ion beam Once the design is ready the system can be tested: where? 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

CHARM a radiation facility for system testing CHARM = Cern High Energy AcceleRator Mixed-Field Facility Main purpose Radiation tests of electronic equipment and components in a radiation environment similar to the one of the accelerator Large dimension of the irradiation room - Large volumes electronic equipment - High number of single components - Full systems Numerous representative radiation fields - Mixed-Particle-Energy: Tunnel and Shielded areas, atmospheric and space environments - Direct beam exposure (proton beam 24 GeV) CHARM! 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Conclusion CERN harsh radiation environment and challenges Criticality and reliability is a key aspect in the FPGA use at CERN Large unit number ~1000 Accelerator equipment common architecture Antifuse FPGAs Flash based FPGAs Experiment common architecture High speed links Research on new solutions for new developments Collaborations and sharing is essential (and welcome) Radiation test facility to make possible a system level testing 3/17/2016 S. Danzeca - An overview of FPGA use in the LHC accelerator and the CERN experiments

Thank you

BACKUP

Spectra vs. position CHARM can reproduce different scenarios of radiation environments: LHC UJ zone LHC shield zone ISS Orbit Atmospheric 200m 99% LHC – UJ zone BACKUP 90% 99% Atmospheric (h=200m) ISS Orbit 12/11/2018

FEE systems of CMS Sub-system Approx radiation FPGAs in 2008-2012 FPGAs after 2012 (radiation ~6x higher) Tracker [2] 200 kGy. 1014 n/cm2 No FPGAs (ASICs only) ECAL [3,4] 25 kGy. HCAL [5] 3 Gy. 1011 n/cm2 Actel anti-fuse FPGA, for control only igloo2 (flash) for control, data processing, TDC and transmission from 2016 Muon detectors 0.4 Gy. 5x1010n/cm2 SRAM FPGAs [6, 7]. igloo2 (flash) for control, data processing, TDC and transmission from 2014 [15] CT-PPS 200 Gy, 2x1012 n/cm2 per 100/fb integrated lumin. Did not exist igloo2 (flash) for control, data processing, TDC and transmission from 2018 ? 23 23

FEE systems of ATLAS Sub-system Approx radiation FPGAs in 2008-2023 FPGAs after 2023 (radiation ~6x higher) Tracker Xx kGy. 1014 n/cm2 No FPGAs (ASICs only) Liquid Argon Calorimeter [16] On chamber (3.4 kGy): probably no FPGAs. On sTGC (90 Gy): investigate with xilinx for processing and 10 Gbps readout links. Tile calorimeter [17] 15 Gy. 1011 n/cm2 640 Mb/s, severe errors in data transmission, loss of configurations Demo project with xilinx for processing and 10 Gbps readout links Muon detectors xx Gy. 5x1010n/cm2 24 24

FEE systems of LHCb 1014 n/cm2 240 Gy, 1012 n/cm2 70 Gy. 50 Gy. 80 Gy. Sub-system [9, 10] Approx radiation in 2008-2018 FPGAs in 2008-2018 FPGAs after 2018 (radiation ~6x higher) Inner Tracker 60 kGy. 1014 n/cm2 No FPGAs in the hot zone (ASICs only) Under study (probably not required) RICH 240 Gy, 1012 n/cm2 Actel AX (antifuse) + Actel ProAsicPlus (flash) for controls Xilinx Kintex7 Outer 70 Gy. SciFi Tracker Under study (Microsemi Igloo2) Calorimeters 50 Gy. Actel AX (antifuse) for 80 MHz processing, Actel ProAsicPlus (flash) for 40MHz processing and control [9] Muons [11] 80 Gy. Actel ProAsicPlus, for calibration system Under study (Flash device) 25 25

FEE systems of ALICE Sub-system Approx radiation FPGAs in 2008-2012 FPGAs after 2012 (radiation higher) TPC [20] 16 Gy. 1011 n/cm2 [21] Virtex-II Pro (SRAM) for datapath, with its configuration verified and refreshed by an Actel ProASIC+ (flash) Microsemi SmartFusion2 (flash). Links up to 5 Gbps. Problems observed. FPGA PLL loss of lock → use TTCrx instead [10]. ProASIC3 (flash) for radmon. DDL (Detector Data Link, common to all subsystems) Actel ProASIC+ (flash). 200 MB/s links All others ~0 at the FPGA location