1. CMS HCAL upgrade and its electronics Tullio Grassi for the CMS HCAL collaboration Rio de Janeiro 2-3 of February 2012

2. 2 HB HE HO Existing CMS “HCAL” Shielding walls Counting room (no radiation). Back-End Electronics Front-End Electronics is mounted on the detector Castor ZDC HF

3. Existing HCAL readout system 3 Electronics built in year ~2004 Similar FEE for HB, HE, HO, HF, castor, ZDC (differences in the photo-sensors and in the form factors) Nearly identical BE-electronics for HB, HE, HO, HF, castor, ZDC (differences limited to mezzanine cards and cabling) ~6000 channels from Hybrid photo-diodes (HPDs), on HB, HE and HO ~3000 channels from Photo-Multipliers Tubes (PMTs), on HF, Castor and ZDC Measure the energy deposition with a dynamic range ratio = 10000. In HF the range is (1 fC; 10000 fC) with a noise RMS < 0.5 fC. The on-detector FE-Electronics performs a full readout every 25 ns The BE-electronics in the counting room receives the data, keeps the time- alignment between all channels, generates trigger data, selects the data based on the trigger, performs a data compression (“zero-suppression”)

4. Upgrade: Motivations 4 Physics: not covered in this presentation. Problems of the existing system: Obsolescence of electronics built in ~2004 HPD: discharge, ion-feedback,  burst noise HPD gain drift PMT: hits on the glass window; hits on fibers (input of PMTs)  burst noise Occasional problems in the control path Improvements: Replace HPDs with SiPMs Replace PMTs with “thin-glass”, 4-anode PMTs; separate readout of some anodes ( subject to approval and funding ). HB, HE have 16 depths, presently read together  separate them Extend the dynamic range of the FEE ADC by a factor of 10 increase the number of channels by a factor of ~3 Add TDC capabilities, in order to separate noise from physics signals Faster, more reliable control path

5. Block diagram of HCAL electronics 5 New RM New CCM New RM Existing backplanes FEC uTCA system (new) Shielding wall New FE power supplies uHTR ; AMC13 Detector cavernCounting room Trigger cDAQ Control path Data path RM = Readout Module CCM = Clock and Control Module TTS bridge Self- Trigger uTCA=microTCA=industry standard crate FEC = Front-End Controller uHTR = uTCA Hcal Trigger Readout

6. Upgrade: Environment 6 Total radiation dose = 10 krad = 100 Gy Neutron fluence = 10 13 /cm 2 Charged hadron fluence = 2 x 10 10 /cm 2 Magnetic field = 4 Tesla on HB, HE. Lower field for the other parts. On HB, HE: very limited possibilities to change power cables, optical fibers, cooling pipes, on-detector mini-crates (“Readout Boxes” a.k.a. RBX). HF, HO, ZDC, Castor are more easily accessible

7. Upgrade: schedule 7 Installation and commissioning: HF PMT’s and supporting structures: 2013-14 (more info on: https://indico.cern.ch/conferenceDisplay.py?confId=165148) https://indico.cern.ch/conferenceDisplay.py?confId=165148 HO SiPM’s: 2013-14 HF Back-End Electronics: 2013-14 * HB and HE Back-End Electronics: staged during 2013-16 * HF FE-Electronics: during a ~3 months technical stop ~2015 * HB and HE SiPM and FE-Electronics: during a long stop in 2016 or after * * subject to approval and funding!

8. HF data-path: Existing  Upgraded (plans) QIE8 8 Single-anode PMT’s CCA GOL cDAQ 4-anode PMT’s FPGA GBTX 7-bit compressed ADC Fibers (50/125 graded- index multimode) 8-bit compr. ADC + ~6-bit TDC Change to: Counting room Trigger VME electronics cDAQ Counting room Trigger microTCA electronics RM VCSEL VCSEL ? Fibers (50/125 graded- index multimode) Linear Regulators Switching Regulators ? QIE8 QIE10 Shielding wall

9. HF data-path: PMTs 9 4-anode PMT’s FPGA GBTX 8-bit compr. ADC + ~6-bit TDC cDAQ Counting room Trigger microTCA electronics RM VCSEL ? Fibers (50/125 graded- index multimode) QIE10 There are 36 PMT boxes on HF- and 36 on HF+ Our PMT boxes are unhappily called ROBOX’s Boxes are located at a radius of ~1.3m from the beam axis Each box houses 24 PMT’s (total number of PMT’s on HF is 1728) PMT signals are carried by 5-meter 50-Ohm coax cables to the FEE New PMT is R7600U-200-M4 Current Status of new PMTs 1800 PMTs needed (1728 + spares) Total of 1050 PMTs delivered so far Tested in Univ. Iowa + U. of MD (USA) Standard tests (timing, gain, dark current, linearity) and special tests (surface scan, single photoelectron, afterpulse, double-pulse linearity). Lifetime studies underway Need to understand recently seen gain loss in old HF PMTs

10. Digitizing photosensor charge pulses THE PROBLEM: digitizing photosensor charge pulses over wide dynamic range (≈15 bits) with negligible quantization error at frequency > 40 MHz (which is a high rate for this kind of problems) THE SOLUTION: Controlled input impedance over the entire dynamic range Integrate input charge on multiple scaled ranges simultaneously Select one appropriate range for any given input amplitude Digitize the integrator output on that range Read out the digitized result (“mantissa”) and the range code (“exponent”) Pipeline all operations to eliminate dead-time Scaling ratios between ranges must be constant to allow slope/offset calibration  QIE = Charge Integrator and Encoder. An ASIC which digitizes a wide dynamic range into a floating point format. Designer: Tom Zimmermann (Fermilab). Various versions have been used in Tevatron, in neutrino experiments, in CMS HCAL (QIE8). A new version is under development for the upgrade of CMS HCAL and ATLAS TileCal. 10

11. QIE10proto3 chip Submitted in November 2011, chips expected in Fermilab in Feb 2012 It could be used in test beams and test-stands. It is not yet the final chip for HCAL. Features of Proto3:  front end and ADC blocks (as on proto2)  test charge injection  digital clocking and control logic  the range select and multiplex function  biasing  slow control register  parallel ADC outputs and timing discriminator output Missing features:  serialization of the ADC outputs  integrated TDC We plan to use the final QIE10 chip in two different cards: in the HF card (PMT readout), and in the HB, HE card (SiPM readout). 11

12. QIE10 development and issues 12 migration to AMS 0.35µm SiGe BiCMOS process …… irradiation tests …………………………………………... input impedance matched to new photosensors …….. greater dynamic range  100000:1 …………………… all the new features of proto3 (previous slide) …................... serialized ADC output data ……………………………………… bit, word and channel alignment ……………………………….. TDC ……………………………………………………….……….. Dual QIE (two circuits on a single chip) ……………………... To be tested To Be Done To Be Decided Tested ok on proto2 Tested ok on proto1

13. GBT project (new optical link) 13 Project coordinated by Cern, with external collaborators. The GBT project includes: GBTX ASIC (130 nm IBM CMOS): 4.8 Gb/s serdes + clock distribution GBLD ASIC: laser driver Laser PIN (optical receiver) GBTIA ASIC: transimpedance amplifier support for FPGA in the counting room Radiation tolerance targeted for tracker levels. The submission of the first full GBTX chip should be in May 2012. We need to evaluate if the packaging and form factor of the “SFP” parts is compatible with the new HCAL Readout Modules. Can be assembled on a Cern “SFP” : VTRX transceiver or VTTX dual transmitter

14. HF data-path: decisions to be taken 14 4-anode PMT’s FPGA GBTX 8-bit compr. ADC + ~6-bit TDC cDAQ Counting room Trigger microTCA electronics RM VTTX ? Fibers (50/125 graded- index multimode) QIE10 New anode cables for 1, 2, or 4 anode readout ? New HF readout: x2 or x4 number of channels ? QIE10 will integrate a TDC on the rising edge of the analog signal. On the HF FPGA we may implement a second TDC for the falling edge  pulse width measurement More fiber optic cables for TDC data and if we read out all 4 anodes. Need to define the number of fibers If resources were unlimited, we would read out all 4 anodes... Present idea is to read out only two.

15. Data-path : QIE10  GBTX QIE10 generates ADC data = 8 bits at 40 MHz = 320 Mb/s. Output could be two streams at 160 Mb/s. Advantages of serialization: reduce power consumption, use fewer lines on the PCB. Problem: no room for encoding or framing. The GBTX chip is designed to transmit 88 “user” bits every 25ns, when running in 8b10b mode. It provides also a few clock outputs. Requirements for the RM design: stable latency, preferably across power cycles ability to tune the sampling clock of the QIE10 ADC high density of channels: read out two anodes, six QIE10 circuits per GBTX 15 4-anode PMT’s FPGA GBTX 8-bit compr. ADC + ~6-bit TDC cDAQ Counting room Trigger microTCA electronics RM VTTX ? Fibers (50/125 graded- index multimode) QIE10

16. FEE FPGA: Radiation 16 The best FPGA candidates seem ProASIC3L or ProASIC3E from Microsemi (formerly Actel). They are flash-based FPGAs, which are less performant than Xilinx FPGAs, so experience on Xilinx FPGAs cannot be ported completely to Microsemi FPGAs. Radiation-induced problems and solutions in the HCAL environment: TID: various tests show a tolerance greater than 300 Gy (30 krad)  OK. SEL: ProASIC3x not sensitive to SEL (as explained by Federico Faccio on the FPGA-radtol interest group). SEU on configuration: immune. SEU on logic: TMR mitigation with automatic tools. SEGR: Single Event Gate Rupture, it can happen if the FPGA is reprogrammed while exposed to radiation. The solution is to reprogram the FPGA only when there is no beam or other source of radiation. SET: Presently there is no easy solution. We may have to accept it and mitigate it at system level. It can appear in the logic and in the FPGA configuration.

17. Control system: New CCM 17 New RM New CCM New RM Old RBX, backplane FEC uTCA system (new) Shielding wall New FE power supplies uHTR ; AMC13 Detector cavernCounting room Trigger cDAQ Control path Data path RM = Readout Module CCM = Clock and Control Module TTS bridge Self- Trigger uTCA=microTCA=industry standard crate FEC = Front-End Controller uHTR = microTCA Hcal Trigger Readout

18. Existing CCM (Clock and Control Module) 18 O/E RS422 (copper) for slow control TTC fiber for fast control I2C Commands, Low-jitter clocks  to Readout Modules backplane....... TTCrx QPLL commercial components (qualified) Slow control over RS422 is not reliable: errors are mitigated by re-reading/re- writing parameters. The time to transmit the entire configuration parameters is several minutes. Counting room CCM Shielding wall

19. new generation CCM (ngCCM) 19 2 fibers I2C Commands, Low-jitter clocks  to Readout Modules RBX backplane....... GBTX as a transceiver CERN SFP+ HCAL FEC uTCA crate in the counting room ngCCM FEC = Front-End Controller, a generic name for the off-detector electronics which controls the FEE. Baseline plan for FEC: a uTCA crate with one AMS13 plus a number of GLIB cards (enough to drive all ngCCM’s). Shielding wall commercial components (to be qualified) FPGA

20. Existing CCM: architecture and weaknesses 20 In the existing HB, HE, a CCM controls 72 data channels (≈ 1% of the total). The point-to-point links to the counting room are an important single point of failure. If a control link or part of a CCM breaks, we would loose control on 1% of the data channels. This is considered non-acceptable, because HB and HE are accessible only after opening CMS (which is every few years). CCM Counting room HF electronics is more easily accessible, so the single point of failure is less relevant

21. New CCM: architecture with redundancy 21 ngCCM Counting room A few schemes are being evaluated to introduce redundancy. We plan to design links from CCM to CCM. These redundant links allow any CCM to take over the control (bidirectionally) of the “twin” CCM. Intelligence is provided by an FPGA. The first prototype has been built. It is mechanically compatible with the existing HB CCM. It should be electrically compatible with the existing HB RBX backplane and RM (to be tested). It is under test at various universities in USA (UVA, UMD, Princeton, …) The main goal for the next prototype is the “radiation hardening”: for some components (GBT, DC-DC converter, FPGA) there is a defined path for other components (commercial) more work may be needed

22. Powering the FEE: an HB, HE issue 22 The existing Readout Boxes dissipate 90W / RBX. We want to increase the data by ~4x, while keeping dissipation < 200W (limit set by the HB, HE cooling system).  switching regulators can play a key role The plan is to use switching regulators (a.k.a. DC-DC converters) on the HB HE RMs. They can be supplied at a higher voltage, while keeping a “low” internal dissipation.  Need higher voltage from external Power Supplies. The simplest solution is to modify the existing PS system (provided by the CAEN company). We have requested a quotation to the same company for modules with the following specs: Magnetic field and radiation tolerant [at least as A3050] Two Channels, individually controlled and protected [as A3050] No “Remote Sensing” input required VMax and IMax set by trimmer for each channel Interlock capability Screw Lock output connectors Power 280 W / channel [~ as A3050] Max Output voltage across the plus and minus output connectors = 14V. The module should be able to work with the minus input at -2V from earth. OFFER FROM CAEN: Development a new module called A3050HBP as a modified version of the existing A3050. Price: one-time contribution to R&D = 20 k€ in order to receive the first prototype. If we are satisfied, than the price of A3050HBP will be the same price of the existing A3050. Lead-time for first prototype: ~8 months.

23. Counting room electronics 23 New RM New CCM New RM Old RBX, backplane FEC uTCA system (new) Shielding wall New FE power supplies uHTR ; AMC13 Detector cavernCounting room Trigger cDAQ Control path Data path RM = Readout Module CCM = Clock and Control Module TTS bridge Self- Trigger uTCA=microTCA=industry standard crate FEC = Front-End Controller uHTR = microTCA Hcal Trigger Readout

24. Counting room: Existing HF VME crate (HB has identical mother boards but more cables ! ) 24 x 16 for the entire HCAL.

25. Core functions 25 ● Monitoring of crate power, temperature, air flow : CAN bus system connected to DCS ● Configuration and monitoring: VME (via VME/PCI bus adapter from CAEN company) ● Clock and fast controls distribution: custom fanout card plus “cable backplane” ● Event collection for DAQ: “cable backplane” ● Other subsystems (non HCAL) integrated many of the cable backplane functions into a custom backplane, which generated issues with spares --> never enough crates with the custom backplane add-ons.

26. 26 Goals for the Upgrade Backend Electronics ● Plan for maintenance over the period 2015-2025 ● VME is not widely-used in industry and 9U cards are a poor match for the typical PCB sizes encountered in industry ● Avoid custom backplanes, while simplifying the interboard connection structure ● Shift from parallel control buses to serial control systems, which are a better-match with current development ● Ethernet is particularly important and valuable as a standard

27. MicroTCA crate 27 Vadatech VT891 Typical MicroTCA Crate with 12 AMC slots Power Modules 1-6 9-12 AMC slots MCH 1 MCH 2 Derived from AMC std ● Up to 12 AMC slots Processing modules --> uHTRs ● 1 or 2 MCH slots Controller Modules: MCH + AMC13 6+ standard 10Gb/s (~ 1 GB/s) point-to-point links from each slot to hub slots (“star” topology) ● No need for all those in-the-crate cables! ● Redundant power, controls, clocks ● Engineering and production costs similar or lower than VME

28. Controls: IPbus 28 Communication path to the cards is via Ethernet, and therefore conceptually unconstrained ● Each subsystem or card designer might pick their own idea (from raw Ethernet to HTTP to iTunes data transport to...) ● Developing standard : IPbus – Defines virtual A32/D32 bus – Separate address space for each card – Simple interfaces in HW 32 ADDR READ WRITE

29. Upgrade: back-end crate 29 uHTR Trigger Links FE Data Links Configuration/ Slow Controls (Ethernet) Control MCH (commercial) Central DAQ DAQ Timing (AMC13) TTC/Fast Controls

30. Open Task List: hardware 30 HF FE card: design and production of ~180 cards, in close coordination with Fermilab 48V-power supplies for uTCA crates. The power supplies do not need to follow particular standards or have any intelligence related to the uTCA. They need to work in the moderate magnetic field of the counting room. design and production of a “TTS-bridge” card, probably a 6U VME card with an FPGA and ~160MHz IOs: TTC-like on SFP, TTS out, spare I/Os maybe for HCAL self-triggering. Changing fw one could use this card as a TTC emulator as well. A few prototypes would be needed by the end of this year, a few dozens cards may be needed in the future. irradiation tests of various electronics components and modules test and purchase of the CAEN power supplies for the front-end LV.

31. Open Task List: non-hardware 31 Develop custom software for data validation and analysis: CMSSW based Databases: component data entry and tracking, storage and access to test data and on-demand analysis XDAQ development for uTCA Configuration, Diagnostic and Monitoring Software – μTCA Firmware support for all new components (DB storage of FW, emulation, validation, distribution, loading…) Online software enhancements – DCS, CCM, Power supplies, environment (temperature) control and DQM Various DPG-related tasks

32. 32 Questions ?