1. A microTCA Based DAQ System for the CMS GEM Upgrade
Thomas Lenzi
on behalf of the CMS Muon Group
TWEPP 2016

2. GE1/1 Upgrade Project

3. Motivations
Problem
During Phase II, foreseen to start in 2025, the luminosity of the LHC will double, resulting in higher fluxes of particles which will impact the detectors.
Solutions
Add redundancy in the most forward region of the muon spectrometer where only CSCs are instrumented.
Integrate GEM and CSC data to improve the performance of the trigger algorithms.
Results
Maintain the excellent trigger capabilities of the muon system at high fluxes of particles.

4. GE1/1 DAQ System
Before the full installation during LS2 ( ), a small slice will be installed in CMS in January 2017 with a near final version of the front-end electronics.

5. VFAT3 Front-End
Zero suppression performed on the OptoHybrid
GEM readout strips
Connected to the GBT chipset
The VFAT ASICs are mounted on small PCBs called the Hybrids which makes the link between the detector and the GEB.
Shown here are the Hybrids of the VFAT2 ASICs.
By CERN

6. GEM Electronics Board Not enough space to run flat cables
1-m-long PCB connecting the VFATs to the OptoHybrid
Provides power to the VFATs and shielding to the detector
Good signal integrity at
40 MHz for slice test
320 MHz for LS2 system
By Lappeenranta University of Technology

7. OptoHybrid Components Xilinx Virtex-6 FPGA (XC6VLX130T) GBT chipset
SCA chipset
2x VTRx & 2x VTTx
Functions
Slow control of the 24 VFAT2s
Forward tracking data to the off-detector electronics (over VTRx)
Pack the trigger data and send it to the CSCs and GEMs electronics (over VTTx)
GBT
Tested and working with two E-Links to and four E-Links from the FPGA at 320 MHz each.
By Université Libre de Bruxelles and Texas A&M

8. Calorimeter Trigger Processor
Zynq Processor (XC7Z045)
Embedded Linux which includes Xilinx’s debugging tools
Custom web application for monitoring
uHal and IPBus over TCP/IP
AXI bus to FPGA to transfer the IPBus requests to the on-detector electronics
Green = GBT RX
Red = GBT TX
Virtex-7 FPGA (XC7VX690T)
36x GBT cores (no errors over 69h test)
24x 8b/10b links
DAQ link to the AMC13
One CTP7 can control 12 OptoHybrids meaning we need 6 boards per endcap.
By University of Wisconsin

9. Test Beam Test beam setup November 2015 test beam using SPS beam
Two GEM detectors fully equipped with their DAQ system
GLIB used instead of CTP7
Gas mixture used is Ar-CO2-CF4 (45%-15%-40%)
Triggers provided by scintillators placed on both sides of the chambers
Results
Various measurements on the chamber efficiency
Robustness of the DAQ system against high rate of triggers
Current through the high voltage divider

10. GEM-CSC Integration Tests
B904 setup
GEM detector placed above CSC chamber
Full DAQ chain
Link between the OH and the CSC’s OTMB
Goals
Create path for trigger data between GEM and CSC
Software development and testing
Data taking with cosmic muons
Integration in the CMS DAQ
Results
The data packer is running on the OptoHybrid and the trigger information is received by the CSCs.

11. Irradiation Setup Goals Estimate SEU rates in CMS and test aging
Beam parameters
Cyclotron at Louvain-La-Neuve
Mono-energetic proton beam
Energy between 14 and 62 MeV
8 cm of diameter beam spot centered on FPGA
10% homogeneity
Flux up to 2 x 108 particles cm-2 s-1
24 hours of testing
Electronics
Irradiated 2 Xilinx Virtex-6 FPGAs (XC6VLX130T) on OptoHybrid boards
Counting done by external OptoHybrid limiting the control logic on the irradiated FPGAs
Communication done through HDMI cable
Monitoring with ChipScope (JTAG)

12. Error Detection Sampling rate of 40 MHz
Use Xilinx’s Soft Error Mitigation (SEM) tool to detect errors in the configuration memory.
Use the ECC embedded in the BRAM controllers.
Use triplication to detect and mitigate errors in the logic (CLB and DSP).
SEM and ECC can correct single bit flips (recoverable) and detect double or more bit flips (critical).

13. SEU Cross Section
SEM and ECC can correct single bit flips (recoverable) and detect double or more bit flips (critical).
Energies within the range of background particles in CMS.

14. Total Ionizing Dose
In CMS, the cumulated dose at the end of Phase II will be on the order of 10 kRad.

15. Consequences on GE1/1
Upper limits on cross sections for a luminosity of 5 x 1034 cm-2 s-1:
SEU
Cross section
SEUs in CMS per FPGA
SEM recoverable
3.08 x 10-7 cm2
27 / day*
SEM critical
3.19 x 10-8 cm2
3 / day*
BRAM recoverable
1.02 x 10-7 cm2
9 / day*
BRAM critical
1.71 x 10-8 cm2
1 / day*
Assuming 24h at nominal luminosity
OptoHybrid firmware resource usage
36% of the LUTs
3% of the BRAMs
Mitigation
SEUs will have limited impact on the firmware due to the low resource consumption.
Triplication can protect sensitive modules (results in backups).
Periodic reset (every 10 minutes) will recover all errors.

16. Summary
DAQ System
The developments on the DAQ system for the slice test are nearly finished.
The GBT chipset and GBT-FPGA core have been tested successfully and can communicate. 36 GBT-FPGA cores have been implemented in the CTP7.
The link to the AMC13 and CMS DAQ is operational.
Tests
The GEM-CSC integration tests are on going in B904 at CERN.
The DAQ chain has been tested during a test beam campaign in November 2015 and showed excellent integration of the components.
The OptoHybrid’s FPGA has been irradiated to quantify the number of expected SEUs in CMS. Mitigation techniques (ECC, triplication, …) can correct for ~99% of the errors.

17. Outlooks
Slice Test
Installation starts in January 2017, commissioning in March 2017, and LHC data taking in May 2017.
Instrumentation of 10 GEM detectors (5 superchambers) in one endcap of CMS.
Real size test before full installation during LS2
LS2
Installation during the long shutdown.
Full ring of detectors in both endcaps.
Upgrade to VFAT3 ASIC.
New version of the OptoHybrid with 3 GBT chipsets to control the VFAT3s.

18. Backups

19. LHC Schedule

20. GEM-CSC Integration

21. Triple-GEM Technology

22. Test Beam Setup

23. Position Dependency

24. Triplication
When an SEU affects the triplicated logic, is the mitigation working ?
SEM takes a few milliseconds to correct errors.
At high fluxes, the errors in the configuration memory accumulate and triplication fails.
At low fluxes, time between errors is large and allows triplication to mitigate the consequences of the upsets.

25. Radiation in CMS

26. Slice Test