1. LHC CMS Detector Upgrade Project Level-1 Trigger Update Wesley Smith, U. Wisconsin Project Management Group July 22, 2015 22 July 2015, Wesley Smith PMG: Trigger Update 1

2. LHC CMS Detector Upgrade Project  Install fully parallel and higher bandwidth optical path for CSC  New CSC Muon Port Cards mezzanine cards installed in cavern during LS1 o Prerequisite done by Operations Program  Alleviates bottleneck & sends all segments from each CSC (robustness to PU and collimated signals)  Build up new Track Finder in 2015 and commission in parallel, ready by 2016 Phase 1 Muon Trigger Transition 22 July 2015, Wesley Smith PMG: Trigger Update 2 Old systemNew system Cavern Counting Room

3. LHC CMS Detector Upgrade Project EMU Trigger Upgrade CavernCounting Room μTCA: Advanced Mezzanine Cards from Telecommunications Computing Architecture (commercial telecommunications hardware) PMG: Trigger Update 3 22 July 2015, Wesley Smith MPC-EMUTF Optical Fibers* (Rice) Muon Port Card Mezzanine* (Rice) Endcap Muon Track-Finder (U. Florida) EMUTF Infrastructure (U. Florida) Muon Sorter (Rice) Trigger MotherBoard Data MotherBoard Clock&Control Sector Processors Clock & Control Muon Sorter *M&S on FY13 Ops. Prog. VME MTF7 (MP7)

4. LHC CMS Detector Upgrade Project Muon Port Card (Ops) (Rice - M&S on Operations Program) 22 July 2015, Wesley Smith PMG: Trigger Update 4  Use the existing MPC main board  Backplane interface to TMB remains unchanged x 60 + 20 spares  3 original optical links are still available  New mezzanine card with new FPGA and new links  Production done, tested, installed (60 needed + 15 spares + 5 test setups = 80)

5. LHC CMS Detector Upgrade Project 5 EMUTF: Hardware Layout MPC Mezzanine Upgrade MTF7 μTCA Board Optical Patch Panel 22 July 2015, Wesley Smith PMG: Trigger Update

6. LHC CMS Detector Upgrade Project  Require three μTCA chasses to process 12 sectors + sort output  MTF7 occupies 2 μTCA slots,  12 units in system, each covers 60 o sector  Muon Sorter also built on MTF7, 1 unit in system  use Boston U AMC13 for clocking and DAQ  Re-route and split optical signals using 12 passive optical patch panels EMUTF Hardware Counts 6 SP MSMS Chassis #1 Chassis #2 Chassis #3 22 July 2015, Wesley Smith PMG: Trigger Update

7. LHC CMS Detector Upgrade Project EMU Track-Finder (U. Florida) 22 July 2015, Wesley Smith PMG: Trigger Update 7 Back: Core FPGA card ⇒ ⇐ Front: Optics card  Muon Track Finder card: MTF7  13 needed + 4 spares + 3 test setups = 20  Optimized for maximum input from muon detectors (84 input links, 28 output links)  Dual card w/large capacity for RAM (~1GB) to be used for p T assignment in track finding Development and testing complete, production started

8. LHC CMS Detector Upgrade Project MTF7 Hardware Prototypes @ CERN 8 Details of board status in next slides Base Board [pT LUT mezzanine not shown] Optical Board Pass-through Backplane Board 22 July 2015, Wesley Smith PMG: Trigger Update

9. LHC CMS Detector Upgrade Project Muon Sorter Overview  Legacy Muon Sorter (VME) with legacy mezzanine (UF)  Part of legacy trigger, sends up to 4 muons to GMT  Legacy Muon Sorter (VME) with upgraded optical mezzanine (Rice)  Has 12 channel optical TX and RX at max of 3.2 Gbps  Backup solution if uGMT or upgraded CSCTF are not ready  uTCA Muon Sorter (aka Upgrade Muon Sorter)  UF MTF7 production boards + custom Rice mezzanine  Sends up to 36 muons to uGMT using 10 Gbps MP7 link protocol  Custom mezzanine generates and sends local trigger requests to TCDS as LVTTL and/or LVDS Upgrade Muon Sorter using UF MTF7 22 July 2015, Wesley Smith PMG: Trigger Update 9

10. LHC CMS Detector Upgrade Project Custom Mezzanine for the Upgraded EMU Muon Sorter - Rice  Intended for local triggering with the new uTCA based CSC Track Finder  Provides L1A Request to TCDS  Resides on the MTF7 processor in place of the PtLUT mezzanine card  Designed at Rice University in spring of 2015 for the uTCA Muon Sorter  Five boards were built, two assembled and tested 22 July 2015, Wesley Smith PMG: Trigger Update 10

11. LHC CMS Detector Upgrade Project  Major HW milestone: all 12 sectors instrumented [Alex Madorsky]  All EMUTF system cabled (not OVTF)  PCIe access to all boards verified o First chance to talk to 12 boards from a single computer o Works perfectly  MCHs configured so we can talk to their CPUs and AMC13s via single Ethernet cable o Thanks to Jamal Rorie for help with MCH configuration  MPC fiber ID codes read out and verified o Except 1 MPC not sending light – might not have been replaced yet o Link IDs from MPCs on 3 channels are sometimes read as zeros [Alex will debug during next visit]  Crate water cooling was off on 07/13, should be on within days EMUTF – Hardware 22 July 2015, Wesley Smith PMG: Trigger Update 11

12. LHC CMS Detector Upgrade Project EMUTF @ PT 5!! [boards] 22 July 2015, Wesley Smith PMG: Trigger Update 12

13. LHC CMS Detector Upgrade Project EMUTF @ PT 5!! [patch panels] 22 July 2015, Wesley Smith PMG: Trigger Update 13

14. LHC CMS Detector Upgrade Project  We have 14 working production modules, plus 2 prototypes.  13 needed for system (12 processors + sorter)  Remaining 6 production modules for EMUTF (for spares and test stands) plus the entire OVTF production (16 modules) are not yet done  Cannot repair the 5 broken production FPGA boards that had been encountered (were examined by company)  4 additional boards were sent to a new assembler (ABC Assembly) with special procedures on bake-out and temperature, but all boards failed tests when received   Conclusion is that PCBs were poorly manufactured, not issues with the the NELCO material used 22 July 2015, Wesley Smith PMG: Trigger Update 14 Remaining MTF7 production - A. Madorsky

15. LHC CMS Detector Upgrade Project  Intend to manufacture new PCBs for the remainder of the production using new I-Speed material with a new PCB vendor  Will order 4 to be made, assembled, tested  If good, produce rest  Expect about 2 months to finish production 22 July 2015, Wesley Smith PMG: Trigger Update 15 Next Steps – MTF7 Boards

16. LHC CMS Detector Upgrade Project  Milestone on packer/unpacker [Ben Michlin, Rice]:  Event generator completed!  Output is binary and txt  All generated values are correct number of bits, but not physics o ie: CSC sectors not 1-12  MTF7 (with AMC13 header/trailer) packer/unpacker completed!  Everything in correct format  All information included o Event record, CSC data, RPC data, etc.  Unity test complete!  Generate data  pack  unpack  pack  unpack  Many bugs/typos/bits shifted in wrong direction were found and fixed  Noise check and recovery completed!  If data is not in expected format of MTF7 (or AMC13), event is skipped and packer/unpacker continues with the next event. EMUTF – Software 22 July 2015, Wesley Smith PMG: Trigger Update 16

17. LHC CMS Detector Upgrade Project  Currently using private tool for generating bitwise identical C and Verilog source code  Nikhil Pratap Ghanathe is Working on High Level Synthesis [Automated conversion of algorithmic description into RTL (Register Transfer Level) code]  Evaluated Vivado HLS to be best choice from alternatives  Latency and resource utilization must be under acceptable limits  Use of C test-bench to verify RTL code  Explored and Evaluated various High level synthesis tool, acclimatized to the design flow  Identified fundamental modules for implementing using HLS tool  Translated a module for converting primitives into angular values  Design projects comparable resource statistics  Pipelined whole design  Latency=2 clock cycles, Throughput=1 EMUTF – CHREC development 22 July 2015, Wesley Smith PMG: Trigger Update 17

18. LHC CMS Detector Upgrade Project Phase 1 Calo. Trigger Upgrade in Parallel: Split inputs from ECAL+HCAL HCAL Energy (HTR) ECAL Energy (TCC) Regional Calo Trigger Global Calo Trigger EM candidates Region energies HF Energy (μHTR) HCAL Energy (μHTR) Layer 1 Calo Trigger Layer 2 Calo Trigger Current L1 Trigger System Upgrade L1 Trigger System oSLB oRM  ECAL: optical Serial Link Board (OSLB) and optical Receiver Mezzanines (oRM) connect to Present and Upgrade Calorimeter Trigger (*Details of evolution in breakout). o All installed and operating in CMS Global Runs  HCAL: optical splitters drive both HTRs and μHTRs 22 July 2015, Wesley Smith PMG: Trigger Update 18 US UK US UK HCAL Optical Splitters 2016: HCAL uHTR & assoc. Cal. Trigger Cards (Cluster Finding) (Object Finding) 2015: ECAL, Legacy RCT & associated Cal. Trigger Cards*

19. LHC CMS Detector Upgrade Project Stage-1 Upgrade in 2015 (ops) 22 July 2015, Wesley Smith PMG: Trigger Update 19 RCT 0 RCT 1RCT 2 RCT 17 … oRSC 0oRSC 1oRSC 2…oRSC 17 Existing RCT 18 crates MP7 (new) Algorithm Card(s) (1-3) Imperial HW + Core FW/SW USCMS Algorithm FW/SW CTP7 (new) Single Readout Card In Situ RCT Test / Monitor USCMS HW/FW/SW Global Trigger (existing) GT Inputs remain unchanged DAQ – AMC 13 (new) 18 oRSCs (new) Layer-1 Layer-2 USCMS

20. LHC CMS Detector Upgrade Project RCT DQM: RUN 249951 -> Collected Sunday June 28 th RCT GCT RCT CTP7 Readout: Regions/EM 22 July 2015, Wesley Smith PMG: Trigger Update 20

21. LHC CMS Detector Upgrade Project Comparison RCT/GCT -> Collected from RCT readout Sunday June 28 th RCT CTP7 Readout: Comparison Now that validated for triggering crossing with GCT readout, can use RCT CTP7 DAQ to view +/- 2 bx for measurements of PU, out-of-time triggers 22 July 2015, Wesley Smith PMG: Trigger Update 21

22. LHC CMS Detector Upgrade Project Stage 1 Algorithm Firmware architecture “GCT in a chip” – J. Berryhill, FNAL team 22 July 2015, Wesley Smith PMG: Trigger Update 22 18 10-Gbps optical fibers of RCT regions, egammas, feature bits Products for GT and DAQ Pipelined processing at 80 MHz with latency 10.5 bx

23. LHC CMS Detector Upgrade Project  Emulator – Data comparison for Energy Sums 22 July 2015, Wesley Smith PMG: Trigger Update 23 Stage-1 Calorimeter Trigger

24. LHC CMS Detector Upgrade Project  Stage 1: Perfect agreement for Jets and Taus! 22 July 2015, Wesley Smith PMG: Trigger Update 24 Jet + Tau Trigger Emulator – Data comparison

25. LHC CMS Detector Upgrade Project  Test of Stage-1 as primary CMS calorimeter trigger for 2 hours in preparation for 25 ns running  Change GT Firmware and manually switch fibers into GT  Run with 25 ns L1 Menu and appropriate HLT  Results  Ran for 2 hours without issues and reverted back in 10 minutes smoothly  DQM monitoring looked as expected.  A number of details in Trigger Supervisor, MP7 alignment and synchronization, BX offset in DQM and TCDS errors need attention, but no show-stoppers  Offline data analysis looks reasonable (next slides)  Stage-1 Cal Trigger should be ready as primary trigger when 25 ns running starts 22 July 2015, Wesley Smith PMG: Trigger Update 25 Stage-1 Calorimeter Trigger - J. Berryhill

26. LHC CMS Detector Upgrade Project Compare Stage-1 and Legacy CT: Central Jets 22 July 2015, Wesley Smith PMG: Trigger Update 26 Stage1 in production GT, Legacy in test crate 10 GeV jet seed threshold in both Stage1 and Legacy systems Central Jets - L. Apanasevich

27. LHC CMS Detector Upgrade Project Compare Stage-1 and Legacy CT: Forward Jets 22 July 2015, Wesley Smith PMG: Trigger Update 27 Stage1 in production GT, Legacy in test crate 10 GeV jet seed threshold in both Stage1 and Legacy systems Forward Jets

28. LHC CMS Detector Upgrade Project Compare Stage-1 and Legacy CT: e-gamma 22 July 2015, Wesley Smith PMG: Trigger Update 28

29. LHC CMS Detector Upgrade Project Stage-2 Upgrade in 2016+ 22 July 2015, Wesley Smith PMG: Trigger Update 29 CTP+0 CTP+1CTP+2 CTP+8… 9 CTP7s +  MP7 Algorithm Card(s) (10+2) Imperial HW + FW/SW USCMS Algorithm FW/SW  TCA Global Trigger DAQ – AMC13 Layer-1 Layer-2 CTP–0 CTP–1CTP–2 CTP–8… 9 CTP7s –  USCMS Full flexibility for traditional or time-multiplexed architecture Highest precision location of objects with dynamic clustering

30. LHC CMS Detector Upgrade Project  CTP7 Goals for Cal Trig Layer 1  Provide needed interconnect flexibility for all proposed architectures and link speeds  Maximize the power available to the Virtex-7 690T FPGA for trigger data processing  Exploit latest-available technologies  First two units delivered in Dec, 2013  Excellent results in checkout  Full Production Complete Mar ‘15, 2015  Excellent checkout results continued o 38 total CTP7s shipped to CERN  True Triple-Link-Rate, Multi-Clock-Domain Card o 6.4 and 4.8 Sync Rx, 10G Async Tx in same MGT quad 22 July 2015, Wesley Smith PMG: Trigger Update CTP7 Card Reminder – U. Wisconsin 30

31. LHC CMS Detector Upgrade Project Production CTP7s 22 July 2015, Wesley Smith PMG: Trigger Update 31 CTP7 Without CXPs + heatsinks ← Front Back→ CTP7s ← in Stage 2 Rack

32. LHC CMS Detector Upgrade Project Stage 2 Layer 1 Installation PMG: Trigger Update 22 July 2015, Wesley Smith32

33. LHC CMS Detector Upgrade Project  Calorimeter to Layer-1 patch panels have been fully connected  576 ECAL Links  504 HCAL Links  72 HF Links  18 patch panels connect:  ECAL TCCs/oSLBs with CTP7s  HCAL/HF μHTRs with CTP7s  Layer 1 → Layer 2 Testing  All 864 links validated o One bad CTP7 transmitter found and replaced (next slide)  Test of mapping with tower ID pattern validated 22 July 2015, Wesley Smith PMG: Trigger Update 33 Stage-2 Calorimeter Trigger

34. LHC CMS Detector Upgrade Project  Defective transmitter →  Avago CXP – one channel failed in operation  Hot swap replacement:  Disconnect optical fiber (MPO12)  Pull handle on CXP on CTP7  CXP slides out, taking care of its own shutdown  Go to cabinet  Get replacement CXP  Insert new CXP into CTP7  CXP reinitializes, turns back on and starts operating  Plug back in optical fiber  Elapsed time: 3 min o Mostly walking over to cabinet!  Much easier because of spacing of CTP7s  18 card installation. 22 July 2015, Wesley Smith PMG: Trigger Update 34 Replacing a CTP7 Transmitter

35. LHC CMS Detector Upgrade Project Layer-1 DAQ  Work on multi-card DAQ progressing well  We have demonstrated that we can successfully operate the DAQ link with complete Layer-1 Stage-2 crate with 6 CTP7s and AMC13 o The screenshot below shows error free overnight run: o The test above was performed using the Stage-1 RCT/CTP7 firmware in one of the final Stage-2 crates at P5. PMG: Trigger Update 22 July 2015, Wesley Smith35

36. LHC CMS Detector Upgrade Project Updated CIO Card  QSFPs upgraded to single CXP  Added ZYNQ as controller (TCP/IP capable)  Added COM and dual USB ports for utility I/O  Allows remote console access to Vadatech MCHs/PMs and AMC13 MMC Backplane Connector CXP COM Port Dual USB Interface ICs ZYNQ 12 Bidirectional Lanes GbE 22 July 2015, Wesley Smith PMG: Trigger Update 36

37. LHC CMS Detector Upgrade Project  Muon Port Cards commissioned, operating  Endcap Muon track-Finder System in production  Endcap Muon track-Finder all 12 Sectors instrumented  Endcap Muon Trigger preliminary firmware and software operating at CERN  Stage-1 Calorimeter Trigger operating at CERN  Stage-1 Calorimeter Trigger firmware and software operating at CERN.  Stage-2 Calorimeter Trigger production complete  Stage-2 Calorimeter Trigger final installation complete and commissioning underway  Stage-2 Calorimeter Trigger operational firmware and preliminary software operating at CERN 22 July 2015, Wesley Smith PMG: Trigger Update 37 Trigger Upgrade Summary