1. LHC CMS Detector Upgrade Project 401-04-04 Calorimeter Trigger Sridhara Dasu, University of Wisconsin DOE CD1 Review 26 August 2013 Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 1

2. LHC CMS Detector Upgrade Project LHC Calendar  2012: Discovered the Higgs  2013-2014: Make upgrades to handle 14 TeV + 2 x 10 34 cm –2 s –1  2015-2017: Discover new high energy physics phenomena using data at 14 TeV + study the Higgs + EWSB mechanism  2018-2019: Make upgrades to handle 14 TeV + 5 x 10 34 cm –2 s –1  2020-2022: Program is dependent on what we find by 2018 CMS Plans  2013: Finalize trigger designs and prototyping  2014: Construct the trigger modules  2015: Early operation with pre-production modules (Stage-1)  2016-2017: Operations with fully upgraded trigger (Stage-2) Rapid plan to ensure success in 2015-2017 run!  Plans backed up by thorough prototyping that is already underway The Grand Plan for Trigger Upgrade Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 2

3. LHC CMS Detector Upgrade Project FPGA & Telecom technology advances  Ubiquitous 10 Gbps links  FPGAs with 72 rx/tx links with built in serdes o Xilinx Vertex-7 platform o Gigantic cores o DSP units, BRAM and Embedded processors  Fiber optic components o Avago mini-pods with 12x  Ripe time to upgrade from VME to uTCA LHC long shutdown 2013-2014  Sufficient time for copper  optical transition for inputs  Ensures that upgraded electronics is ready for operation in 2016  Partial upgrade operation from 2015 itself Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 3 Technical Opportunity

4. LHC CMS Detector Upgrade Project Calorimeter Trigger Layers 1 & 2 Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 4 US: UK: Vienna:

5. LHC CMS Detector Upgrade Project Algorithms Algorithm layers  First layer builds local clusters of energy deposits o Determines energy and cluster type (EM or Hadronic)  Second layer builds triggerable objects from clusters o Electron/Photon, Tau, Jet, MET, … Mapping to architectures  Traditional architecture o Algorithm layer-1 maps to physical layer-1 electronics –Key feature: nearest-neighbor sharing –Sufficient to build clusters locally o Algorithm layer-2 maps to physical layer-2 electronics –All clusters available in one FPGA  Time multiplexing architecture o All primitive data passes through one FPGA o Potential for clever on-the-fly processing, if feasible/necessary Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 5

6. LHC CMS Detector Upgrade Project Current & Upgraded Calo Trigger Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 6 HCAL Energy ECAL Energy Regional Calo Trigger Global Calo Trigger EM candidates Region energies HF Energy HCAL Energy Layer 1 Calo Trigger Layer 2 Calo Trigger Current L1 Trigger System Upgrade L1 Trigger System HCAL Optical Splitters Risk mitigation by operating current and adiabatic stages of upgrade CT concurrently Copper Optical US UK US UK oSLB oRM oRSC CTP MP7 HTRTCC  HTR

7. LHC CMS Detector Upgrade Project Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 7 Calorimeter Trigger Components TypeNamePurpose TCCTrigger Concentrator CardExisting ECAL trigger primitives (TP) (France) oSLBOptical Serial Link BoardTransmit ECAL TPs to RCT & CTP (Portugal / France)  HTR HCAL Trigger & DAQ BoardCreate and transmit HCAL TPs RCTRegional Calorimeter TriggerExisting trigger system oRMOptical Receiver ModulesReceive ECAL TPs on RCT (Portugal / France for Stage-1) oRSCOptical Regional Summary CardConvert & transmit RCT output (US DOE NP for Stage-1) CTPCompact Trigger ProcessorLarge FPGA w/ optical & backplane CIOXCompact IO cross-point switchIntra-crate sharing switch MP7Multipurpose ProcessorLarge FPGA w/ only optical links (UK)

8. LHC CMS Detector Upgrade Project Algorithm Studies Stage-1 uses clusters from RCT+HF processed by new  TCA cards  Adiabatic improvements to calorimeter trigger dataflow  Algorithm stage takes advantage of larger FPGA resources  Performance of the “revised” algorithms is very good o eGammaTau studied by FNAL/Wisconsin o eGamma and muon isolation studied by UCSD o Muon isolation and b-jet tagging with muons studied by Ohio o Heavy Ion jet cleanup studied by MIT o Improved jet finding with PU subtraction by UIC  Further improvements using full ECAL granularity in Stage-1 o Improved eGammaTau isolation & position (FNAL, Wisconsin) o Pile-up handling with lower LSB for ECAL (FNAL, Wisconsin) o Improved jet position and PU subtraction (Davis) Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 8

9. LHC CMS Detector Upgrade Project Electron / Photon Trigger Improved control on isolation  Factor of 2-3 reduction in rate with small loss in efficiency Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 9

10. LHC CMS Detector Upgrade Project Tau Trigger Big improvement in efficiency with ~10X rate reduction! Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 10 Current tau trigger has large and negative PU dependence

11. LHC CMS Detector Upgrade Project Jet Triggers Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 11 Single jet Single jet thresholds similar to current  Multi-jet trigger thresholds better in upgrade (PU sub.)  Small improvement in efficiency turn-on Quad jet H T =  P T Jets

12. LHC CMS Detector Upgrade Project Process (x2 improvement highlighted) 1.1 x 10 34 cm –2 s –1 2.2 x 10 34 cm –2 s –1 CurrentUpgradeCurrentUpgrade W(e ),H(bb) 57.7%87.0%37.5%71.5% W(  ),H(bb) 95.9%100%69.6%97.9% VBF H(  ) 42.6%51.3%19.4%48.4% VBF H(  ) 24.4%44.3%14.0%39.0% VBF H(  ) 17.2%53.7%14.9%50.1% H(WW(ee )) 91.4%97.8%74.2%95.3% H(WW(  )) 99.9% 89.3%99.9% H(WW(e  )) 97.6%99.4%86.9%99.3% H(WW(  e )) 99.6%99.5%90.7%99.7% Stop  bW   e, jets (600 – 450 GeV) 55.8%68.2%50.3%64.8% Stop  bW    jets  600 – 450 GeV  78.1%81.6%76.4%84.5% RPV Stop  jets (200 GeV)70.1%99.9%43.6%99.9% RPV Stop  jets (300 GeV)93.7%99.9%79.7%99.9% Physics Performance Summary Average Improvement: 17% (Low Lumi) & 40% (High Lumi) Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 12

13. LHC CMS Detector Upgrade Project  Prototypes for all major cards exist & have passed tests  Upgrade is based upon common μTCA hardware platform also used by HCAL & other CMS systems  Upgrade trigger is replacing existing trigger hardware built by the same team → interfaces well understood  Same team wrote software & firmware for existing trigger system → scope & requirements well understood  Requirements are frozen in approved Trigger TDR Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 13 Technical Feasibility

14. LHC CMS Detector Upgrade Project  Wesley Smith  CMS Trigger Project Manager 1994-2007,  Trigger Coordinator 2007 – 2012  US CMS L2 Trigger Project Manager 1998 – present  Sridhara Dasu  US CMS L3 Manager for Calorimeter Trigger, 1998 – present  Author of original & upgrade cal. trig. Algorithms 1994 – present  Pam Klabbers – Operations & Testing  CMS RCT Operations Manager (more than a decade on RCT project)  CMS Deputy Trigger Technical Coordinator  Tom Gorski – Hardware & Firmware  Over a decade of engineering on the CMS Calorimeter Trigger  Mathias Blake – Firmware  New to project -- 5 years Applications Engineer at Xilinx, 2 years FPGA Engineer at DRS Technologies, 3 years FPGA Design Engineer at Dolby Labs  Jes Tikalsky and Evan Friis – Software  Providing prototype testing software (Friis has experience with current RCT) Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 14 Calo Trigger Team Experience

15. LHC CMS Detector Upgrade Project Calorimeter Trigger WBS Detail Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 15 ← Muon Trigger M&S paid by CMS-France

16. LHC CMS Detector Upgrade Project  Upgraded HCAL Readout & Trigger Electronics (μHTR) is after split that sends data to old electronics (HTR)  μHTR supplies TP to upgrade trigger, HTR continues to send to present trigger  ECAL Trigger Concentrator Cards (TCC) are not upgraded, instead their mezzanines with trigger serial links are replaced (Lisbon):  Optical Serial Link Board oSLB replaces single copper link to current Regional Calorimeter Trigger (RCT – U. Wisc.) with two optical links (WBS 401.04.04.06)  One optical link to optical Receiver Module (oRM) on RCT (WBS 401.04.04.05) o Replaces copper Receiver Module  One optical link to input of Upgraded Calorimeter Trigger: CTP7 Card  574 oSLB’s & oRM’s with Fibers installed in 2014  French Contribution to Calorimeter Trigger (no US Upgrade Funds)  Presently oSLB & oRM in final prototype stage  oSLB:oRM: Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 16 HCAL & ECAL Trigger Primitives OSLB Output / oRM Input: 4x channels @ 1.2 Gbps  4.8 Gbps

17. LHC CMS Detector Upgrade Project  Main processor card  Virtex-7 690T for processing  ZYNQ for TCP/IP + linux  60 10G optical Input links  36 10G optical Output links  Function: Find ET clusters & transmit to Layer-2 Card Count = 46 36 total + 8 spares + 2 test setups Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 17 401.04.04.02: CTP Card Concept Virtex-7 VX690T FPGA ZYNQ XC7Z03 0 EPP 1.5V Supply 2.5V Supply 3.3V Supply 1V 30A Supply CXP Module 12Tx + 12 Rx CXP Module 12Tx + 12Rx CXP Module 12Tx + 12 Rx 12X Rx 12X Rx (CTP-6 BG View)

18. LHC CMS Detector Upgrade Project CTP: Virtex-6 Prototype Exists Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 18 Back End FPGA XC6VHX250T/ XC6VHX380T Front End FPGA XC6VHX250T/ XC6VHX380T Avago AFBR- 810B Tx Module 4X Avago AFBR-820B Rx Module MMC Circuitry JTAG/USB Console Interface Mezzanine Power Modules Dual SDRAM for dedicated DAQ and TCP/IP buffering 12x Multi Gig Backplane Connections

19. LHC CMS Detector Upgrade Project  CTP7 is simpler than CTP6 prototype  Engineering effort for in-house design based on CTP6 work  Procurement of PCBs is based on recent CTP6 purchase  Good estimate of cost and schedule from prototype manufacture  Quotes available for all parts  Virtex-7 690T is the major cost driver – with recent quote o 3 690Ts donated by Xilinx o 5 690Ts purchased from Xilinx o Total of 8 FPGAs in-hand for prototypes  Optical modules based on recent purchase  Miscellaneous parts also have quotes or estimated from prototype purchases (power modules) Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 19 CTP: Basis of Estimate - Hardware

20. LHC CMS Detector Upgrade Project  Firmware and software efforts estimated from prototype testing + prior RCT experience  Firmware algorithms for testing provide the basis for the core firmware, which is the most complicated o Tom Gorski and Mathias Blake  Trigger algorithm firmware work is based on prior experience managing GCT project (UK) o Wesley Smith was L1 project manager overseeing GCT  Software effort is based on current RCT o Sophistication of embedded Linux / TCP and shorter timeline for implementation requires professional software effort –Jes Tikalsky o Bulk of testing software from post-doc and graduate student –Pam Klabbers, Evan Friis and graduate students will develop software  Board and Firmware/Software done by the same group (Wisconsin) results in efficiencies Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 20 CTP: Basis of Estimate - Effort

21. LHC CMS Detector Upgrade Project  Function: For inter-board connections  Input/Output: 48x 4.8-6.4 Gbps  Prototype exists  Fully tested  Firmware is minimal and exists  CIOX cost well known  Ready for production Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 21 401.04.04.03: Crosspoint IO Card – U. Wisconsin Controller (MMC and link mgmt) 4X Avago AFBR-79EQDZ QSFP+ Module Positions 4x4 Lane Bidirectional Multi Gig Backplane Connections Backplane Rx/Tx Redriver ICs (top and bottom sides) (2/crate x 3 crates = 6 + spares)

22. LHC CMS Detector Upgrade Project Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 22 401.04.04.04: CTP Infrastructure: Vadatech VT894 Crate Test Setup BU AMC13 Vadatech MCH UW CTP-6 TTC Downlink UW Aux (Final system: 3 crates w/ 12 CTP7 ea. + 2 test setups + spare = 6) U. Wisconsin designed backplane with dense card interconnects manufactured & installed in commercial Vadatech VT892 Crate available in Vadatech Catalog

23. LHC CMS Detector Upgrade Project  The dominant risks are:  Availability of ALL trigger primitive inputs from ECAL, HCAL in optical format  Validation of large optical link plant in limited time  Fully validated trigger algorithms in firmware  Control and operations software  Risk mitigation strategy:  Continue to provide fully operational current trigger system in parallel with upgrade commissioning  Partial operation of the upgrade systems provide tangible benefits from 2015 onwards Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 23 CTP: Risk and Mitigation Strategy

24. LHC CMS Detector Upgrade Project  Goal (Original):  Provide readout of original RCT descoped during construction project  Present readout through GCT input buffer not workable with trigger evolution  Uses connection to a single CTP6 prototype or CTP7 card for DAQ readout  optical Receiver Summary Card (oRSC) paid by DOE Nuclear  VME Slave Interface Card  Fits in current RCT Crates (1 per crate)  18 Cards in 2015 System  Receives RCT Jet Sum Card Output to GCT on Copper “SCSI” Data Cables  Provides direct optical input to GCT o Bypassing old optical conversion cards  Planned use for Heavy Ion Triggers  Prototype under test   Multiple optical outputs provide:  Inputs for upgrade calorimeter trigger & parallel operation of old & new trigger wherever ECAL & HCAL electronics not available in 2015 Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 24 Calorimeter Trigger Risk Mitigation oRSC Function: Conversion to optical oRSC Input: RCT e  and jet clusters 80 MHz Parallel ECL oRSC Output: 6x copies at 6.4 Gbps optical

25. LHC CMS Detector Upgrade Project  Input/Output verified – low bit error rate – clear eyes Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 25 Prototypes Status (Optical Links) CTP6 to CTP6 (48x 6.4 Gbps links) oRSC to CTP6 (11x 6.4 Gbps links) oRSC to MP7 (4x 6.4 Gbps links)

26. LHC CMS Detector Upgrade Project  12-links tested at 6.4 Gbps – no errors Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 26 HCAL TPG to CTP Integration Test

27. LHC CMS Detector Upgrade Project Running Linux on CTP6 FPGA – PetaLinux  Startup screen via USB console  Root login, pinging controller PC at 192.168.1.2  Running on CTP-6 back end FPGA (`VHX250T)  Important step towards CTP-7 ZYNQ-based Linux Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 27

28. LHC CMS Detector Upgrade Project   HTR to CTP6 integration test at Madison (done)  Wisconsin/Minnesota : 12 links operated successfully  oRSC to CTP6 demonstrator at Madison (done)  Test 2-3 oRSCs and interface before shipment to CERN  oRSC to CTP6 platform at Prevessin (done)  2 oRSCs emulate RCT + 1 CTP6 for RCT readout and algorithm tests  Full platform for system operations and control software development  oRSC - MP7 Integration test at Prevessin (done)  Wisconsin/Imperial July 2013 Milestone  CTP6 - oRSC-oRM/RCT - oSLB/TCC Integration Test at Prevessin  Wisconsin/Lisbon/LLR October 2013 Milestone  CTP6 - oRSC-oRM/RCT - oSLB/TCC -uHTR Integration Test at 904  Wisconsin/Lisbon/LLR/Minnesota November 2013 Milestone  CTP7 - MP7 Integration test at Prevessin  Wisconsin/Imperial March 2014 Milestone Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 28 Demonstrators & Critical Integration Tests

29. LHC CMS Detector Upgrade Project  2013  M&O + R&D Activity : oRSC construction, installation  Firmware development for RCT readout and testing using CTP6  Lisbon / France : oSLB and oRM construction, installation  Minnesota / Wisconsin :  HTR to CTP6 testing  2014  Ensure fully working legacy RCT system with oRSC + CTP6 RO  CTP7, CIOX, … construction, validation  Firmware development for CTP7 testing and 2015 operation  2015  Commission and operate Stage-1 system  Firmware development and integration of Stage-2 system  2016  Commission and operate Stage-2 system Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 29 Adiabatic Upgrade Timeline

30. LHC CMS Detector Upgrade Project  Launch Production (Buy parts, fabricate PCBs): 1QFY2014  Deliver Stage-1 Calorimeter Trigger (4 CTP boards + 1 crate + oRSCs): 4QFY2014  Initial operation at CERN (Firmware for Stage-1 + HF clusters): 2QFY2015  Deliver Stage-2 Calorimeter Trigger (All CTPs, CIOx, 3 crates): 2QFY2015  Test High Luminosity Operation of Calorimeter Trigger (All Firmware for Stage-2): 4QFY2016 Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 30 CTP: Production, Installation, etc