LHC CMS Detector Upgrade Project B04-3 Calorimeter Trigger Sridhara Dasu, University of Wisconsin DOE CD1 Review 26 August 2013 Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 1

LHC CMS Detector Upgrade Project LHC Calendar  2012: Discovered the Higgs  : Make upgrades to handle 14 TeV + 2 x cm –2 s –1  : Discover new high energy physics phenomena using data at 14 TeV + study the Higgs + EWSB mechanism  : Make upgrades to handle 14 TeV + 5 x cm –2 s –1  : Program is dependent on what we find by 2018 CMS Plans  2013: Finalize trigger designs and prototyping  2014: Construct the trigger modules  2015: Early operation with pre-production modules (Stage-1)  : Operations with fully upgraded trigger (Stage-2) Rapid plan to ensure success in run!  Plans backed up by thorough prototyping that is already underway The Grand Plan for Trigger Upgrade Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 2

LHC CMS Detector Upgrade Project FPGA & Telecom technology advances  Ubiquitous 10 Gbps links  FPGAs with 72 rx/tx links with built in serdes o Xilinx Vertex-7 platform o Gigantic cores o DSP units, BRAM and Embedded processors  Fiber optic components o Avago mini-pods with 12x  Ripe time to upgrade from VME to uTCA LHC long shutdown  Sufficient time for copper  optical transition for inputs  Ensures that upgraded electronics is ready for operation in 2016  Partial upgrade operation from 2015 itself Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 3 Technical Opportunity

LHC CMS Detector Upgrade Project Calorimeter Trigger Layers 1 & 2 Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 4 US: UK: Vienna:

LHC CMS Detector Upgrade Project Algorithms Algorithm layers  First layer builds local clusters of energy deposits o Determines energy and cluster type (EM or Hadronic)  Second layer builds triggerable objects from clusters o Electron/Photon, Tau, Jet, MET, … Mapping to architectures  Traditional architecture o Algorithm layer-1 maps to physical layer-1 electronics –Key feature: nearest-neighbor sharing –Sufficient to build clusters locally o Algorithm layer-2 maps to physical layer-2 electronics –All clusters available in one FPGA  Time multiplexing architecture o All primitive data passes through one FPGA o Potential for clever on-the-fly processing, if feasible/necessary Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 5

LHC CMS Detector Upgrade Project Current & Upgraded Calo Trigger Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 6 HCAL Energy ECAL Energy Regional Calo Trigger Global Calo Trigger EM candidates Region energies HF Energy HCAL Energy Layer 1 Calo Trigger Layer 2 Calo Trigger Current L1 Trigger System Upgrade L1 Trigger System HCAL Optical Splitters Risk mitigation by operating current and adiabatic stages of upgrade CT concurrently Copper Optical US UK US UK oSLB oRM oRSC CTP MP7 HTRTCC  HTR

LHC CMS Detector Upgrade Project Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 7 Calorimeter Trigger Components TypeNamePurpose TCCTrigger Concentrator CardExisting ECAL trigger primitives (TP) (France) oSLBOptical Serial Link BoardTransmit ECAL TPs to RCT & CTP (Portugal / France)  HTR HCAL Trigger & DAQ BoardCreate and transmit HCAL TPs RCTRegional Calorimeter TriggerExisting trigger system oRMOptical Receiver ModulesReceive ECAL TPs on RCT (Portugal / France for Stage-1) oRSCOptical Regional Summary CardConvert & transmit RCT output (US DOE NP for Stage-1) CTPCompact Trigger ProcessorLarge FPGA w/ optical & backplane (This Project) CIOXCompact IO cross-point switchIntra-crate sharing switch (This Project) MP7Multipurpose ProcessorLarge FPGA w/ only optical links (UK)

LHC CMS Detector Upgrade Project Evolution: Stage-1 Upgrade in 2015 Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 8 Layer 2 Calo Trigger HCAL energy Regional Calo Trigger Global Calo Trigger HF energy oRSC Layer 2 Calo Trigger oRM ECAL energy Layer 1 Processors oSLB ECAL & HF Clusters Half-tower position 4x4 E+H Clusters 2x1 E+H Clusters 2-bit region ID Layer 2 Processors Muon Jets & Sums eGammaTau Heavy Ion Spare RCT RO + Testing GCT fallback remains With just a fraction of final cards or even prototypes, derive many benefits of full upgrade, incl. muon isolation Uses reprogrammed RCT clusters with improved algorithms in Layer-2 Also, brings in all the power of finer grain HF using “Slice Test” of Layer-1 & 2 Improved PU subtraction, Isolation Calculation (e/γ/τ/μ) & Half-tower Position Resolution US Items: oRSCs, oSLB/oRM commissioning CTP Layer 1 Processors RCT RO+Testing CTP UK Items: MP7 Layer 2 Processors 36  HTRs 2 CTP7s  9 CTP7s Add to bring in finer grain EM clusters 2 CTP7s 4 MP7s

LHC CMS Detector Upgrade Project Algorithm Studies Stage-1 uses clusters from RCT+HF processed by new  TCA cards  Adiabatic improvements to calorimeter trigger dataflow  Algorithm stage takes advantage of larger FPGA resources  Performance of the “revised” algorithms is very good o eGammaTau studied by FNAL/Wisconsin o eGamma and muon isolation studied by UCSD o Muon isolation and b-jet tagging with muons studied by Ohio o Heavy Ion jet cleanup studied by MIT o Improved jet finding with PU subtraction by UIC  Further improvements using full ECAL granularity in Stage-1 o Improved eGammaTau isolation & position (FNAL, Wisconsin) o Pile-up handling with lower LSB for ECAL (FNAL, Wisconsin) o Improved jet position and PU subtraction (Davis) Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 9

LHC CMS Detector Upgrade Project Electron / Photon Trigger Improved control on isolation  Factor of 2-3 reduction in rate with small loss in efficiency Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 10

LHC CMS Detector Upgrade Project Tau Trigger Big improvement in efficiency with ~10X rate reduction! Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 11 Current tau trigger has large and negative PU dependence

LHC CMS Detector Upgrade Project Jet Triggers Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 12 Single jet Single jet thresholds similar to current  Multi-jet trigger thresholds better in upgrade (PU sub.)  Small improvement in efficiency turn-on Quad jet H T =  P T Jets

LHC CMS Detector Upgrade Project Process (x2 improvement highlighted) 1.1 x cm –2 s –1 2.2 x cm –2 s –1 CurrentUpgradeCurrentUpgrade W(e ),H(bb) 57.7%87.0%37.5%71.5% W(  ),H(bb) 95.9%100%69.6%97.9% VBF H(  ) 42.6%51.3%19.4%48.4% VBF H(  ) 24.4%44.3%14.0%39.0% VBF H(  ) 17.2%53.7%14.9%50.1% H(WW(ee )) 91.4%97.8%74.2%95.3% H(WW(  )) 99.9% 89.3%99.9% H(WW(e  )) 97.6%99.4%86.9%99.3% H(WW(  e )) 99.6%99.5%90.7%99.7% Stop  bW   e, jets (600 – 450 GeV) 55.8%68.2%50.3%64.8% Stop  bW    jets  600 – 450 GeV  78.1%81.6%76.4%84.5% RPV Stop  jets (200 GeV)70.1%99.9%43.6%99.9% RPV Stop  jets (300 GeV)93.7%99.9%79.7%99.9% Physics Performance Summary Average Improvement: 17% (Low Lumi) & 40% (High Lumi) Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 13

LHC CMS Detector Upgrade Project  Prototypes for all major cards exist & have passed tests  Upgrade is based upon common μTCA hardware platform also used by HCAL & other CMS systems  Upgrade trigger is replacing existing trigger hardware built by the same team → interfaces well understood  Same team wrote software & firmware for existing trigger system → scope & requirements well understood  Requirements are frozen in approved Trigger TDR Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 14 Technical Feasibility

LHC CMS Detector Upgrade Project  Wesley Smith  CMS Trigger Project Manager ,  Trigger Coordinator 2007 – 2012  US CMS L2 Trigger Project Manager 1998 – present  Sridhara Dasu  US CMS L3 Manager for Calorimeter Trigger, 1998 – present  Author of original & upgrade cal. trig. Algorithms 1994 – present  Pam Klabbers – Operations & Testing  CMS RCT Operations Manager (more than a decade on RCT project)  CMS Deputy Trigger Technical Coordinator  Tom Gorski – Hardware & Firmware  Over a decade of engineering on the CMS Calorimeter Trigger  Mathias Blake – Firmware  New to project -- 5 years Applications Engineer at Xilinx, 2 years FPGA Engineer at DRS Technologies, 3 years FPGA Design Engineer at Dolby Labs  Jes Tikalsky and Evan Friis – Software  Providing prototype testing software (Friis has experience with current RCT) Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 15 Calo Trigger Team Experience

LHC CMS Detector Upgrade Project Calorimeter Trigger WBS Detail Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 16 ← Muon Trigger M&S paid by CMS-France

LHC CMS Detector Upgrade Project  Upgraded HCAL Readout & Trigger Electronics (μHTR) is after split that sends data to old electronics (HTR)  μHTR supplies TP to upgrade trigger, HTR continues to send to present trigger  ECAL Trigger Concentrator Cards (TCC) are not upgraded, instead their mezzanines with trigger serial links are replaced (Lisbon):  Optical Serial Link Board oSLB replaces single copper link to current Regional Calorimeter Trigger (RCT – U. Wisc.) with two optical links (WBS )  One optical link to optical Receiver Module (oRM) on RCT (WBS ) o Replaces copper Receiver Module  One optical link to input of Upgraded Calorimeter Trigger: CTP7 Card  574 oSLB’s & oRM’s with Fibers installed in 2014  French Contribution to Calorimeter Trigger (no US Upgrade Funds)  Presently oSLB & oRM in final prototype stage  oSLB:oRM: Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 17 HCAL & ECAL Trigger Primitives OSLB Output / oRM Input: 4x 1.2 Gbps  4.8 Gbps

LHC CMS Detector Upgrade Project  Main processor card  Virtex-7 690T for processing  ZYNQ for TCP/IP + linux  60 10G optical Input links  36 10G optical Output links  Function: Find ET clusters & transmit to Layer-2 Card Count = total + 8 spares + 2 test setups Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger : CTP Card Concept Virtex-7 VX690T FPGA ZYNQ XC7Z03 0 EPP 1.5V Supply 2.5V Supply 3.3V Supply 1V 30A Supply CXP Module 12Tx + 12 Rx CXP Module 12Tx + 12Rx CXP Module 12Tx + 12 Rx 12X Rx 12X Rx (CTP-6 BG View)

LHC CMS Detector Upgrade Project CTP: Virtex-6 Prototype Exists Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 19 Back End FPGA XC6VHX250T/ XC6VHX380T Front End FPGA XC6VHX250T/ XC6VHX380T Avago AFBR- 810B Tx Module 4X Avago AFBR-820B Rx Module MMC Circuitry JTAG/USB Console Interface Mezzanine Power Modules Dual SDRAM for dedicated DAQ and TCP/IP buffering 12x Multi Gig Backplane Connections Prototype with two V6 chips useful for most studies – limited to 6.4 Gbps IO & fewer (48) links Final V7 version offers more IO links and at higher 10 Gbps speed and larger logic

LHC CMS Detector Upgrade Project  CTP7 is simpler than CTP6 prototype  Engineering effort for in-house design based on CTP6 work  Procurement of PCBs is based on recent CTP6 purchase  Good estimate of cost and schedule from prototype manufacture  Quotes available for all parts  Virtex-7 690T is the major cost driver – with recent quote o 3 690Ts donated by Xilinx o 5 690Ts purchased from Xilinx o Total of 8 FPGAs in-hand for prototypes  Optical modules based on recent purchase  Miscellaneous parts also have quotes or estimated from prototype purchases (power modules) Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 20 CTP: Basis of Estimate - Hardware

LHC CMS Detector Upgrade Project  Firmware and software efforts estimated from prototype testing + prior RCT experience  Firmware algorithms for testing provide the basis for the core firmware, which is the most complicated o Tom Gorski and Mathias Blake  Trigger algorithm firmware work is based on prior experience managing GCT project (UK) o Wesley Smith was L1 project manager overseeing GCT  Software effort is based on current RCT o Sophistication of embedded Linux / TCP and shorter timeline for implementation requires professional software effort –Jes Tikalsky o Bulk of testing software from post-doc and graduate student –Pam Klabbers, Evan Friis and graduate students will develop software  Board and Firmware/Software done by the same group (Wisconsin) results in efficiencies Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 21 CTP: Basis of Estimate - Effort

LHC CMS Detector Upgrade Project  Function: For inter-board connections  Input/Output: 48x Gbps  Prototype exists  Fully tested  Firmware is minimal and exists  CIOX cost well known  Ready for production Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger : Crosspoint IO Card – U. Wisconsin Controller (MMC and link mgmt) 4X Avago AFBR-79EQDZ QSFP+ Module Positions 4x4 Lane Bidirectional Multi Gig Backplane Connections Backplane Rx/Tx Redriver ICs (top and bottom sides) (2/crate x 3 crates = 6 + spares)

LHC CMS Detector Upgrade Project Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger : CTP Infrastructure: Vadatech VT894 Crate Test Setup BU AMC13 Vadatech MCH UW CTP-6 TTC Downlink UW Aux (Final system: 3 crates w/ 12 CTP7 ea. + 2 test setups + spare = 6) U. Wisconsin designed backplane with dense card interconnects manufactured & installed in commercial Vadatech VT892 Crate available in Vadatech Catalog

LHC CMS Detector Upgrade Project  The dominant risks are:  Availability of ALL trigger primitive inputs from ECAL, HCAL in optical format  Validation of large optical link plant in limited time  Fully validated trigger algorithms in firmware  Control and operations software  Risk mitigation strategy:  Continue to provide fully operational current trigger system in parallel with upgrade commissioning  Partial operation of the upgrade systems provide tangible benefits from 2015 onwards  Firmware or Software issues are mitigated by provisioning qualified additional temporary personnel o Ability to hire short term qualified staff at UW / FNAL / Elsewhere Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 24 CTP: Risk and Mitigation Strategy

LHC CMS Detector Upgrade Project  Goal (Original):  Provide readout of original RCT descoped during construction project  Present readout through GCT input buffer not workable with trigger evolution  Uses connection to a single CTP6 prototype or CTP7 card for DAQ readout  optical Receiver Summary Card (oRSC) paid by DOE Nuclear  VME Slave Interface Card  Fits in current RCT Crates (1 per crate)  18 Cards in 2015 System  Receives RCT Jet Sum Card Output to GCT on Copper “SCSI” Data Cables  Provides direct optical input to GCT o Bypassing old optical conversion cards  Planned use for Heavy Ion Triggers  Prototype under test   Multiple optical outputs provide:  Inputs for upgrade calorimeter trigger & parallel operation of old & new trigger wherever ECAL & HCAL electronics not available in 2015 Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 25 Calorimeter Trigger Risk Mitigation oRSC Function: Conversion to optical oRSC Input: RCT e  and jet clusters 80 MHz Parallel ECL oRSC Output: 6x copies at 6.4 Gbps optical

LHC CMS Detector Upgrade Project  Input/Output verified – low bit error rate – clear eyes Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 26 Prototypes Status (Optical Links) CTP6 to CTP6 (48x 6.4 Gbps links) oRSC to CTP6 (11x 6.4 Gbps links) oRSC to MP7 (4x 6.4 Gbps links)

LHC CMS Detector Upgrade Project  12-links tested at 6.4 Gbps – no errors Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 27 HCAL TPG to CTP Integration Test

LHC CMS Detector Upgrade Project Running Linux on CTP6 FPGA – PetaLinux  Startup screen via USB console  Root login, pinging controller PC at  Running on CTP-6 back end FPGA (`VHX250T)  Important step towards CTP-7 ZYNQ-based Linux Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 28

LHC CMS Detector Upgrade Project   HTR to CTP6 integration test at Madison (done)  Wisconsin/Minnesota : 12 links operated successfully  oRSC to CTP6 demonstrator at Madison (done)  Test 2-3 oRSCs and interface before shipment to CERN  oRSC to CTP6 platform at Prevessin (done)  2 oRSCs emulate RCT + 1 CTP6 for RCT readout and algorithm tests  Full platform for system operations and control software development  oRSC - MP7 Integration test at Prevessin (done)  Wisconsin/Imperial July 2013 Milestone  CTP6 - oRSC-oRM/RCT - oSLB/TCC Integration Test at Prevessin  Wisconsin/Lisbon/LLR October 2013 Milestone  CTP6 - oRSC-oRM/RCT - oSLB/TCC -uHTR Integration Test at 904  Wisconsin/Lisbon/LLR/Minnesota November 2013 Milestone  CTP7 - MP7 Integration test at Prevessin  Wisconsin/Imperial March 2014 Milestone Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 29 Demonstrators & Critical Integration Tests

LHC CMS Detector Upgrade Project  2013  M&O + R&D Activity : oRSC construction, installation  Firmware development for RCT readout and testing using CTP6  Lisbon / France : oSLB and oRM construction, installation  Minnesota / Wisconsin :  HTR to CTP6 testing  2014  Ensure fully working legacy RCT system with oRSC + CTP6 RO  CTP7, CIOX, … construction, validation  Firmware development for CTP7 testing and 2015 operation  2015  Commission and operate Stage-1 system  Firmware development and integration of Stage-2 system  2016  Commission and operate Stage-2 system Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 30 Adiabatic Upgrade Timeline

LHC CMS Detector Upgrade Project CT Milestones Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 31 Activity IDActivity NameTarget Completion Milestone T1020L2 – Deliver Stage-1 Layer 1 to CERN15-Nov May-2015 T1030L2 – Deliver Stage-2 Layer 1 to CERN10-Mar Sep-2015 T1040L2 – Test operation at CERN15-Aug Feb-2016 T1070L2 – Test High-Luminosity Capability28-Sep Mar-2017 T3310L3 – Commission CTP7 for Initial Ops15-Apr Jul-2015 T4000L3 – Commission CIOX for Final Ops04-Sep Dec-2016 T3320L3 – Commission CTP7 for Final Ops29-Sep Dec-2016 T4740L4 – oSLB-oRM Optical Fibers Complete15-Sep-2014 T4090L4 – CIOX Software and Firmware Complete10-Dec-2014 T4210L4 – CIOX Production Complete08-Jan-2015 T3710L4 – CTP7 Production Complete08-Jan-2015 T4510L4 – CTP Infrastructure Production Complete29-Jul-2015 T4300L4 – CIOX Commissioning Complete09-Sep-2015 T3980L4 – CTP7 Test, Install and Commissioning Complete30-Sep-2016 T3520L4 – CTP7 Software and Firmware Complete30-Sep-2016

LHC CMS Detector Upgrade Project  Calorimeter Trigger Prototypes are in good state  Both CTP6 and CIOX validated  CTP7 is in design stage and is based on CTP6 experience  Crate infrastructure is also ready to go  Calorimeter Trigger Cost Basis is solid  Hardware costs based on purchase orders or firm quotes  Firmware and Software estimates are based on prior experience o Experienced team with risk mitigation by ability to add people  Calorimeter Trigger Risk Analysis  Cost and schedule risks with mitigation plans are presented  Calorimeter Trigger Team ready to get started  CD1 approval will be much appreciated! Sridhara Dasu, 26 August 2013 DOE CD1 Review - US CMS Upgrade - Trigger 32 Conclusion