LHC CMS Detector Upgrade Project US CMS Trigger Upgrade Proposal Wesley Smith, U. Wisconsin Fermilab Directors’ Review 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 1Wesley Smith, 16 July 2013

LHC CMS Detector Upgrade Project Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 2 L1 Trigger Upgrades: Detectors Electromagnetic Calorimeter Electromagnetic Calorimeter Hadron Calorimeter Hadron Calorimeter Muon Detectors Muon Detectors

LHC CMS Detector Upgrade Project  Trigger TDR Status, Overall Goals  Overview of Trigger Upgrade Design  Muon Trigger Upgrade  Calorimeter Trigger Upgrade  Overview of Cost & Schedule Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 3 Outline

LHC CMS Detector Upgrade Project L1 Upgrade TDR  Final presentation to LHCC was Tuesday June 11th  Public version of the TDR here:  Given approval, although still awaiting final formal LHCC write-up Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 4

LHC CMS Detector Upgrade Project  Trigger rates are driven by the increase in luminosity, the center-of-mass energy, and by higher PU  Mitigate by improving e/γ isolation, τ id, muon p T resolution, muon isolation, jets with PU subtraction, and L1 menu sophistication  Increase system flexibility with high bandwidth optical links and large FPGAs, and standardize on μTCA telecom standard  Build and commission upgrade in parallel with current trigger system to safeguard physics, decouple from LHC schedule  Overall design:  Two-layer calorimeter trigger with tower-level precision and PU subtraction  Integrated muon trigger combining all 3 muon systems in track-finding  Global Trigger not limited by number of triggers (>128) and offering more sophisticated conditions (e.g. invariant mass) Fermilab Director's Review -- US CMS Trigger Upgrade 5 L1 Trigger Upgrade – overview Wesley Smith, 16 July 2013

LHC CMS Detector Upgrade Project Final Overall Upgrade Scheme Fermilab Director's Review -- US CMS Trigger Upgrade 6Wesley Smith, 16 July 2013

LHC CMS Detector Upgrade Project Muon Trigger Transition Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 7  Install fully parallel and higher bandwidth optical path for CSC  New CSC MPC mezzanine cards installed in cavern during LS1  Alleviates bottleneck and send all segments from each CSC (robustness to PU and collimated signals)  Build up new Track Finder in 2015 and commission in parallel, ready by 2016  Endcap hardware design is proposed by CMS to be used for all muon regions: endcap, barrel, and overlap Old systemNew system

LHC CMS Detector Upgrade Project Muon Trigger WBS Detail Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 8 → Calorimeter Trigger

LHC CMS Detector Upgrade Project EMU Trigger Upgrade CavernCounting Room μTCA: Advanced Mezzanine Cards from Telecommunications Computing Architecture (commercial telco, see backup slide) Fermilab Director's Review -- US CMS Trigger Upgrade 9Wesley Smith, 16 July MPC-EMUTF Optical Fibers Muon Port Card Endcap Muon Track-Finder EMUTF Infrastructure : Muon Sorter Trigger MotherBoard Data MotherBoard Clock&Control Sector Processors Clock & Control Muon Sorter

LHC CMS Detector Upgrade Project Muon Port Card Upgrade (Rice) x spares Fermilab Director's Review -- US CMS Trigger Upgrade 10  Use the existing MPC main board  Backplane interface to TMB remains unchanged  3 original optical links are still available  New mezzanine card with new FPGA and new links Wesley Smith, 16 July 2013 (60 needed + 15 spares + 5 test setups = 80)

LHC CMS Detector Upgrade Project  Muon track finder processor: MTF7  13 needed + 4 spares + 3 test setups = 20  Optimized for maximum input from muon detectors (84 input links)  Dual card with large capacity for RAM (~1GB) to be used for p T assignment in track finding  Current prototype based on Virtex 6 FPGA with 10 links at 10 Gb/s  Virtex 7 FPGA preproduction design in progress Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade : EMU Track-Finder Upgrade (U. Florida) Optical module Core module (FPGA and RAM) LUT mezzanine

LHC CMS Detector Upgrade Project : EMUTF Infrastructure (U. Florida) Fermilab Director's Review -- US CMS Trigger Upgrade 12 Not US* *Processing the resulting tracks with RPC or CSC hits is a US responsibility Wesley Smith, 16 July Endcap Muon Track-Finder : Muon Sorter

LHC CMS Detector Upgrade Project  Full muon trigger upgrade won’t be ready until 2016, nevertheless 2015 with the LHC at 13 TeV will be an important year for physics  Add isolation to endcap muons from old CSCTF with a new mezzanine card on the old muon sorter to send an optical path to 2015 calorimeter trigger initial upgrade  Analogous to MPC upgrade: Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade : Intermediate Sorter Upgrade (Rice)

LHC CMS Detector Upgrade Project Upgrade Cal. Trig. Layer 1 & 2 Hardware US: UK: Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 14 Vienna:

LHC CMS Detector Upgrade Project Calorimeter Trigger WBS Detail Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 15 ← Muon Trigger M&S paid by CMS-France

LHC CMS Detector Upgrade Project  Upgraded HCAL Readout & Trigger Electronics (μHTR) is after split that sends data to old electronics (HTR)  μHTR supplies TP to upgrade cal. Trigger, HTR continues to send to present cal. Trigger  ECAL Trigger Concentrator Cards (TCC) are not upgraded, instead their mezzanines with trigger serial links are replaced (built by Lisbon):  Optical Serial Link Board oSLB replaces single copper link to current Regional Calorimeter Trigger (RCT – U. Wisc.) with two optical links (WBS )  One optical link to optical Receiver Module (oRM) on RCT (WBS ) o Replaces copper Receiver Module  One optical link to input of Upgraded Calorimeter Trigger: CTP7 Card  574 oSLB’s & oRM’s with Fibers installed in 2014  French Contribution to Calorimeter Trigger (no US Upgrade Funds)  Presently oSLB & oRM in final prototype stage  oSLB:oRM: Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 16 HCAL & ECAL Trigger Primitives

LHC CMS Detector Upgrade Project Calo. Trigger Upgrade in Parallel: Split inputs from ECAL & HCAL HCAL Energy (HTR) ECAL Energy (TCC) Regional Calo Trigger Global Calo Trigger EM candidates Region energies HF Energy (μHTR) HCAL Energy (μHTR) Layer 1 Calo Trigger Layer 2 Calo Trigger Current L1 Trigger System Upgrade L1 Trigger System oSLB oRM  Install optical SLB and optical RM mezzanines during LS1  Install HCAL optical splitters during LS1 or YETS  Install HCAL backend μHTR cards for input to new trigger  Install HCAL frontend electronics after LS2 (finer long. segmentation) Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 17 US UK US UK HCAL Optical Splitters Stage HCAL uHTR & assoc. Cal. Trigger Cards

LHC CMS Detector Upgrade Project : Calorimeter Trigger Processor Virtex-6 Prototype Board (CTP-6) – U. Wisconsin Back End FPGA XC6VHX250T/ XC6VHX380T Front End FPGA XC6VHX250T/ XC6VHX380T Avago AFBR- 810B Tx Module 4X Avago AFBR-820B Rx Module MMC Circuitry JTAG/USB Console Interface Mezzanine Power Modules Dual SDRAM for dedicated DAQ and TCP/IP buffering 12x Multi Gig Backplane Connections Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade total + 8 spares + 2 test setups = 46

LHC CMS Detector Upgrade Project : Calorimeter Trigger Processor Final Version: Virtex-7 (CTP-7) – U. Wisconsin Replace 2 Virtex-6s with a Virtex 7 for processing+ZYNQ for embedded TCP/IP endpoint 30A, 1V power module for FPGA logic core 3x CXP Pluggable modules for 36 Tx + 36 Rx 10G optical links 2x AFBR-820 modules for 24 Rx 10G optical links Simpler design to execute than the CTP-6 36 total + 8 spares + 2 test setups = 46 Virtex-7 VX690T FPGA ZYNQ XC7Z03 0 EPP 1.5V Supply 2.5V Supply 3.3V Supply 1V 30A Supply CXP Module 12Tx + 12 Rx CXP Module 12Tx + 12Rx CXP Module 12Tx + 12 Rx 12X Rx 12X Rx (CTP-6 CAD View) Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 19

LHC CMS Detector Upgrade Project : Crosspoint IO Card: provides crate interconnections – U. Wisconsin Controller (MMC and link mgmt) 4X Avago AFBR-79EQDZ QSFP+ Module Positions 4x4 Lane Bidirection al Multi Gig Backplane Connectio ns Backplane Rx/Tx Redriver ICs (top and bottom sides) Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 20 (2/crate x 3 crates = 6 + spares)

LHC CMS Detector Upgrade Project : CTP Infrastructure: Vadatech VT894 Crate Test Setup BU AMC13 Vadatech MCH UW CTP-6 TTC Downlink UW Aux Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 21 (Final system: 3 crates w/ 12 CTP7 ea. + 2 test setups + spare = 6) U. Wisconsin designed backplane with dense card interconnects manufactured & installed in commercial Vadatech VT892 Crate available in Vadatech Catalog

LHC CMS Detector Upgrade Project  Goal (Original):  Provide readout of original RCT descoped during construction project  Present readout through GCT input buffer not workable with trigger evolution  Uses connection to a single CTP6 prototype or CTP7 card for DAQ readout  optical Receiver Summary Card (oRSC) paid by DOE Nuclear  VME Slave Interface Card  Fits in current RCT Crates (1 per crate)  18 Cards in 2015 System  Receives RCT Jet Sum Card Output to GCT on Copper “SCSI” Data Cables  Provides direct optical input to GCT o Bypassing old optical conversion cards  Planned use for Heavy Ion Triggers  Prototype under test   Multiple optical outputs provide:  Inputs for upgrade calorimeter trigger & parallel operation of old & new trigger wherever ECAL & HCAL electronics not available in 2015 Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 22 Calorimeter Trigger Risk Mitigation

LHC CMS Detector Upgrade Project Evolution: Partial Upgrade in 2015 Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 23 Layer 2 Calo Trigger HCAL energy Regional Calo Trigger Global Calo Trigger HF energy oRSC Layer 2 Calo Trigger oRM ECAL energy Layer 1 Processors oSLB ECAL & HF Clusters Half-tower position 4x4 E+H Clusters 2x1 E+H Clusters 2-bit region ID Layer 2 Processors Muon Jets & Sums eGammaTau Heavy Ion Spare RCT RO + Testing GCT fallback and Minimal Stage-1 fallback remains available With just a fraction of final cards or even prototypes, derive many benefits of full upgrade, incl. muon isolation Uses reprogrammed RCT clusters with improved algorithms in Layer-2 Also, brings in all the power of finer grain HF using “Slice Test” of Layer-1 & 2 Improved PU subtraction, Isolation Calculation (e/γ/τ/μ) & Half-tower Position Resolution US Items: oRSCs, oSLB/oRM commissioning CTP Layer 1 Processors RCT RO+Testing CTP UK Items: MP7 Layer 2 Processors 36  HTRs 2 CTP7s  9 CTP7s Add to bring in finer grain EM clusters 2 CTP7s 4 MP7s

LHC CMS Detector Upgrade Project  CTP7 M&S: no HB/HE Backend until 2016  Delay purchase of 34 CTP7 cards until FY15  Use 12 CTP7 Cards w/prototypes as additional spares & for test setups.  Connected as shown on previous slide  EMU CSCTF M&S: one endcap in 2015 & other in 2016  Commission & test one endcap thoroughly so 2 nd can be integrated quickly  Delay spares & use prototypes for spares, test setups  Compatible with Global Muon Trigger using different Endcap inputs  Stages and Content: Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 24 Trigger Staging

LHC CMS Detector Upgrade Project  Prototypes for all major cards exist & have passed tests  Upgrade is based upon common μTCA hardware platform also used by HCAL & other CMS systems  Upgrade trigger is replacing existing trigger hardware built by the same team → interfaces well understood  Same team wrote software & firmware for existing trigger system → scope & requirements well understood  Requirements are frozen in approved Trigger TDR Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 25 Technical Feasibility

LHC CMS Detector Upgrade Project Trigger L2 Manager: Wesley Smith Deputies: Darin Acosta, Sergo Jindariani Muon Trigger L3 Manager: Darin Acosta Muon Port Card & Sorter: Paul Padley Muon Track-Finder: Ivan Furic Calorimeter Trigger L3 Manager: Sridhara Dasu CERN Integration: Pam Klabbers Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 26 Trigger Project Organization

LHC CMS Detector Upgrade Project  W.S. (U. Wisconsin)  CMS Trigger Project Manager ,  Trigger Coordinator 2007 – 2012  US CMS L2 Trigger Project Manager 1998 – present  Darin Acosta (U. Florida)  CMS Trigger Project Manager  EMU Track-Finder, US CMS Trigger, present  Sridhara Dasu (U. Wisconsin)  US CMS L3 Manager for Calorimeter Trigger, 1998 – present  Author of original & upgrade cal. trig. Algorithms 1994 – present  Pam Klabbers (U. Wisconsin)  CMS Regional Calorimeter Trigger Operations Manager (more than a decade on RCT project)  CMS Deputy Trigger Technical Coordinator  Ivan Furic (U. Florida)  CMS CSC muon trigger maintenance & operations  CMS CSCTF Upgrade leader  Paul Padley (Rice U.)  US CMS EMU Project Manager  EMU Port Card, Clock Card, Sorter, 1998 – present Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 27 Trigger Management CMS Experience

LHC CMS Detector Upgrade Project  Tom Gorski (U. Wisconsin) – Calorimeter Trigger  Over a decade of engineering on the CMS Calorimeter Trigger  Alex Madorsky (U. Florida) – Muon Track-Finder  Over a decade of engineering on CMS Trigger, EMU, CSCTF  Mike Matveev (Rice U.) – Muon Port Cards & Sorter  Over a decade of engineering on CMS Trigger, EMU, Port Cards, Sorter  Mathias Blake (U. Wisconsin) – Cal. Trig. Firmware  New to project  5 years Applications Engineer at Xilinx, 2 years FPGA Engineer at DRS Technologies, 3 years FPGA Design Engineer at Dolby Labs Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 28 CMS Trigger Engineering Experience

LHC CMS Detector Upgrade Project  M&S:  Parts based on quotes & prototype purchases  Ordering times based on experience with prototype orders  Board cost & delivery times based on prototype boards  Assembly cost & delivery times based on prototype boards  EDIA:  Labor Hours based on experience with corresponding tasks done for prototypes and on experience in delivering the original CMS trigger system (done by the same people & groups).  Labor costs based on actual labor rates of all of the individuals who are now working on the project  Contingency:  Follows US CMS Contingency rules Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 29 C&S Methodology

LHC CMS Detector Upgrade Project Fermilab Director's Review -- US CMS Trigger Upgrade Base Cost & Contingency in AY K$ 30Wesley Smith, 16 July Cost Estimate with Contingency

LHC CMS Detector Upgrade Project Fermilab Director's Review -- US CMS Trigger Upgrade 31 Schedule/Milestones Wesley Smith, 16 July 2013  The project completes at the end of FY16  FY14:  Launch production of trigger electronics 1QFY14  Deliver on-detector muon trigger electronics to CERN: 3QFY14  Deliver Stage 1 calorimeter layer 1 & muon track-finder electronics to CERN: 4QFY14  FY15:  Test operation of trigger calorimeter and muon trigger electronics*: 2QFY15 (*except muon sorter)  Deliver Stage 2 calorimeter layer 1 & muon track-finder electronics to CERN: 2QFY15  Deliver Muon Sorter electronics to CERN: 3QFY15  Test Operation of Muon Sorter electronics: 4QFY15  FY16:  Test high luminosity capability of endcap muon trigger electronics: 4QFY16  Test high luminosity capability of calorimeter trigger electronics: 4QFY16

LHC CMS Detector Upgrade Project Fermilab Director's Review -- US CMS Trigger Upgrade 32 Base Cost = AY $K (No Contingency) Wesley Smith, 16 July : Trigger Resource Profile

LHC CMS Detector Upgrade Project  Senior Engineer becomes unavailable  Hire new engineer, subcontract to consulting firm, use FNAL engineer  Funding is delayed  Commission with prototypes and/or fewer production boards  Software or Firmware does not meet requirements  Hire extra expert effort to recover schedule and help personnel  Boards are delayed (design, manufacture or testing)  Hire extra effort to speed up testing schedule  Vendor non-performance  Acquire spending authority to use alternative vendors (while original funds are being unencumbered).  Input or output electronics (non-trigger) delayed  Use built in capabilities of trigger electronics to provide signals for their own inputs and outputs Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 33 Trigger Risks

LHC CMS Detector Upgrade Project  C&S understood since prototypes for all major cards have been built & have passed tests  C&S based on experience of the same team that built original CMS calorimeter & muon trigger systems  Hardware is based upon common μTCA hardware platform also used by HCAL & other CMS systems  Same team wrote software & firmware for existing trigger system → scope & requirements well understood  Requirements are frozen in approved Trigger TDR  Schedule is robust because it always provides a working trigger at each stage with parallel operation of old & new trigger systems, commissioning & testing of new system during data-taking (with data) & evolution to final system while providing immediate availability of improved trigger in 2015 Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 34 Cost & Schedule Understanding

LHC CMS Detector Upgrade Project  Trigger Upgrade is technically feasible since based on working prototypes, & built by the same team that built original CMS Calorimeter & Muon triggers.  Trigger Upgrade can be accomplished within Cost & Schedule since based on working prototypes and experience in building the original CMS Calorimeter & Muon triggers.  Risks are understood and familiar due to team experience  Plan to complete prototyping and begin production when approved. Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 35 Conclusions

LHC CMS Detector Upgrade Project BACKUP Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 36

LHC CMS Detector Upgrade Project Calorimeter Trigger Components Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 37 TypeNamePurpose TCCTrigger Concentrator CardExisting ECAL trigger primitives (TP) (FR) oSLBOptical Serial Link BoardTransmit ECAL TPs to RCT & CTP (PT)  HTR HCAL Trigger & DAQ BoardCreate and transmit HCAL TPs RCTRegional Calorimeter TriggerExisting trigger system oRMOptical Receiver ModulesReceive ECAL TPs on RCT oRSCOptical Regional Summary CardConvert and transmit RCT output CTPCompact Trigger ProcessorLarge FPGA w/ optical & backplane CIOXCompact IO cross-point switchIntra-crate sharing switch MP7Multipurpose ProcessorLarge FPGA w/ only optical links (UK)

LHC CMS Detector Upgrade Project TMT Calorimeter Trigger Upgrade Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 38 Time-multiplexed architecture, calo data from an entire event processed by a single processor

LHC CMS Detector Upgrade Project Crate G Up to 12 Trigger Path Modules Trig Path 1 (MP7) 2x288-fiber patch panel (2x24×12-fiber optical ribbons in & out) Crate BCrate C CTP CIO-U CIO-L 6 x 8-fiber ribbons for Lateral Network 4.8 Gbps) Trig Path 2 (MP7) Trig Path 3 (MP7) Trig Path 4 (MP7) Trig Path 5 (MP7) Trig Path 6 (MP7) HCALECALHCALECALHCALECAL Each 1 st -Level CTP7 Drives 2 fibers out to 6-12 Trigger 9.6 Gbps Each Trigger Path MP7 rcvs 48 total 1 st -Level 9.6 Gbps (2/region) To GT CTP CIO-L CTP CIO-U CIO-L CIO-U Crate A  Crate G Pipelined Calorimeter Trigger Upgrade Calorimeter Trigger Processor (CTP) Card Crosspoint I/O (CIO) Card Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade 39 Each Layer 2 processor calculates suite of trigger paths Layer 1 sends dedicated clusters to Layer 2