LHC CMS Detector Upgrade Project P04 – Trigger Upgrade, Wesley Smith, U. Wisconsin L2 Manager, WBS August 5, 2014 Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 1

LHC CMS Detector Upgrade Project  Final Design  Requirements, Performance  Upgrade Scheme  Muon Trigger Upgrade and transition  Calorimeter Trigger Upgrade and transition  KPP  Project Organization  Organizational Structure and WBS  Baseline Cost and Schedule  Risk, ESH&Q  Summary Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 2 Outline

LHC CMS Detector Upgrade Project Final Design Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 3

LHC CMS Detector Upgrade Project  LHC beams cross every 25 ns with ~ 23 interactions → 0.5 MB of data, but can only store ~ 1 kHz of 0.5 MB data → Trigger  Trigger rates are driven by the increase in luminosity, the center- of-mass energy, and by higher Pile-Up (PU)  Detector readout systems limit Level-1 trigger output rate to 100 kHz, latency to 4 μsec.  Mitigate by improving e/γ isolation, τ id, μ p T resolution, μ isolation, jets with PU subtraction and L1 menu sophistication  Increase system flexibility with high bandwidth optical links and large FPGAs, and standardize on μTCA telecom standard  Build and commission upgrade in parallel with current trigger system to safeguard physics, decouple from LHC schedule  Overall design:  Two-layer calorimeter trigger with tower-level precision and PU subtraction  Integrated muon trigger combining all CSC, DT and RPCs in track-finding Director's CD-23 Review -- Trigger Upgrade 4 Requirements and Design Summary Wesley Smith, 5 August 2014

LHC CMS Detector Upgrade Project  Physics priorities:  Measure all Higgs BR precisely to confirm Standard Model or not ⇒ retain or improve current trigger capability critical  Want to be able to answer question of naturalness - whether or not there is new physics stabilizing Higgs mass o SUSY remains a leading candidate, but if so, must have light stops o Also must be able to trigger on and search for all variants (e.g. RPV with all hadronic final states) to draw a firm conclusion  Studied a set of benchmark physics channels  Looked at performance of these channels in 2012 analyses with and without L1 trigger upgrade at different luminosities o Based on a simplified trigger menu with a total rate < 100 kHz  Results for luminosities up to 2.2 x cm –2 s –1  Acceptance Improvements > x 2 in some channels for same rate  Average Improvement 40%  Results summarized in breakout sessions Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 5 Upgrade Trigger Physics Performance

LHC CMS Detector Upgrade Project Final Overall Upgrade Scheme Director's CD-23 Review -- Trigger Upgrade 6Wesley Smith, 5 August 2014 US CMS Detector Upgrade Project ↑HCAL Upgrade↑ Trigger Upgrade ↓Trigger Upgrade↓ (Object Finding) (Cluster Finding) (Object Finding) (Cluster Finding) MTF7 MTF7 (MS FW) MP7 (UK) CTP7 Vienna

LHC CMS Detector Upgrade Project  Install fully parallel and higher bandwidth optical path for CSC  New CSC Muon Port Cards mezzanine cards installed in cavern during LS1 o Prerequisite done by Operations Program  Alleviates bottleneck & sends all segments from each CSC (robustness to PU and collimated signals)  Build up new Track Finder in 2015 and commission in parallel, ready by 2016 Muon Trigger Transition Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 7 Old systemNew system Cavern Counting Room

LHC CMS Detector Upgrade Project EMU Trigger Upgrade CavernCounting Room μTCA: Advanced Mezzanine Cards from Telecommunications Computing Architecture (commercial telecommunications hardware) Director's CD-23 Review -- Trigger Upgrade 8 Wesley Smith, 5 August MPC-EMUTF Optical Fibers* (Rice) Muon Port Card Mezzanine* (Rice) Endcap Muon Track-Finder (U. Florida) EMUTF Infrastructure (U. Florida) : Muon Sorter (Rice) Trigger MotherBoard Data MotherBoard Clock&Control Sector Processors Clock & Control Muon Sorter *M&S on FY13 Ops. Prog. VME MTF7 (MP7)

LHC CMS Detector Upgrade Project : EMU Track-Finder (U. Florida) Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 9 Back: Core FPGA card ⇒ ⇐ Front: Optics card  Muon Track Finder card: MTF7  13 needed + 4 spares + 3 test setups = 20  Optimized for maximum input from muon detectors (84 input links, 28 output links)  Dual card w/large capacity for RAM (~1GB) to be used for p T assignment in track finding Pre-production Prototypes undergoing testing, integration at CERN

LHC CMS Detector Upgrade Project  MTF7 Testing  IBERT programmed for 80 channels o 10 Gbps async o 9.6 Gbps sync  Synchronous communication tested w/clock frequency fluctuating +/- 2kHz from MHz  Testing in loopback mode via fibers o 1 and 50 meters o 28 channels at a time (transmitter count limit) o Also tested 1.6, 3.2, 6.4 sync, 5.0 async  Results: o All channels BER < o Error free for hours  Integration at CERN  Test: MPC→MTF7, MTF7→MP7 (GMT) o PRBS o Random data via test FIFOs o Latest MPC data format used  Result: No errors. D. Acosta, 5-7 Aug 2014 CD-23 Review -- Muon Trigger Overview 10 EMU Tests: Board,

LHC CMS Detector Upgrade Project Calo. Trigger Upgrade in Parallel: Split inputs from ECAL and HCAL HCAL Energy (HTR) ECAL Energy (TCC) Regional Calo Trigger Global Calo Trigger EM candidates Region energies HF Energy (μHTR) HCAL Energy (μHTR) Layer 1 Calo Trigger Layer 2 Calo Trigger Current L1 Trigger System Upgrade L1 Trigger System oSLB oRM  ECAL: optical Serial Link Board (OSLB) and optical Receiver Mezzanines (oRM) connect to Present and Upgrade Calorimeter Trigger (*Details of evolution in breakout). o All installed and under test  HCAL: optical splitters drive both HTRs and μHTRs Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 11 US UK US UK HCAL Optical Splitters 2016: HCAL uHTR & assoc. Cal. Trigger Cards (Cluster Finding) (Object Finding) 2015: ECAL, Legacy RCT & associated Cal. Trigger Cards*

LHC CMS Detector Upgrade Project Upgrade Calorimeter Trigger Layer 1 and 2 Hardware US: UK: Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 12 Vienna: (Cluster Finding) (Object Finding)

LHC CMS Detector Upgrade Project : Cal. Trig. Processor Preproduction Prototype – U. Wisconsin 12x Multi Gig Backplane Connections Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade total + 8 spares + 2 test setups = 46 Virtex-7 VX690T FPGA ZYNQ XC7Z03 0 EPP CXP Module 12Tx + 12 Rx CXP Module 12Tx + 12Rx CXP Module 12Tx + 12 Rx Minipods: 31Rx + 12Tx JTAG/USB Console Interface Mezzanine CTP7: Virtex-7 690T for processing ZYNQ for TCP/IP + Linux 67 10Gbps optical Input links 48 10Gbps optical Output links Status: Two Tested 6 more in production

LHC CMS Detector Upgrade Project  2 units built  One at CERN, one at UW for development and integration  6 additional pre-production boards being assembled now o Two back this week at UW, remainder in next 2 weeks  Excellent results with processor system, power supplies, clock system and optical/backplane links  Both CPLLs and QPLLs working very well within their intended speed range, see no problems whatsoever operating CPLLs at 4.8 & 6.4 & 9.6 Gbps synchronous and as well at 10 Gbps asynchronous  Linux environment in ZYNQ chip fuly validated and being actively used.  CERN Integration tests w/ECAL, HCAL, Layer 2 (MP7) completely successful  Including event playback, capture and multiplexed transmission  Multi-day integration tests with oRSC in setup at CERN with no errors: CTP7 Status and Testing oRSC 1 oRSC 2 oRSC 1 Patch Panel MP7 1 CTP7 1 5 Ch 18 Ch 23 Ch 36 Ch 10 Ch Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 14 oRSC MP to CTP7 CXP

LHC CMS Detector Upgrade Project  After full testing at institute, shipped to CERN  All tests recorded (of all types) for individual boards in database  Tests use and validate software and firmware test release  Acceptance Testing in Electronics Integration Center (EIC) at CERN  Individual labs for CSC and Calorimeter Trigger  Boards retested to validate institute test results  Tests use software and firmware test release  Integration Testing in EIC  Row of racks with DAQ, Trigger, Central Clock, Crates of other subsystem electronics  Operation of a vertical slice with electronics to be tested installed.  Tests use and validate software and firmware commissioning release  Integration Testing at P5: Global Runs/Parallel Operation  Test with all CMS with cosmics when beam not running/with beam when running  Electronics installed in final locations with final cables  Full-scale tests with full CMS DAQ/Trigger/Clocking  Tests use software and firmware commissioning release  Post commissioning Operations at P5: Global Runs/Parallel Operation  After testing completes, continue with Global Runs/Parallel Operation  Validate software and firmware initial operational release Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 15 QA/QC: Testing and Validation

LHC CMS Detector Upgrade Project  Threshold:  Delivery, installation and validation, based on test data patterns, of Layer 1 of the upgrade calorimeter trigger and the upgrade endcap muon trigger. Demonstration of 98% agreement between the installed upgrade trigger electronics at CERN and trigger emulation of this electronics using test data patterns, followed by a factor of two reduction of calorimeter and endcap muon trigger rates for electrons, photons, muons and taus with respect to the present system with less than 15% efficiency loss, demonstrated using trigger emulation on data taken after LS1. Incorporation of additional CSC chamber data into the endcap muon trigger logic.  Objective:  Delivery, installation and validation, based on test data patterns, of Layer 1 of the upgrade calorimeter trigger and the upgrade endcap muon trigger. Demonstration of 99.5% agreement between the installed upgrade trigger electronics at CERN and trigger emulation of this electronics using test data patterns, followed by a factor of two reduction of calorimeter and endcap muon trigger rates for electrons, photons, muons and taus with respect to the present system with less than 10% efficiency loss, demonstrated using trigger emulation on data taken after LS1. Incorporation of additional CSC chamber data into the endcap muon trigger logic. Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 16 Key Performance Parameters NB: Separated from beam schedule

LHC CMS Detector Upgrade Project Project Organization Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 17

LHC CMS Detector Upgrade Project Wesley Smith, 5 August Trigger Organization Chart Muon Port Card & Sorter: L4 Manager: Paul Padley CERN Integration: L4 Manager: Pam Klabbers Muon Track-Finder: L4 Manager: Ivan Furic CD-23 Review -- Trigger Upgrade Ivan Furic

LHC CMS Detector Upgrade Project  W.S. (U. Wisconsin)  CMS Trigger Project Manager ,  Trigger Coordinator 2007 – 2012  Trigger Performance and Strategy Working Group present  US CMS L2 Trigger Project Manager 1998 – present  Darin Acosta (U. Florida)  CMS Trigger Project Manager 2012-present  EMU Track-Finder, US CMS Trigger, present  Sridhara Dasu (U. Wisconsin)  US CMS L3 Manager for Calorimeter Trigger, 1998 – present  Author of original and upgrade cal. trig. Algorithms 1994 – present  Pam Klabbers (U. Wisconsin)  CMS Regional Calorimeter Trigger Operations Manager (more than a decade on RCT project)  CMS Deputy Trigger Technical Coordinator  Ivan Furic (U. Florida)  CMS CSC muon trigger maintenance and operations  CMS CSCTF Upgrade leader  Paul Padley (Rice U.)  US CMS EMU Project Manager  US CMS Deputy L1 Operations Manager, present  EMU Port Card, Clock Card, Sorter, 1998 – present Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 19 Trigger Mgm’t CMS Experience Over 75 years of combined CMS Trigger Management Experience!

LHC CMS Detector Upgrade Project  Tom Gorski (U. Wisconsin) – Calorimeter Trigger  Over a decade of engineering on the CMS Calorimeter Trigger  Alex Madorsky (U. Florida) – Muon Track-Finder  Over a decade of engineering on CMS Trigger, EMU, CSCTF  Mike Matveev (Rice U.) – Muon Port Cards and Sorter  Over a decade of engineering on CMS Trigger, EMU, Port Cards, Sorter  Ales Svetek (U. Wisconsin) – Cal. Trig. Firmware  New to project  4 years ATLAS Beam Conditions Monitor Firmware, DAQ, Commissioning, Detector Operations Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 20 CMS Trig. Engineering Exp. > 40 years of engineering experience, >30 on CMS Trigger

LHC CMS Detector Upgrade Project 21 Muon Trigger WBS Detail Wesley Smith, 16 July 2013 Fermilab Director's Review -- US CMS Trigger Upgrade → Calorimeter Trigger M&S on Ops. Prog. Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade

LHC CMS Detector Upgrade Project Calorimeter Trigger WBS Detail Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 22 ← Muon Trigger M&S paid by CMS-France

LHC CMS Detector Upgrade Project Schedule Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 23

LHC CMS Detector Upgrade Project Trigger L2 Schedule/Milestones Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 24  Targeting completion of L2 milestones 6 months earlier MPCM connection Prepro d Prod Prototype Soft/Firmware Commission Muon Trigger Preprod Prod Soft/Firmware Commission Prototype Calorimeter Trigger Calorimeter Connection L1T

LHC CMS Detector Upgrade Project Trigger Schedule/Milestones L4 Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 25

LHC CMS Detector Upgrade Project Critical Path – Muon Trigger Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 26

LHC CMS Detector Upgrade Project Critical Path – Calorimeter Trigger Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 27

LHC CMS Detector Upgrade Project Cost Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 28

LHC CMS Detector Upgrade Project Trigger Cost Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 29 Labor Breakdown Total

LHC CMS Detector Upgrade Project  Muon Trigger: 2.3M + 0.7M contingency = 3.0M  Calorimeter Trigger: 3.0M + 0.7M contingency = 3.7M Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 30 Trigger Cost Cal./Mu. Breakdown

LHC CMS Detector Upgrade Project Trigger Labor Profile Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 31

LHC CMS Detector Upgrade Project Trigger Cost Profile Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 32 M&S + Labor: Total + Estimate Uncertainty:

LHC CMS Detector Upgrade Project Trigger Cost Profile by Cal./Mu. Trig. Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 33

LHC CMS Detector Upgrade Project  C&S understood since pre-production prototypes for all major cards have been built and have passed tests  Have vendor quotes for all parts  C&S based on experience of the same team that built and wrote software and firmware for existing trigger system  Hardware is based upon common μTCA hardware platform also used by HCAL and other CMS systems  Same team wrote software and firmware for existing trigger system → scope and requirements well understood  Requirements are frozen in approved Trigger TDR  Schedule is robust because it always provides a working trigger at each stage with parallel operation of old and new trigger systems, commissioning and testing of new system during data-taking (with data) and evolution to final system while providing immediate availability of improved trigger in 2015 Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 34 Maturity of Estimate

LHC CMS Detector Upgrade Project Maturity of Design Detail Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 35

LHC CMS Detector Upgrade Project Risk Analysis Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 36

LHC CMS Detector Upgrade Project  Plus an additional 20 threats (including same risk applied to different RLS activities)  Overall Risk Mitigation Strategy:  Continue to provide fully operational current trigger system in parallel with upgrade commissioning  Partial operation of the upgrade systems provide tangible benefits from 2015 onwards Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade Trigger – top cost risks

LHC CMS Detector Upgrade Project Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade Trigger: Cost impact of all threats Total cost impact at 90% C.L.

LHC CMS Detector Upgrade Project  Senior Engineer becomes unavailable (Low Risk)  Hire new engineer, subcontract to consulting firm, use FNAL engineer  Funding is delayed (Low Risk)  Commission with prototypes and/or fewer production boards  Software or Firmware does not meet requirements (Low Risk)  Hire extra expert effort to recover schedule and help personnel  Boards are delayed (design, manufacture or testing) (Low Risk)  Hire extra effort to speed up testing schedule  Vendor non-performance (Low Risk)  Acquire spending authority to use alternative vendors (while original funds are being unencumbered).  Input or output electronics (non-trigger) delayed (Low Risk)  Built in capabilities of trigger electronics provide signals for their own inputs & outputs Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 39 Overall Trigger Risks & Mitigation

LHC CMS Detector Upgrade Project  Safety: follows procedures in CMS-doc-11587, FESHM  L2 Manager (W.S.) responsible for applying ISM to trigger upgrade. o Under direction of US CMS Project Management.  Modules similar to others built before, of small size and no high voltage  Quality Assurance: follows procedures in CMS-doc  Regularly evaluate achievement relative to performance requirements and appropriately validate or update performance requirements and expectations to ensure quality. Evaluation at CERN in EIC, Global Runs, Parallel Operation (earlier slide)  Quality Control: also follows procedures in CMS-doc  Equipment inspections and verifications; Software code inspections, verifications, and validations; Design reviews; Baseline change reviews; Work planning; and Self- assessments.  All modules have hardware identifiers which are tracked in a database logging QC data through all phases of construction, installation, operation and repair.  Graded Approach:  Apply appropriate level of analysis, controls, and documentation commensurate with the potential to have an environmental, safety, health, radiological, or quality impact.  Four ESH&Q Risk levels are defined and documented in CMS-doc Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 40 Trigger ESH&Q

LHC CMS Detector Upgrade Project  None  Since both muon and calorimeter triggers have full self-test capabilities, both threshold and objective KPP are satisfied w/o requiring connected inputs or outputs.  Trigger electronics can store up sequences of test patterns and inject them into the front end of the trigger electronics at speed  Trigger electronics can receive its output, process and record this at speed for subsequent readout by DAQ. Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 41 External Dependencies

LHC CMS Detector Upgrade Project Summary Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 42

LHC CMS Detector Upgrade Project  Trigger Upgrades meet technical performance requirements  Designs are mature with pre-production boards tested.  Including detailed integration testing with all interfacing systems at CERN  Scope and Specifications of Trigger Upgrade are sufficiently well- defined in TDR to support the C&S estimates  Risk managed by parallel operation of old and new trigger systems, commissioning and testing of new system during data-taking (with data)  Upgrade based upon common μTCA hardware platform and components used by other CMS systems  ES&H, QA/QC plans, C&S based on experience with working, tested pre-production prototypes  Management and Engineering teams are experienced with sufficient design skills, having designed and built original CMS trigger.  Trigger Upgrade not on Project Critical Path  Ready for CD-2/3 Wesley Smith, 5 August 2014 Director's CD-23 Review -- Trigger Upgrade 43 Conclusions

LHC CMS Detector Upgrade Project Additional Slides Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 44

LHC CMS Detector Upgrade Project  Calorimeter Trigger Upgrade Documentation:   Muon Trigger Upgrade Documentation:  Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 45 Documentation

LHC CMS Detector Upgrade Project Calorimeter Trigger Components Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 46 TypeNamePurpose TCCTrigger Concentrator CardExisting ECAL trigger primitives (TP) (Paid by FR) oSLBOptical Serial Link BoardTransmit ECAL TPs to RCT & CTP (Paid by PT)  HTR HCAL Trigger & DAQ BoardCreate and transmit HCAL TPs RCTRegional Calorimeter TriggerExisting trigger system oRMOptical Receiver ModulesReceive ECAL TPs on RCT (Paid by FR) oRSCOptical Regional Summary CardConvert and transmit RCT output (paid by DOE Nuclear) CTPCalorimeter Trigger ProcessorLarge FPGA w/ optical & backplane CIOXCompact IO cross-point switchIntra-crate sharing switch MP7Multipurpose ProcessorLarge FPGA w/ only optical links (UK)

LHC CMS Detector Upgrade Project Muon Trigger Components Wesley Smith, 5 August 2014 CD-23 Review -- Trigger Upgrade 47 TypeNamePurpose MPCMuon Port CardExisting CSC trigger primitives (TP) MPCMMuon Port Card MezzanineAdds new CSC TPs to existing MTFMuon Track FinderFit and assign angle, P T to track MSMMuon Sorter MezzanineAdds upgrade Sorter to existing MS- EMUTF Muon Sorter firmware used on EMU Track-Finder Card Muon sorter function by reprogramming MTF7 card