BO1-1 Calorimeter Trigger W. H. Smith, L3 Manager, Calorimeter Trigger, Director’s Review Draft January 21, Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 1

 WBS definition  Basis of Estimate  Schedule  Cost and Labor Profiles  Risk and Contingency  R&D status and plans  ES&H and QA  Summary 2 Outline Phase 2 Calorimeter Trigger Upgrade21-Jan-2016 W. Smith

X Organization Chart to L3 (L4 for tracker) Phase 2 Calorimeter Trigger Upgrade21-Jan-2016 W. Smith Trigger Jeff Berryhill (FNAL) Calorimeter Trigger Wesley Smith (UW) Muon Trigger Darin Acosta (UF) Track Correlator Rick Cavanaugh (UIC)

 W.S. (U. Wisconsin) – US CMS HL-LHC L3 Calorimeter Trigger Project Manager  CMS Trigger Project Manager ,  Trigger Coordinator 2007 – 2012  Trigger Performance and Strategy Working Group  US CMS L2 Trigger Project Manager (construction and operations) 1998 – present  US CMS Phase 1 Upgrade L2 Trigger Project Manager 2013 – present  Sridhara Dasu (U. Wisconsin)  US CMS L3 Manager for Calorimeter Trigger (construction and operations) 1998 – present  US CMS L3 Manager for Phase 1 Calorimeter Trigger Upgrade 2013 – present  Author of original and upgrade cal. trig. Algorithms 1994 – present  Pam Klabbers (U. Wisconsin) – Cal. Trig. On-site Manager  CMS Regional Calorimeter Trigger Operations Manager (more than a decade on RCT project)  CMS Deputy Trigger Technical Coordinator  Jeff Berryhill  CMS & US CMS Phase-1 Stage-1 Calorimeter Trigger Project Manager  Tom Gorski (U. Wisconsin) – Cal. Trig. Electrical Engineer – Lead Engineer  Over a decade of engineering on the CMS Calorimeter Trigger  Delivered final phase of original Regional CMS Calorimeter Trigger  Delivered Phase 1 Layer-1 Calorimeter Trigger Upgrade Electronics  Ales Svetek (U. Wisconsin) – Cal. Trig. Firmware Engineer  3 years on Phase 1 Calorimeter Trigger Upgrade Firmware  (4 years ATLAS Beam Conditions Monitor Firmware, DAQ, Commissioning, Detector Operations)  Marcelo Vicente (U. Wisconsin) – Cal. Trig. Firmware Engineer  3 years on Phase 1 Calorimeter Trigger Upgrade Firmware + HCAL Firmware  2 Years on ECAL Phase 1 Upgrade Trigger Primitive Generation Electronics (oSLB, oRM)  Jes Tikalski (U. Wisconsin) – Cal. Trig. Software Engineer  3 years on Phase 1 Calorimeter Trigger Upgrade Software and embedded systems 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 4 CMS Calorimeter Trigger Personnel

WBS Definition 5 Phase 2 Calorimeter Trigger Upgrade21-Jan-2016 W. Smith

Level-1 Trigger Architecture 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 6 U.S. covers fraction (mostly regional) Calorimeter Regional Trigger Calorimeter Global Trigger

 Calorimeter Trigger Management  Calorimeter Trigger Milestones, Interfaces  Calorimeter Trigger Travel  Regional Calorimeter Trigger  Regional Calorimeter Trigger M&S (Detail Next Slide)  Regional Calorimeter Trigger Engineering  Regional Calorimeter Trigger Technical Work  Regional Calorimeter Trigger FW  Regional Calorimeter Trigger SW  Global Calorimeter Trigger  Global Calorimeter Trigger M&S  Global Calorimeter Trigger Engineering  Global Calorimeter Trigger Technical Work  Global Calorimeter Trigger FW  Global Calorimeter Trigger SW  Calorimeter Trigger Infrastructure  Crates and Power Supplies M&S  Cables, Fibers and Patch Panel M&S  Test Facilities M&S  Infrastructure Engineering  Infrastructure Technical Work 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade WBS: Calorimeter Trigger I

 X, (X=1,2) (Regional, Global) Calorimeter Trigger M&S  X.1 Cal. Trig. Preproduction Optics  X.2 Cal. Trig. Preproduction FPGAs  X.3 Cal. Trig. Preproduction Misc. Comp.  X.4 Cal. Trig. Preproduction PCB Fabrication  X.5 Cal. Trig. Preproduction Assembly  X.6 Cal. Trig. Optics  X.7 Cal. Trig. FPGAs  X.8 Cal. Trig. Misc. Comp.  X.9 Cal. Trig. PCB Fabrication  X.10 Cal. Trig. Assembly 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 8 Calorimeter Trigger M&S Detail

 EB/EE/HB/HE: Process individual readout granularity cells from Calorimeter Back-end (Trigger Primitive Generation – TPG) electronics to be optimally matched with track trigger information  Produce Tau, Jet, e/γ clusters….  New Endcap calorimeter TPG electronic produces clusters with Tau, Jet, e/γ topologies which are then processed for optimal matching with track trigger info.  Data processed by input Regional Layer and then final Global Layer providing the output. Similar to current calorimeter trigger, essentially scaled to higher number of channels involved.  Tasks: Isolation, duplicate removal, boundaries, global energy sums  Produces/refines candidate objects/clusters to send to the different track correlator processors o Logic is based on adaptation of Particle Flow ideas to L1T o Different correlators for muons, e/γ, Tau, Jet….  Also provides stand-alone calorimeter trigger 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 9 Calorimeter Trigger Design

Crate BCrate CCrate A Processor Track Correlator …… Regional Processing: Global Processing: Model for L1 Cal. Trigger Hardware 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 10 Base processors on existing CMS Virtex7 trigger processor cards cluster ECAL using fine granularity information for e/γ candidates for track matching/veto + track isolation, and use wider H clusters behind for veto, etc. Global Trigger Stand-alone calo. trig. output: e/γ, τ, jet, E tmiss, E T HCALECAL HGCAL HF HCALECAL HGCAL HF HCALECAL HGCAL HF

L1 Calorimeter Trigger Upgrade  Calorimeter Trigger:  Process individual crystal energies instead of present 5x5 towers  Higher resolution matching to tracks: ΔR <  Improvement in stand- alone electron trigger efficiency + rate→ 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 11

 M&S costs are based on escalated prices of similar components used for the Phase 1 upgrade of the L1 trigger. Details on next slide  Labor costs are estimated from engineers currenlty on staff, or on standard rates as needed. Effort calculated as per the Phase 1 Trigger Upgrade Project.  International travel is estimated at $3K per trip, and domestic travel is estimated at $1K per trip. 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 12 Basis of Estimate

21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 13 Cost Estimate (from CMS Phase 2 TP) EB Channels (via Back-End Elect)61200# of xtals EE Channels (via Back-End Elect) – New Endcap61000 Use # of Shashlik channels temporarily until #’s from HGCAL available HB/HE Chan. (via Back-End Elect)13824From HCAL Phase 1 HF Channels (via Back-End Elect)1728 From HCAL Phase 1 but combine 2 measurements/PMT Information per channel (bits)12assume 10 bits energy and 2 bits quality Total Bits Bandwidth (bits/sec)6.61E+13Data transmitted at 40 MHz Card BW (bits/sec)4.92E+11 Assume present cards with 80x10 Gbps links running 192 bits at 40 MHz with 80% packing efficiency No. cards in Layer 1135 Cards This number of cards assumes the present Phase 1 Upgrade Boards Add 33% more cards for Layer 2179 Cards Multiply by 15 k$/card + 15% Spares + 16 k$/12 Cards Infrastucture3.5 M$ NB: Estimate done with Phase 1 CTP7 (Virtex7) card capabilities and costs. For Phase 2 Expect UltraScale, Ultrascale+ FPGA costs higher, but fewer boards used (next slide).

Cost Driver: FPGAs 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 14 Virtex 7 Ultrascale Ultrascale+ Virtex 7 used in CMS Phase 1: 80 x 10 Gbps transceivers = 800 Gbps Ultrascale+: 120 x 30 Gbps transceivers = 3600 Gbps (e.g. 4.5 x V7) Virtex 7: retail cost now: $7.5K ea. Ultrascale+: unofficial estimated retail cost: $15K ea. (e.g. twice V7 cost) (~ 50% of Phase 1 Cal. Trig. cost)

15 Construction Schedule Phase 2 Calorimeter Trigger Upgrade21-Jan-2016 W. Smith FY25 FY24 FY23FY22FY21FY20 FY19FY18 FY17 CD4 CD1 CD2 CD3 CD0 Specification and Technology R&D Trigger TDR Pre- production Installation LS 2 LS 3 Physics LHC Schedule CDR PDR CD3A FDR Prototyping and Demonstrators Production Readiness Review Production and Test Test & Commission

16 Cost Cost = AY $M (No Contingency) L3 AreaM&S*LaborTotalR&D Calorimeter Trigger884K2079 K 2963K535K 21-Jan-2016 W. SmithPhase 2 Calorimeter Trigger Upgrade *Includes Travel

17 Cost Profile Phase 2 Calorimeter Trigger Upgrade21-Jan-2016 W. Smith

18 Labor FTE Profile (double click to edit numbers) Phase 2 Calorimeter Trigger Upgrade21-Jan-2016 W. Smith

 C&S understood since based on Phase 1 Trigger Upgrade Systems experience  Cards are extrapolations of existing Phase 1 Trigger Upgrade Cards  C&S based on experience of the same team that built and wrote software and firmware for Phase 1 Trigger Upgrade 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 19 Cost and Schedule Risks

 Senior Engineer becomes unavailable (Low Risk)  Hire new engineer, subcontract to consulting firm, use FNAL engineer  Software or Firmware does not meet requirements (Low Risk)  Hire extra expert effort to recover schedule and help personnel  Boards are delayed (design, manufacture or testing) (Low Risk)  Hire extra effort to speed up testing schedule  Vendor non-performance (Low Risk)  Acquire spending authority to use alternative vendors (while original funds are being unencumbered).  Input or output electronics (non-trigger) delayed (Low Risk)  Built in capabilities of trigger electronics provide signals for their own inputs & outputs 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 20 Managed Trigger Risks & Mitigation

 Safety: follows procedures in CMS-doc-11587, FESHM  L3 Manager (W.S.) responsible for applying ISM to trigger upgrade. o Under direction of US CMS Project Management.  Modules similar to others built before, of small size and no high voltage  Quality Assurance: follows procedures in CMS-doc  Regularly evaluate achievement relative to performance requirements and appropriately validate or update performance requirements and expectations to ensure quality.  QA: Equipment inspections and verifications; Software code inspections, verifications, and validations; Design reviews; Baseline change reviews; Work planning; and Self-assessments.  All modules have hardware identifiers which are tracked in a database logging QA data through all phases of construction, installation, operation and repair.  Graded Approach:  Apply appropriate level of analysis, controls, and documentation commensurate with the potential to have an environmental, safety, health, radiological, or quality impact.  Four ESH&Q Risk levels are defined and documented in CMS-doc Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 21 Trigger ESH&Q

 After full testing at institute, shipped to CERN  All tests recorded (of all types) for individual boards in database  Tests use and validate software and firmware test release  Acceptance Testing in Electronics Integration Center (EIC) at CERN  Individual labs for CSC and Calorimeter Trigger  Boards retested to validate institute test results  Tests use software and firmware test release  Integration Testing in EIC  Row of racks with DAQ, Trigger, Central Clock, Crates of other subsystem electronics  Operation of a vertical slice with electronics to be tested installed.  Tests use and validate software and firmware commissioning release  Integration Testing at P5: Global Runs/Parallel Operation  Test with all CMS with cosmics when beam not running/with beam when running  Electronics installed in final locations with final cables  Full-scale tests with full CMS DAQ/Trigger/Clocking  Tests use software and firmware commissioning release  Handover to Operations at P5: Global Runs/Parallel Operation  After testing completes, continue with Global Runs/Parallel Operation  Validate software and firmware initial operational release 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 22 QA/QC: Testing and Validation

Phase 2 Trigger Algorithm R&D  Goal:  Allow development of calorimeter, correlation trigger electronics – specify: o Planned Algorithms o Necessary trigger primitives o Link counts and formats  Plan (with CMS HL-LHC Technical Proposal Milestones):  Initial definition of trigger algorithms, primitive objects and inter- layer objects (TP.L1.1) – 2QCY16  Baseline definition of trigger algorithms, primitive objects and interchange requirements with subdetectors. (TP.L1.3) – 2QCY17  Detailed Software emulator demonstrates implementation of core phase 2 trigger menu with baseline objects (TP.L1.4) – 4QCY17 o Used to inform the final implementation of the trigger hardware. 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 23

Phase 2 Trigger Hardware R&D - I  2 R&D activities:  Calorimeter Trigger Processor  Track Correlator Processor  Hardware R&D Milestones - I  Initial demonstration of key implementation technologies (TP.L1.2) – 4QCY16 o e.g. > 25 Gb data links, general applicability across Phase 2 o Start Construction of initial prototype circuits for demonstration of feasibility of trigger design, leads to:  Definition of hardware technology implementation baseline (TP.L1.5) – 1QCY18 o Testing and revisions of prototypes. o Used with algorithm and emulation baseline to define what is needed for → 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 24

Phase 2 Trigger Hardware R&D - II  Hardware R&D Milestones – II  Full-function prototypes produced which allow local comparison with emulator (TP.L1.6) – 4QCY18 o First boards which have sufficient channels, processing capability and bandwidth optical links to meet the requirements of the final boards o These boards will cover only a portion of the trigger processing logic, however, and only local comparisons will be possible between hardware behavior and the emulator.  Demonstrator system shows integration and scaling, global/full- chain comparison with emulator (TP.L1.7) – 4QCY19 o End-to-end comparisons over a slice of the detector which include multiple full-capability prototype boards and the prototype full-capability infrastructure o Goal of demonstrating a prototype system with its infrastructure and testing environment capable of being connected to its front end detector for test-beam validation to follow.  Final Milestone:  Phase 2 Trigger TDR (TP.L1.8) – 1Q2020 o Based on results from Trigger Demonstrators. 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 25

 Phase 1 upgrade: two generations (V5, V6) before production boards—similar path reasonable for Phase 2  Phase 2 upgrade working terminology:  “APDx”—Advanced Processor Demonstrator, APD1 for gen-1, APD2 for gen-2, etc. o Evolution of the successful CTP7 architecture, staying current with advances in FPGA, SoC, PCB, embedded OS and optical technologies o Supported by simpler auxiliary boards as necessary (RTMs, etc.)  APM—Advanced Processor Module—Phase 2 production platform  Today: CTP7 a very capable “Gen 0” demonstrator  Supporting Phase 2 Tracking Trigger and Calorimeter Trigger R&D  Comparatively “young” platform (< 2 years old) w/ new technology  Receiving interest from other groups as an upgrade and/or Phase 2 R&D platform 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 26 Phase 2 Cal. Trig. Demonstrators

 CTP7 “Gen zero” demonstrator  12 MGT MicroTCA backplane links  67 Rx and 48 Tx 10G optical links  Modular V7 firmware architecture for ease of customization  Currently have 4 different configurations in P5 and R&D use 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 27 CTP7 as a Phase-2 Demonstrator 12 Backplane MGT Connections (plus DAQ) 10G capable frontpanel optical links, 67 Rx and 48 Tx

 MicroTCA.0—the MicroTCA for Phase 1  MicroTCA.4—a standard with a rear transition module (RTM) about the same size as a double-width AMC  MicroTCA.4 shares payload power between AMC and RTM  New Vadatech chassis (VT815) supports 12 full size AMC+RTM combinations with 120W per slot  ATCA—older standard, physically larger  Shape of the RTM in ATCA limits its utility, but overall ATCA provides about 2X the board and frontpanel area as MicroTCA.4  IPMI: CMS-common IPMI solutions (MMC, System Manager) supplied by Wisconsin can easily migrate to ATCA  Board costs and FPGAs  Xilinx UltraScale at about same cost/gate as Virtex-7, but gate/MGT ratios are higher in UltraScale—MGTs drive part selection in CMS  FPGA costs are going to dominate over form factor costs in the high performance applications 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 28 Upgrade Form Factors

Processi ng FPGA(s)  Next-gen FPGA and ZYNQ SoC devices  General upgrade to embedded Linux platform over CTP7  Direct optical interfaces for the ZYNQ PL section  DDR4 SDRAM on main FPGA for higher density and bandwidth  Optical module mix for compatibility with current and next-gen optical links 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 29 APD Architecture Example Processin g FPGA(s) ZYNQ SoC High BW DDR4 SDRAM System Memory GbE Control Path Optical Interfaces Front Side Optical Interface Flash File System Flash File System Backplane/RTM MGT Links

Summary 30 Phase 2 Calorimeter Trigger Upgrade21-Jan-2016 W. Smith

 R&D Program will result in designs for the Trigger Upgrade that will meet technical performance requirements  Scope and Specifications of Trigger Upgrade are sufficiently well-defined to support the C&S estimates  Upgrade based upon common hardware platforms and components  ES&H, QA plans, C&S based on experience with original trigger construction and Phase-1 upgrade  Management and Engineering teams are experienced with sufficient design skills, having designed and built original CMS trigger and Phase-1 Upgrade 21-Jan-2016 W. Smith Phase 2 Calorimeter Trigger Upgrade 31 Conclusions